# Picasso Post Mortem

On Friday December 17, 2:55:54PM the Picasso parachain on the Kusama relay chain stopped producing blocks, the last block it produced was backed by [relay block #10,561,126](https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fkusama-rpc.polkadot.io#/explorer/query/0x46d9d6036ad79e55f31d10b841cabb5fec3303618e91dfb99b860f289d6b697c). The reason was a faulty runtime upgrade that was applied at parachain block [#75,775](https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fpicasso-rpc.composable.finance#/explorer/query/0xcaf7f7aa3801aace93afb2828918c3719d688bd299be196b26daa1b1967adfe1).

  

![](https://t20465559.p.clickup-attachments.com/t20465559/cfd749f3-82af-4adb-a03f-85be03323ca3/Screenshot%202021-12-18%20at%207.18.22%20PM.png)

  

Specifically, a runtime upgrade that changed it's block-times for the aura consensus worker. This piece of documentation here doesn't specify that this value shouldn't be changed once the chain is live and so was updated from `3000` (milli-seconds) to `6000` (milli-seconds). Which was more in line with the relay chain's own expected block times for para-chains which is 12 seconds.

  

![](https://t20465559.p.clickup-attachments.com/t20465559/2fc0fe24-5ed8-4556-950f-8c719cf5f1af/Screenshot%202021-12-18%20at%207.20.48%20PM.png)

  

The reason it was `6s` in the first place was because our runtime was first a standalone substrate blockchain and had a `6s` expected block time, unfortunately when it was converted to a parachain, this value wasn't updated to parachain standard block time. Which in turn caused spotty block production on Picasso with block times averaging `30s`. Which makes sense, since the collators are authoring blocks every `6s` but are really only allowed to produce blocks every `12s` by the relay chain, so collators often miss their slots.

  

After the runtime upgrade, all our collators stopped producing blocks. Primarily because from the perpective of the parachain, the new slot numbers produced by the new runtime code are for earlier slots. From the logs of one of the collators:

  

```
2021-12-18 05:54:00.506 DEBUG tokio-runtime-worker aura: [Parachain] Starting authorship at slot 136650570; timestamp = 1639806840450
```

  

Note the slot number, now compare with slot number of the last block on the chain.

  

![](https://t20465559.p.clickup-attachments.com/t20465559/6b6e8c8b-8645-4f50-98e9-a0be410de316/Screenshot%202021-12-18%20at%206.59.53%20AM.png)

  

It would take roughly 53 years for our chain to start producing blocks.

  

With the chain unable to produce blocks it became clear we have only 2 options:

  

1.  Reset our parachain runtime wasm on the relay chain with a wasm file that reverts the changes to the `MinimumPeriod` then, create a new chain-spec with that same wasm blob specified as a wasm override, then update our collators so they can resume producing blocks.

  

Pros

  

*   None whatsoever

  

Cons

  

*   We still haven't fixed the spotty block production issue which causes collators to miss their slots and would need to do a more complicated runtime upgrade in the future to address this issue.

  

2\. We can call `paras.force_set_current_head` and `paras.force_set_current_wasm` inorder to reset the chain with a new wasm/header with the correct `MinimumPeriod`

  

Pros

  

*   We don't have to plan a more complicated runtime upgrade in the future in order to fix block production times on Picasso as the new chain would be properly configured from the get-go

  

  

Cons

  

*   We lose our chain state, but technically its still a Poa chain and nothing critical yet has been stored on it.

  

After careful consideration we decided to go with approach 2. primarily because it's the faster route to solving all our problems (un-bricking the chain and `30s` block production). Otherwise we'd have to first modify the aura pallet to support slot offsets allowing us upgrade our chain and successfully change the `MinimumPeriod` value then properly test this new functionality. But luckily because Picasso is also still a Poa (proof of authority) chain, it can afford to be restarted.

  

Starting from scratch with the right config will prevent us from having any more troubles with potentially bricking Picasso even after it becomes community run.

## Proposal

We'd like to restart the chain from genesis with a new header and wasm file using `paras.force_set_current_head` and `paras.force_set_current_code`

Send a batch of dispatch calls.

May be called from any origin.

- `calls`: The calls to be dispatched from the same origin. The number of call must not
  exceed the constant: `batched_calls_limit` (available in constant metadata).

If origin is root then call are dispatch without checking origin filter. (This includes
bypassing `frame_system::Config::BaseCallFilter`).

# <weight>
- Complexity: O(C) where C is the number of calls to be batched.
# </weight>

This will return `Ok` in all circumstances. To determine the success of the batch, an
event is deposited. If a call failed and the batch was interrupted, then the
`BatchInterrupted` event is deposited, along with the number of successful calls made
and the error of the failed call. If all were successful, then the `BatchCompleted`
event is deposited.

- The Picasso parachain on the Kusama relay chain stopped producing blocks.
- The reason was a faulty runtime upgrade that changed the block times.
- The collators stopped producing blocks because the new slot numbers were for earlier slots.
- There are two options to fix the issue: reset the parachain runtime or reset the chain with a new wasm/header.
- The decision was made to go with the second option to solve the problems faster.

#162 Unbrick Picasso

No Active Proposals