The Incident

On Saturday, June 29, Kreivo stopped producing blocks after a runtime upgrade to version 0.13.0. This upgrade aimed to introduce pallet contracts, enable bridged DOT reception from Kusama AssetHub, and update our PolkadotSDK dependency from version 1.5 to 1.12. The main goal was to prepare Kreivo for async backing, which would reduce block times from 12 to 6 seconds.

The Issue

Despite thorough testing before the upgrade, the problem only appeared hours after enactment(at the end of the era). The current rapid development of the PolkadotSDK makes it challenging for small teams like ours to constantly update and migrate code. Our main issue stemmed from the significant version jump (1.5 to 1.12), likely missing crucial intermediate migrations.

Impact

While most of our users (DAOs) are aware of Kreivo's beta testing stage and aren't critically affected, two key exceptions are:

1. Virto team: Our salaries are paid through the DAO's governance, which is currently inaccessible.
2. KSM Community: This decentralized voice in Kusama can't vote on OpenGov proposals.

Proposed Solution

While this proposal is being voted on, we'll prepare an updated WASM of the runtime that fixes the issue. Unlocking Kreivo means the "manager account" will be able to do a schedule_code_upgrade to manually set the updated parachain's code in the relay chain to match the code manually set by collators with the codeSubstitute feature of the chainspec.

Request

We ask Kusama governance and the Fellowship to approve unlocking the Kreivo parachain so we can restore functionality. After resolution, we'll conduct a public postmortem and implement additional safeguards for the future.

We are glad this kind of issue occurred this early in the lifetime of the chain when impact is manageable and are now more convinced builders need simpler ways to manage this kind of outages, ironically I believe a "governance parachain" like Kreivo where parachains can host their governance can be a great solution to this issue as people can vote to unlock a parachain even if it has halted.