Notice: Polkadot has migrated to AssetHub. Balances, data, referenda, and other on-chain activity has moved to AssetHub.Learn more
[Whitelisted Caller] Referendum #622
The Incident
On 25 November, the Robonomics parachain on Kusama stopped producing blocks following a major runtime and node upgrade to v4.0.
This upgrade was intended to transition the parachain consensus from OpenCollator to AURA.
During the upgrade, temporary AURA authorities were expected to be set in the OnRuntimeMigration block.
The Issue
Although the runtime upgrade executed, collators running OpenConsensus were no longer able to author blocks, while the new AURA authority set was not yet fully established. As a result, the chain halted mid-migration, before the new authorities could take over.
All runtime testing was previously performed using Chopsticks, where this issue did not appear due to differences in the execution environment.
Impact
The block production halt affects several Robonomics services:
- Sensors.social currently operates in off-chain mode only
- CEX integrations (e.g., Kraken) temporarily suspended withdrawals
- Democracy and TechComm motions are frozen
- General degradation of parachain functionality
Proposed Solution
While this proposal is being voted on, the Robonomics team will prepare an updated runtime WASM that resolves the underlying issue (see: https://github.com/paritytech/polkadot-sdk/issues/10428).
Unlocking the parachain will allow the manager account to perform a schedule_code_upgrade, enabling:
- The relay chain to apply the corrected runtime code, and
- Collators to temporarily use codeSubstitute in their chainspec to align with the updated runtime.
This will restore block production and return the parachain to normal operation.
Request
We respectfully ask Kusama Governance and the Polkadot Fellowship to approve unlocking the Robonomics parachain so that full functionality can be restored.
After resolution, we will publish a public postmortem and implement improved migration testing procedures (including Zombienet) to prevent similar incidents in the future.
Comments (0)