Optimism Mainnet Blockchain Faces Outages and Instabilities

According to Blockworks, the Optimism Mainnet blockchain experienced a series of outages and instabilities on February 15, 2024, starting around 6:00 am ET. The team began investigating reports of an 'unsafe head stall,' which meant the sequencer stopped producing blocks. Developers identified the cause of the issue and applied a fix at 6:10 am ET, temporarily restoring operations. However, by 6:26 am ET, reports of additional instability questioned the effectiveness of the patch, as per the network's status page. The Optimism sequencer remains centralized, with OP Labs responsible for identifying and troubleshooting problems. The network resumed around 8:00 am ET after finding the suspected root cause. The incident was unrelated to the planned pause of OP Mainnet withdrawals, a scheduled maintenance that is ongoing at the time of writing. Arbitrum, the largest Ethereum rollup by total value locked (TVL), experienced a similar outage in December 2023. Although the team stated that no action was required from users or node operators following the resolution, community members in Discord dispute that. One member, hazim-j.eth, co-founder of Stackup, who works on account abstraction tooling, wrote that a full node restart of op-node and op-geth was required to get unstuck. During the outage, node operators observed single-block reorganizations (reorg) of the unsafe chain. A single-block reorg, not uncommon in blockchain operations, occurs when the chain diverges into two paths due to discrepancies in block validation and then reconverges on a single, agreed-upon path. In the case of Optimism, an 'unsafe' head reorg refers to blocks produced by the sequencer but not yet posted in batches to the Ethereum mainnet, according to a pseudonymous developer support engineer at OP Labs known as Soyboy Vegan. Optimism's documentation states that 'it is still possible for [unsafe transactions] to be excluded from the final blockchain if the Sequencer fails to publish the block to Ethereum in a timely manner.' The maintenance window scheduled for 9:00 to 10:00 am ET was intended to 'test Optimism's incident response capabilities,' according to an OP Labs X post.