Sepolia Incident

This blog station discloses a menace against the Ethereum web that was contiguous from the Merge up until the Dencun hard fork.

Background

Prior to the merge, antithetic connection size limits for RPC connection were acceptable to support clients from denial-of-service (DOS) attacks. These limits, applied to messages received via HTTP endpoints, were carried implicit to the motor API, which plays a important relation successful connecting Execution and Consensus Layer clients during artifact production. Due to the motor API's engagement successful artifact production, it became imaginable for blocks to beryllium produced that surpassed the RPC size limits of immoderate clients but remained wrong the acceptable scope for others.

If an attacker creates a connection that exceeds the size bounds of the lawsuit with the lowest setting, portion inactive adhering to the state bounds requirements, and past waits for a artifact to beryllium produced, it could effect successful a concern wherever immoderate clients respect the artifact arsenic valid, portion others cull it, issuing a HTTP mistake codification "413: Content Too Large."

Impact

An attacker that could trade these messages would beryllium capable to unit the bulk of nodes (=geth) to cull blocks that a number would accept. These blocks would beryllium forked distant and the proposer would miss retired connected rewards.

In the opening we thought that it was lone imaginable to make these blocks by utilizing builders oregon a modified mentation of a client. Geth has a builtin bounds of 128KB for transactions, which means that a large transaction similar the 1 nether treatment would not extremity up successful the transaction pools of immoderate geth node. It was nevertheless imaginable to inactive trigger the bounds by having a lawsuit with a higher bounds suggest the artifact and the CL requesting validation of this projected bigger block.

We projected a solution successful temporarily lowering the RPC bounds connected each clients to the lowest worth (5MB). This would marque the artifact invalid and an attacker would beryllium precise constricted successful the chaos they tin origin successful the web since the bulk of the nodes would cull their blocks.

However connected February 7th we discovered that it was imaginable to make a artifact that would deed the 5MB bounds with a clump of transactions that are beneath the 128KB bounds and not transcend 30 cardinal gas.

This is simply a bigger contented due to the fact that we realized an attacker could make a clump of precocious paying transactions and nonstop them to the network. Since helium outpays everyone other successful the mempool, each node (even geth nodes) would see the onslaught transactions successful their artifact frankincense creating a artifact that would not beryllium accepted by the bulk of the network, resulting successful a batch of forks (all being deemed valid by the number nodes) and the concatenation keeps reorging implicit and implicit again.

Later connected February 7th, we came to the decision that everyone raising their RPC limits would beryllium the safer alternative.

Timeline

• 2024-02-06 13:00: Toni (EF), Pari (EF) and Justin (Besu) effort to taxable a specificly grinded transaction to the network. The transaction contributes to up to 2.7 MB blocks erstwhile snappy compressed.
• 2024-02-06 13:25: Pari receives errors from his section Geth node though the transaction should beryllium valid.
• 2024-02-06 15:14: Justin managed to enactment the transaction successful a artifact and submitted it done the Besu client.
• 2024-02-06 20:46: Sam (EF) alerts Pari (special acknowledgment to mysticryuujin connected X), Toni and Alex astir definite Sepolia nodes struggeling.
• 2024-02-06 21:05: Team treble checks with Marius from Geth and confirms the bug.
• 2024-02-06 21:10: The pack gets unneurotic to debug it
• 2024-02-07 23:40: We decided for each clients to bounds their RPC petition bounds to 5MB
• 2024-02-07 6:40: We discovered that determination mightiness beryllium a bigger contented and the onslaught tin beryllium executed with transactions little than 128KB size.
• 2024-02-07 10:00: We decided for each clients to summation the RPC petition limit.
• 2024-02-07 21:00: The hole was merged successful geth.
• 2024-02-09: Geth was released

While Geth was the lone lawsuit affected by this bug, different clients person besides updated their defaults to beryllium harmless of this onslaught adjacent if state limits are increased. The lawsuit teams indicated that the pursuing updates person the harmless rpc limits:

Geth: v1.13.12

Nethermind: v1.25.4

Besu: 24.1.2

Erigon: v2.58.0

Reth: v0.1.0-alpha.18