Subtraction Overflow Panic in AA Transaction Execution #2204
Description
Describe the bug
Human Summary
I'm running a testnet node on (v1.0.0-rc.5) and it keeps panicking and I need to restart the node, it works then. The block I panicked on is listed below, but I've ran into this issue like 4-5 times, and when I applied the fix described in the issue and it stopped. I can create a PR with the fix described below and link it.
Block number: 1866723
Block hash: 0xdc813f22fbe2804ce96b066c9c9e1c40b422902ad2e46cb2149f2fab7390db0b
I'm not 100% where the odd behaviour comes from but it definitely to do with AA transactions and forking gas logic.
AI Summary
Unchecked subtraction of intrinsic gas from transaction gas limit causes node panic during block sync when executing AA transactions with 2D nonces on pre-T0 blocks.
Affected Version
- v1.0.0-rc.5 (confirmed)
- Current main branch (confirmed)
Panic Location
crates/revm/src/handler.rs:295 in execute_multi_call_with:
let mut remaining_gas = gas_limit - init_and_floor_gas.initial_gas;Root Cause
In Handler::execution() (v1.0.0-rc.5):
// 2D nonce gas is added unconditionally
let adjusted_gas = InitialAndFloorGas::new(
init_and_floor_gas.initial_gas + evm.nonce_2d_gas,
init_and_floor_gas.floor_gas,
);
// Gas limit check only happens for T0
if spec.is_t0() {
if gas_limit < adjusted_gas.initial_gas {
return Err(...); // Clean error
}
}
// For pre-T0: NO CHECK → proceeds to execute_multi_call with adjusted_gas
// → subtraction overflows when adjusted_gas.initial_gas > gas_limitTrigger Conditions
- Pre-T0 hardfork (Genesis) — bypasses T0 gas limit check
- AA transaction with 2D nonce (
nonce_key != 0) - gas_limit <
intrinsic_gas + nonce_2d_gas
Real-World Impact
- Testnet nodes crash during sync when replaying blocks
- Block 1866723 caused panic
- Nodes recover on restart if chain moves past problematic block
Stack Trace
attempt to subtract with overflow
3: execute_multi_call_with<...>
4: tempo_revm::handler::TempoEvmHandler<DB,I>::execute_multi_call
5: <tempo_revm::handler::TempoEvmHandler<DB,I> as revm_handler::handler::Handler>::execution
...
17: alloy_evm::block::BlockExecutor::execute_block
18: <reth_evm::execute::BasicBlockExecutor<F,DB> as reth_evm::execute::Executor<DB>>::execute_one
Suggested Fix
Replace unchecked subtraction with checked_sub in both paths:
execute_single_call_with:
let gas_limit = evm.ctx().tx().gas_limit()
.checked_sub(init_and_floor_gas.initial_gas)
.ok_or_else(|| TempoInvalidTransaction::InsufficientGasForIntrinsicCost {
gas_limit: evm.ctx().tx().gas_limit(),
intrinsic_gas: init_and_floor_gas.initial_gas,
})?;execute_multi_call_with:
let mut remaining_gas = gas_limit
.checked_sub(init_and_floor_gas.initial_gas)
.ok_or_else(|| TempoInvalidTransaction::InsufficientGasForIntrinsicCost {
gas_limit,
intrinsic_gas: init_and_floor_gas.initial_gas,
})?;Steps to reproduce
File: crates/revm/src/handler.rs (add to mod tests block)
Run with:
cargo test -p tempo-revm test_pre_t0_2d_nonce_gas_underflow_panics/// Regression test for pre-T0 2D nonce gas underflow panic.
///
/// When:
/// - Pre-T0 (Genesis hardfork) — bypasses T0 gas limit validation
/// - AA transaction with 2D nonce (nonce_key != 0)
/// - gas_limit passes initial check but insufficient after nonce gas added
///
/// The unchecked subtraction in execute_multi_call_with panics with
/// "attempt to subtract with overflow".
///
/// This test documents the bug. The fix should use checked_sub and return
/// InsufficientGasForIntrinsicCost error instead.
#[test]
#[should_panic(expected = "attempt to subtract with overflow")]
fn test_pre_t0_2d_nonce_gas_underflow_panics() {
use crate::evm::TempoEvm;
// Base intrinsic gas for AA tx with one call
const BASE_INTRINSIC_GAS: u64 = 21_000;
// Gas limit that passes validation but fails after 2D nonce gas is added
// Must be >= BASE_INTRINSIC_GAS but < BASE_INTRINSIC_GAS + NEW_NONCE_KEY_GAS
let gas_limit = BASE_INTRINSIC_GAS + 1000;
let db = CacheDB::new(EmptyDB::default());
let journal = Journal::new(db);
// Use Genesis hardfork - pre-T0 adds nonce gas AFTER validation check
#[allow(deprecated)]
let cfg = CfgEnv::<TempoHardfork>::default().with_spec(TempoHardfork::Genesis);
let ctx = Context::mainnet()
.with_db(CacheDB::new(EmptyDB::default()))
.with_block(TempoBlockEnv::default())
.with_cfg(cfg)
.with_tx(TempoTxEnv {
inner: revm::context::TxEnv {
gas_limit,
..Default::default()
},
tempo_tx_env: Some(Box::new(TempoBatchCallEnv {
aa_calls: vec![Call {
to: TxKind::Call(Address::random()),
value: U256::ZERO,
input: Bytes::new(),
}],
// Non-zero nonce_key triggers 2D nonce gas calculation
nonce_key: U256::from(1),
..Default::default()
})),
..Default::default()
})
.with_new_journal(journal);
let mut evm: TempoEvm<_, ()> = TempoEvm::new(ctx, ());
let mut handler: TempoEvmHandler<CacheDB<EmptyDB>, ()> = TempoEvmHandler::new();
// validate_initial_tx_gas calculates intrinsic gas including 2D nonce gas for pre-T0
// The gas returned will be BASE_INTRINSIC_GAS + NEW_NONCE_KEY_GAS
let init_and_floor_gas = handler.validate_initial_tx_gas(&mut evm).unwrap();
// This panics because init_and_floor_gas.initial_gas > gas_limit
let _ = handler.execution(&mut evm, &init_and_floor_gas);
}
### Logs
```text
2026-01-21T11:20:58.391450Z INFO Received block from consensus engine number=1866723 hash=0xdc813f22fbe2804ce96b066c9c9e1c40b422902ad2e46cb2149f2fab7390db0b
2026-01-21T11:20:58.394373Z INFO Received forkchoice updated message when syncing head_block_hash=0xdc813f22fbe2804ce96b066c9c9e1c40b422902ad2e46cb2149f2fab7390db0b safe_block_hash=0x8e2776a53a590f24e211acb6aa1da5b1da2a43b44b6aea1f8da3a55bcaa2565b finalized_block_hash=0x095ee491525b61cfb34d173f9522beba33b5d61955c183adbee55fe48da527fa
thread 'tokio-runtime-worker' (98075) panicked at /root/tempo/crates/revm/src/handler.rs:295:33:
attempt to subtract with overflow
stack backtrace:
2026-01-21T11:20:58.682862Z INFO Received block from consensus engine number=1866724 hash=0x028b17692b8525118210d6edb0e12cd13741ce669aaaa3eb1bacfc9fe7cc1f74
2026-01-21T11:20:58.683967Z INFO Received forkchoice updated message when syncing head_block_hash=0x028b17692b8525118210d6edb0e12cd13741ce669aaaa3eb1bacfc9fe7cc1f74 safe_block_hash=0x404bb7c8952b423ef688dc0fbc74125c4366f94a6c6ce898ef7edf5e9fc32c42 finalized_block_hash=0x9f927a1ca24a9af30ae37aaad12e04108f0622fcdd5bf3f79ec7d8309dcbb24e
0: __rustc::rust_begin_unwind
at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/std/src/panicking.rs:698:5
1: core::panicking::panic_fmt
at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/panicking.rs:80:14
2: core::panicking::panic_const::panic_const_sub_overflow
at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/panicking.rs:180:17
3: execute_multi_call_with<&mut revm_database::states::state::State<reth_revm::database::StateProviderDatabase<reth_provider::providers::state::latest::LatestStateProviderRef<reth_provider::providers::database::provider::DatabaseProvider<reth_db::implementation::mdbx::tx::Tx<reth_libmdbx::transaction::RW>, reth_node_types::NodeTypesWithDBAdapter<tempo_node::node::TempoNode, alloc::sync::Arc<reth_db::implementation::mdbx::DatabaseEnv, alloc::alloc::Global>>>>>>, revm_inspector::noop::NoOpInspector, fn(&mut tempo_revm::handler::TempoEvmHandler<&mut revm_database::states::state::State<reth_revm::database::StateProviderDatabase<reth_provider::providers::state::latest::LatestStateProviderRef<reth_provider::providers::database::provider::DatabaseProvider<reth_db::implementation::mdbx::tx::Tx<reth_libmdbx::transaction::RW>, reth_node_types::NodeTypesWithDBAdapter<tempo_node::node::TempoNode, alloc::sync::Arc<reth_db::implementation::mdbx::DatabaseEnv, alloc::alloc::Global>>>>>>, revm_inspector::noop::NoOpInspector>, &mut tempo_revm::evm::TempoEvm<&mut revm_database::states::state::State<reth_revm::database::StateProviderDatabase<reth_provider::providers::state::latest::LatestStateProviderRef<reth_provider::providers::database::provider::DatabaseProvider<reth_db::implementation::mdbx::tx::Tx<reth_libmdbx::transaction::RW>, reth_node_types::NodeTypesWithDBAdapter<tempo_node::node::TempoNode, alloc::sync::Arc<reth_db::implementation::mdbx::DatabaseEnv, alloc::alloc::Global>>>>>>, revm_inspector::noop::NoOpInspector>, &revm_interpreter::gas::calc::InitialAndFloorGas) -> core::result::Result<revm_handler::frame_data::FrameResult, revm_context_interface::result::EVMError<reth_storage_errors::provider::ProviderError, tempo_revm::error::TempoInvalidTransaction>>>
at ./crates/revm/src/handler.rs:295:33
4: tempo_revm::handler::TempoEvmHandler<DB,I>::execute_multi_call
at ./crates/revm/src/handler.rs:417:14
5: <tempo_revm::handler::TempoEvmHandler<DB,I> as revm_handler::handler::Handler>::execution
at ./crates/revm/src/handler.rs:520:18
6: revm_handler::handler::Handler::run_without_catch_error
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/revm-handler-14.1.0/src/handler.rs:150:36
7: <tempo_revm::handler::TempoEvmHandler<DB,I> as revm_handler::handler::Handler>::run
at ./crates/revm/src/handler.rs:479:20
8: tempo_revm::exec::<impl revm_handler::api::ExecuteEvm for tempo_revm::evm::TempoEvm<DB,I>>::transact_one
at ./crates/revm/src/exec.rs:42:11
9: revm_handler::api::ExecuteEvm::transact
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/revm-handler-14.1.0/src/api.rs:72:36
10: <tempo_evm::evm::TempoEvm<DB,I> as alloy_evm::evm::Evm>::transact_raw
at ./crates/evm/src/evm.rs:190:24
11: alloy_evm::evm::Evm::transact
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/alloy-evm-0.25.2/src/evm.rs:91:14
12: <alloy_evm::eth::block::EthBlockExecutor<E,Spec,R> as alloy_evm::block::BlockExecutor>::execute_transaction_without_commit
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/alloy-evm-0.25.2/src/eth/block.rs:132:18
13: <tempo_evm::block::TempoBlockExecutor<DB,I> as alloy_evm::block::BlockExecutor>::execute_transaction_without_commit
at ./crates/evm/src/block.rs:352:33
14: alloy_evm::block::BlockExecutor::execute_transaction_with_commit_condition
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/alloy-evm-0.25.2/src/block/mod.rs:208:27
15: alloy_evm::block::BlockExecutor::execute_transaction_with_result_closure
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/alloy-evm-0.25.2/src/block/mod.rs:175:14
16: alloy_evm::block::BlockExecutor::execute_transaction
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/alloy-evm-0.25.2/src/block/mod.rs:157:14
17: alloy_evm::block::BlockExecutor::execute_block
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/alloy-evm-0.25.2/src/block/mod.rs:319:18
18: <reth_evm::execute::BasicBlockExecutor<F,DB> as reth_evm::execute::Executor<DB>>::execute_one
at /root/.cargo/git/checkouts/reth-e231042ee7db3fb7/d76babb/crates/evm/evm/src/execute.rs:590:14
19: <reth_stages::stages::execution::ExecutionStage<E> as reth_stages_api::stage::Stage<Provider>>::execute::{{closure}}
at /root/.cargo/git/checkouts/reth-e231042ee7db3fb7/d76babb/crates/stages/stages/src/stages/execution.rs:347:26
20: reth_evm::metrics::ExecutorMetrics::metered_one::{{closure}}
at /root/.cargo/git/checkouts/reth-e231042ee7db3fb7/d76babb/crates/evm/evm/src/metrics.rs:80:53
21: reth_evm::metrics::ExecutorMetrics::metered
at /root/.cargo/git/checkouts/reth-e231042ee7db3fb7/d76babb/crates/evm/evm/src/metrics.rs:56:34
22: reth_evm::metrics::ExecutorMetrics::metered_one
at /root/.cargo/git/checkouts/reth-e231042ee7db3fb7/d76babb/crates/evm/evm/src/metrics.rs:80:14
23: execute<tempo_evm::TempoEvmConfig, reth_provider::providers::database::provider::DatabaseProvider<reth_db::implementation::mdbx::tx::Tx<reth_libmdbx::transaction::RW>, reth_node_types::NodeTypesWithDBAdapter<tempo_node::node::TempoNode, alloc::sync::Arc<reth_db::implementation::mdbx::DatabaseEnv, alloc::alloc::Global>>>>
at /root/.cargo/git/checkouts/reth-e231042ee7db3fb7/d76babb/crates/stages/stages/src/stages/execution.rs:346:39
24: reth_stages_api::stage::_::<impl reth_stages_api::stage::Stage<Provider> for alloc::boxed::Box<T>>::execute
at /root/.cargo/git/checkouts/reth-e231042ee7db3fb7/d76babb/crates/stages/api/src/stage.rs:240:1
25: {async_fn#0}<reth_node_types::NodeTypesWithDBAdapter<tempo_node::node::TempoNode, alloc::sync::Arc<reth_db::implementation::mdbx::DatabaseEnv, alloc::alloc::Global>>>
at /root/.cargo/git/checkouts/reth-e231042ee7db3fb7/d76babb/crates/stages/api/src/pipeline/mod.rs:479:43
26: {async_fn#0}<reth_node_types::NodeTypesWithDBAdapter<tempo_node::node::TempoNode, alloc::sync::Arc<reth_db::implementation::mdbx::DatabaseEnv, alloc::alloc::Global>>>
at /root/.cargo/git/checkouts/reth-e231042ee7db3fb7/d76babb/crates/stages/api/src/pipeline/mod.rs:232:86
27: {async_block#0}<reth_node_types::NodeTypesWithDBAdapter<tempo_node::node::TempoNode, alloc::sync::Arc<reth_db::implementation::mdbx::DatabaseEnv, alloc::alloc::Global>>>
at /root/.cargo/git/checkouts/reth-e231042ee7db3fb7/d76babb/crates/stages/api/src/pipeline/mod.rs:174:42
28: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=(reth_stages_api::pipeline::Pipeline<reth_node_types::NodeTypesWithDBAdapter<tempo_node::node::TempoNode, alloc::sync::Arc<reth_db::implementation::mdbx::DatabaseEnv, alloc::alloc::Global>>>, core::result::Result<reth_stages_api::pipeline::ctrl::ControlFlow, reth_stages_api::error::PipelineError>)> + core::marker::Send), alloc::alloc::Global>>
at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/future/future.rs:133:9
29: {async_block#0}<reth_node_types::NodeTypesWithDBAdapter<tempo_node::node::TempoNode, alloc::sync::Arc<reth_db::implementation::mdbx::DatabaseEnv, alloc::alloc::Global>>>
at /root/.cargo/git/checkouts/reth-e231042ee7db3fb7/d76babb/crates/engine/tree/src/backfill.rs:144:72
30: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>
at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/future/future.rs:133:9
31: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::future::future::Future>::poll
at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/panic/unwind_safe.rs:299:9
32: <futures_util::future::future::catch_unwind::CatchUnwind<Fut> as core::future::future::Future>::poll::{{closure}}
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/future/catch_unwind.rs:37:44
33: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/panic/unwind_safe.rs:274:9
34: do_call<core::panic::unwind_safe::AssertUnwindSafe<futures_util::future::future::catch_unwind::{impl#1}::poll::{closure_env#0}<core::panic::unwind_safe::AssertUnwindSafe<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>>>>, core::task::poll::Poll<()>>
at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/std/src/panicking.rs:590:40
35: __rust_try
36: catch_unwind<core::task::poll::Poll<()>, core::panic::unwind_safe::AssertUnwindSafe<futures_util::future::future::catch_unwind::{impl#1}::poll::{closure_env#0}<core::panic::unwind_safe::AssertUnwindSafe<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>>>>>
at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/std/src/panicking.rs:553:19
37: catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<futures_util::future::future::catch_unwind::{impl#1}::poll::{closure_env#0}<core::panic::unwind_safe::AssertUnwindSafe<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>>>>, core::task::poll::Poll<()>>
at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/std/src/panic.rs:359:14
38: <futures_util::future::future::catch_unwind::CatchUnwind<Fut> as core::future::future::Future>::poll
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/future/catch_unwind.rs:37:9
39: <F as futures_core::future::TryFuture>::try_poll
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/future.rs:92:14
40: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/try_future/into_future.rs:34:31
41: poll<futures_util::future::try_future::into_future::IntoFuture<futures_util::future::future::catch_unwind::CatchUnwind<core::panic::unwind_safe::AssertUnwindSafe<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>>>>, futures_util::fns::MapErrFn<reth_tasks::{impl#6}::spawn_critical_as::{closure_env#0}<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>>>, core::result::Result<(), ()>>
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/future/map.rs:55:44
42: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/lib.rs:86:35
43: <futures_util::future::try_future::MapErr<Fut,F> as core::future::future::Future>::poll
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/lib.rs:86:35
44: poll<futures_util::future::try_future::MapErr<futures_util::future::future::catch_unwind::CatchUnwind<core::panic::unwind_safe::AssertUnwindSafe<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>>>, reth_tasks::{impl#6}::spawn_critical_as::{closure_env#0}<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>>>>
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tracing-futures-0.2.5/src/lib.rs:283:20
45: poll<&mut tracing_futures::Instrumented<futures_util::future::try_future::MapErr<futures_util::future::future::catch_unwind::CatchUnwind<core::panic::unwind_safe::AssertUnwindSafe<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>>>, reth_tasks::{impl#6}::spawn_critical_as::{closure_env#0}<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>>>>>
at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/future/future.rs:133:9
46: futures_util::future::future::FutureExt::poll_unpin
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/future/mod.rs:558:24
47: poll<reth_tasks::shutdown::Shutdown, core::pin::Pin<&mut tracing_futures::Instrumented<futures_util::future::try_future::MapErr<futures_util::future::future::catch_unwind::CatchUnwind<core::panic::unwind_safe::AssertUnwindSafe<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>>>, reth_tasks::{impl#6}::spawn_critical_as::{closure_env#0}<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>>>>>>
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/select.rs:118:37
48: {async_block#1}<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>>
at /root/.cargo/git/checkouts/reth-e231042ee7db3fb7/d76babb/crates/tasks/src/lib.rs:474:47
49: {closure#0}<reth_tasks::{impl#6}::spawn_critical_as::{async_block_env#1}<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>>>
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/park.rs:285:71
50: with_budget<core::task::poll::Poll<()>, tokio::runtime::park::{impl#4}::block_on::{closure_env#0}<reth_tasks::{impl#6}::spawn_critical_as::{async_block_env#1}<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>>>>
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/task/coop/mod.rs:167:5
51: budget<core::task::poll::Poll<()>, tokio::runtime::park::{impl#4}::block_on::{closure_env#0}<reth_tasks::{impl#6}::spawn_critical_as::{async_block_env#1}<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>>>>
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/task/coop/mod.rs:133:5
52: block_on<reth_tasks::{impl#6}::spawn_critical_as::{async_block_env#1}<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>>>
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/park.rs:285:31
53: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/context/blocking.rs:66:14
54: tokio::runtime::handle::Handle::block_on_inner::{{closure}}
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/handle.rs:369:22
55: tokio::runtime::context::runtime::enter_runtime
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/context/runtime.rs:65:16
56: tokio::runtime::handle::Handle::block_on_inner
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/handle.rs:368:9
57: block_on<reth_tasks::{impl#6}::spawn_critical_as::{async_block_env#1}<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>>>
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/handle.rs:347:18
58: reth_tasks::TaskExecutor::spawn_on_rt::{{closure}}
at /root/.cargo/git/checkouts/reth-e231042ee7db3fb7/d76babb/crates/tasks/src/lib.rs:374:59
59: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/blocking/task.rs:42:21
60: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/task/core.rs:365:24
61: with_mut<tokio::runtime::task::core::Stage<tokio::runtime::blocking::task::BlockingTask<reth_tasks::{impl#6}::spawn_on_rt::{closure_env#0}<reth_tasks::{impl#6}::spawn_critical_as::{async_block_env#1}<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>>>>>, core::task::poll::Poll<()>, tokio::runtime::task::core::{impl#6}::poll::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<reth_tasks::{impl#6}::spawn_on_rt::{closure_env#0}<reth_tasks::{impl#6}::spawn_critical_as::{async_block_env#1}<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>>>>, tokio::runtime::blocking::schedule::BlockingSchedule>>
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/loom/std/unsafe_cell.rs:16:9
62: poll<tokio::runtime::blocking::task::BlockingTask<reth_tasks::{impl#6}::spawn_on_rt::{closure_env#0}<reth_tasks::{impl#6}::spawn_critical_as::{async_block_env#1}<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>>>>, tokio::runtime::blocking::schedule::BlockingSchedule>
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/task/core.rs:354:30
63: tokio::runtime::task::harness::poll_future::{{closure}}
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/task/harness.rs:535:30
64: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/core/src/panic/unwind_safe.rs:274:9
65: do_call<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<reth_tasks::{impl#6}::spawn_on_rt::{closure_env#0}<reth_tasks::{impl#6}::spawn_critical_as::{async_block_env#1}<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>>>>, tokio::runtime::blocking::schedule::BlockingSchedule>>, core::task::poll::Poll<()>>
at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/std/src/panicking.rs:590:40
66: __rust_try
67: catch_unwind<core::task::poll::Poll<()>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<reth_tasks::{impl#6}::spawn_on_rt::{closure_env#0}<reth_tasks::{impl#6}::spawn_critical_as::{async_block_env#1}<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>>>>, tokio::runtime::blocking::schedule::BlockingSchedule>>>
at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/std/src/panicking.rs:553:19
68: catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<reth_tasks::{impl#6}::spawn_on_rt::{closure_env#0}<reth_tasks::{impl#6}::spawn_critical_as::{async_block_env#1}<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>>>>, tokio::runtime::blocking::schedule::BlockingSchedule>>, core::task::poll::Poll<()>>
at /rustc/ded5c06cf21d2b93bffd5d884aa6e96934ee4234/library/std/src/panic.rs:359:14
69: tokio::runtime::task::harness::poll_future
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/task/harness.rs:523:18
70: tokio::runtime::task::harness::Harness<T,S>::poll_inner
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/task/harness.rs:210:27
71: tokio::runtime::task::harness::Harness<T,S>::poll
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/task/harness.rs:155:20
72: tokio::runtime::task::raw::poll
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/task/raw.rs:325:13
73: tokio::runtime::task::raw::RawTask::poll
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/task/raw.rs:255:18
74: tokio::runtime::task::UnownedTask<S>::run
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/task/mod.rs:546:13
75: tokio::runtime::blocking::pool::Task::run
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/blocking/pool.rs:161:19
76: run
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/blocking/pool.rs:516:22
77: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/blocking/pool.rs:474:47
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2026-01-21T11:20:59.412119Z ERROR Critical task `pipeline task` panicked: `attempt to subtract with overflow`
2026-01-21T11:20:59.412855Z ERROR backfill sync task dropped err=channel closed
2026-01-21T11:20:59.413116Z ERROR Fatal error in consensus engine
2026-01-21T11:20:59.417323Z ERROR shutting down due to error
Error: execution node failed
Caused by:
Critical task `pipeline task` panicked: `attempt to subtract with overflow`
Platform(s)
Linux (x86)
Container Type
Not running in a container
What version/commit are you on?
v1.0.0-rc.5
If you've built from source, provide the full command you used
just build-all
Code of Conduct
- I agree to follow the Code of Conduct