Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sync assertion / crash on invalid block #3043

Open
arnetheduck opened this issue Jan 30, 2025 · 4 comments
Open

Sync assertion / crash on invalid block #3043

arnetheduck opened this issue Jan 30, 2025 · 4 comments

Comments

@arnetheduck
Copy link
Member

On the forked-layers branch (merged with master):

...skipping...
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker.nim(129) _ZN9runDaemon28runDaemon
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker/blocks_staged.nim(271) _ZN13blocks_staged18blocksStagedImportE6CtxRefIN11worker_desc13BeaconCtxDataEE6staticI6stringE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker/blocks_staged.nim(361) _ZN18blocksStagedImport18blocksStagedImportE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker/blocks_unproc.nim(65) _ZN13blocks_unproc18blocksUnprocCommitE6CtxRefIN11worker_desc13BeaconCtxDataEE4uInt6uInt646uInt64
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker/blocks_unproc.nim(73) _ZN13blocks_unproc18blocksUnprocCommitE6CtxRefIN11worker_desc13BeaconCtxDataEE4uInt6uInt646uInt64
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/std/assertions.nim(41) _ZN10assertions16failedAssertImplE6string
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/std/assertions.nim(36) _ZN10assertions11raiseAssertE6string
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/fatal.nim(53) _ZN10assertions8sysFatalE8typeDescIN10exceptions15AssertionDefectEE6string
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/excpt.nim(345) _ZN6system18rawWriteStackTraceE3varI3seqIN6system15StackTraceEntryEEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(62) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI3seqIN6system15StackTraceEntryEEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: [[reraised from:
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(879) mainJan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(867) NimMain
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/nimbus_execution_client.nim(281) _ZN23nimbus_execution_client3runE3refIN11nimbus_desc26NimbusNodecolonObjectType_EEN6config10NimbusConfE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncengine.nim(150) _ZN11asyncengine4pollE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/sync_sched.nim(221) _ZN10daemonLoop10daemonLoopE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN6beacon9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN9runDaemon9runDaemonE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker.nim(102) _ZN6worker9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE6staticI6stringE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker.nim(129) _ZN9runDaemon28runDaemon
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker/blocks_staged.nim(271) _ZN13blocks_staged18blocksStagedImportE6CtxRefIN11worker_desc13BeaconCtxDataEE6staticI6stringE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker/blocks_staged.nim(369) _ZN18blocksStagedImport18blocksStagedImportE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(62) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI3seqIN6system15StackTraceEntryEEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: ]]
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: [[reraised from:
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(879) main
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(867) NimMain
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/nimbus_execution_client.nim(281) _ZN23nimbus_execution_client3runE3refIN11nimbus_desc26NimbusNodecolonObjectType_EEN6config10NimbusConfE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncengine.nim(150) _ZN11asyncengine4pollE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/sync_sched.nim(221) _ZN10daemonLoop10daemonLoopE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN6beacon9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN9runDaemon9runDaemonE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker.nim(102) _ZN6worker9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE6staticI6stringE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker.nim(129) _ZN9runDaemon28runDaemon
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker/blocks_staged.nim(271) _ZN13blocks_staged18blocksStagedImportE6CtxRefIN11worker_desc13BeaconCtxDataEE6staticI6stringE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker/blocks_staged.nim(370) _ZN18blocksStagedImport18blocksStagedImportE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(62) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI3seqIN6system15StackTraceEntryEEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: ]]
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: [[reraised from:
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(879) main
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(867) NimMain
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/nimbus_execution_client.nim(281) _ZN23nimbus_execution_client3runE3refIN11nimbus_desc26NimbusNodecolonObjectType_EEN6config10NimbusConfE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncengine.nim(150) _ZN11asyncengine4pollE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/sync_sched.nim(221) _ZN10daemonLoop10daemonLoopE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN6beacon9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN9runDaemon9runDaemonE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker.nim(102) _ZN6worker9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE6staticI6stringE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker.nim(127) _ZN9runDaemon28runDaemon
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(62) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI3seqIN6system15StackTraceEntryEEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: ]]
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: [[reraised from:
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(879) main
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(867) NimMain
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/nimbus_execution_client.nim(281) _ZN23nimbus_execution_client3runE3refIN11nimbus_desc26NimbusNodecolonObjectType_EEN6config10NimbusConfE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncengine.nim(150) _ZN11asyncengine4pollE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/sync_sched.nim(221) _ZN10daemonLoop10daemonLoopE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN6beacon9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN9runDaemon9runDaemonE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker.nim(102) _ZN6worker9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE6staticI6stringE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker.nim(105) _ZN9runDaemon28runDaemon
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(62) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI3seqIN6system15StackTraceEntryEEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: ]]
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: [[reraised from:
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(879) main
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(867) NimMain
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/nimbus_execution_client.nim(281) _ZN23nimbus_execution_client3runE3refIN11nimbus_desc26NimbusNodecolonObjectType_EEN6config10NimbusConfE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncengine.nim(150) _ZN11asyncengine4pollE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/sync_sched.nim(221) _ZN10daemonLoop10daemonLoopE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN6beacon9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN9runDaemon9runDaemonE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker.nim(102) _ZN6worker9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE6staticI6stringE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon/worker.nim(106) _ZN9runDaemon28runDaemon
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(62) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI3seqIN6system15StackTraceEntryEEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: ]]
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: [[reraised from:
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(879) main
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(867) NimMain
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/nimbus_execution_client.nim(281) _ZN23nimbus_execution_client3runE3refIN11nimbus_desc26NimbusNodecolonObjectType_EEN6config10NimbusConfE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncengine.nim(150) _ZN11asyncengine4pollE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/sync_sched.nim(221) _ZN10daemonLoop10daemonLoopE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN6beacon9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN9runDaemon9runDaemonE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(62) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI3seqIN6system15StackTraceEntryEEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: ]]
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: [[reraised from:
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(879) main
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(867) NimMain
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/nimbus_execution_client.nim(281) _ZN23nimbus_execution_client3runE3refIN11nimbus_desc26NimbusNodecolonObjectType_EEN6config10NimbusConfE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncengine.nim(150) _ZN11asyncengine4pollE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/sync_sched.nim(221) _ZN10daemonLoop10daemonLoopE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(37) _ZN6beacon9runDaemonE6CtxRefIN11worker_desc13BeaconCtxDataEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/beacon.nim(38) _ZN9runDaemon9runDaemonE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(62) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI3seqIN6system15StackTraceEntryEEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: ]]
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: [[reraised from:
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(879) main
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(867) NimMain
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/nimbus_execution_client.nim(281) _ZN23nimbus_execution_client3runE3refIN11nimbus_desc26NimbusNodecolonObjectType_EEN6config10NimbusConfE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncengine.nim(150) _ZN11asyncengine4pollE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/sync_sched.nim(242) _ZN10daemonLoop10daemonLoopE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(62) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI3seqIN6system15StackTraceEntryEEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: ]]
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: [[reraised from:
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(879) main
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/protocol/snap/snap_types.nim(867) NimMain
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/nimbus_execution_client.nim(281) _ZN23nimbus_execution_client3runE3refIN11nimbus_desc26NimbusNodecolonObjectType_EEN6config10NimbusConfE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncengine.nim(150) _ZN11asyncengine4pollE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nim-chronos/chronos/internal/asyncfutures.nim(371) _ZN12asyncfutures14futureContinueE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/nimbus/sync/sync_sched.nim(243) _ZN10daemonLoop10daemonLoopE3refIN7futures26FutureBasecolonObjectType_EE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: /opt/nimbus-eth1/vendor/nimbus-build-system/vendor/Nim/lib/system/stacktraces.nim(62) _ZN11stacktraces30auxWriteStackTraceWithOverrideE3varI3seqIN6system15StackTraceEntryEEE
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: ]]
Jan 30 22:11:21 node run_mainnet_ec.sh[1064493]: Error: unhandled exception: /opt/nimbus-eth1/nimbus/sync/beacon/worker/blocks_unproc.nim(72, 3) `ctx.blk.unprocessed.merge(rMinPt, rMaxPt) == rMaxPt - rMinPt + 1`  [AssertionDefect]

INF 2025-01-30 22:11:20.450+01:00 Importing blocks                           topics="beacon sync" iv=#21013593..#21014200 nBlocks=608 base=#21013464 head=#21013592 target=#21739790
DBG 2025-01-30 22:11:21.027+01:00 wrong state root in block                  blockNumber=21013593 blockHash=0xe2d459fd8c2c972391da8cddfbac19f31413d779ba1b1c73c31bcabd7fd058aa parentHash=0x85cb24b00ed4204fe042865c4d70c8b6a86d6c212800075cba99e4be5f83ff62 expected=0x2db6858070460c80b6c9802f5dffca67bdb0ecfa184b279d71a1ca1ce2ca48aa actual=0x1159a10bc4fa3d2e13c4d5335aa382c7f9b6b509201c477ede5ab7da07424730 arrivedFrom=0x6284332f5b991dcac7dfc12e465dca8b7829a1e99e1458d2d241300091d63690
WRN 2025-01-30 22:11:21.027+01:00 RunDaemon: import block error              topics="beacon sync" n=0 iv=#21013593..#21014200 B=#21013464 L=#21013592 nthBn=#21013593 nthHash=e2d459d058aa error="stateRoot mismatch, expect: 0x2db6858070460c80b6c9802f5dffca67bdb0ecfa184b279d71a1ca1ce2ca48aa, got: 0x1159a10bc4fa3d2e13c4d5335aa382c7f9b6b509201c477ede5ab7da07424730"
@mjfh
Copy link
Contributor

mjfh commented Jan 31, 2025 •
edited
Loading

Is there a way to reproduce that? As a side note, there are some important syncer changes missing on this branch. The only changes against the current syncer source code tree that make currently sense to me are with the worker/db.nim file.

There is a potential problem with the error recovery after an import error (when there was an internal FC module reorg) which I will fix.

There is also a quick abort (for this sync scrum) useful when one can assume that the current scrum/sprint will fail at most or all blocks to be imported as it seems the case here. This will be updated when the forked-layers branch is merged and the return code messages are stable. This mechanism is available at other places already.

@arnetheduck
Copy link
Member Author

Is there a way to reproduce that?

Not sure, it happened a few hours after starting my mainnet node from an snapshot db (from November)..

Of course, the fact that the block failed to apply is also an issue to be investigated, but that's a separate problem.

@mjfh
Copy link
Contributor

mjfh commented Jan 31, 2025

What seemed to have happened was that the FC module switched to another canonical branch shortly before the syncer started importing (was the first block of the current batch.)

The assert came from a (reproducible) bookkeeping error when handling repeated import errors.

@arnetheduck
Copy link
Member Author

What seemed to have happened was that the FC module switched to another canonical branch shortly before

to be clear, this is normal behavior - ie FC can switch to a different branch at any time and the syncer must make no assumptions about it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@arnetheduck @mjfh