Validator nil deref, missing proposer key

[anhductn2001]

It happen after seq B come and rollbacked (remove /data folder) (No seq coming testcase)

2024-11-19 11:32:41 time="Nov 19 04:32:41.042" level=info msg="rollapp left with no proposer. chain is halted[]"\
2024-11-19 11:32:41 time="Nov 19 04:32:41.042" level=info msg="Synced from DA[store height 242 target height 242]" module=block_manager\
2024-11-19 11:32:41 time="Nov 19 04:32:41.042" level=debug msg="disregarding new state update, node is still validating[]" module=block_manager\
2024-11-19 11:32:41 time="Nov 19 04:32:41.042" level=info msg="Synced.[current height 242 last submitted height 242]" module=block_manager\
2024-11-19 11:32:41 time="Nov 19 04:32:41.043" level=debug msg="indexed block[height 242]" module=txindex\
2024-11-19 11:32:41 time="Nov 19 04:32:41.043" level=debug msg="indexed block txs[height 242 num_txs 0]" module=txindex\
2024-11-19 11:32:41 time="Nov 19 04:32:41.044" level=debug msg="Added bytes produced to bytes pending submission counter.[bytes added 0 pending 0]" module=block_manager\
2024-11-19 11:32:41 time="Nov 19 04:32:41.045" level=debug msg="grpc client assigned to client context[target 127.0.0.1:9090]"\
2024-11-19 11:32:41 time="Nov 19 04:32:41.045" level=info msg="Started block producer loop.[]" module=block_manager\
2024-11-19 11:32:41 time="Nov 19 04:32:41.075" level=info msg="starting API server...[]" module=api-server\
2024-11-19 11:32:41 time="Nov 19 04:32:41.075" level=info msg="serve[msg Starting RPC HTTP server on [::]:1317]" module=api-server\
2024-11-19 11:32:41 time="Nov 19 04:32:41.474" level=debug msg="Publishing internal event.[event NewBatchAccepted data EndHeight: 244, StateIndex: 16]" module=settlement_client\
2024-11-19 11:32:50 time="Nov 19 04:32:50.499" level=debug msg="ABCIQuery[path /ibc.core.channel.v1.Query/ChannelClientState height 242]" module=events\
2024-11-19 11:32:50 time="Nov 19 04:32:50.508" level=debug msg="ABCIQuery[path /ibc.core.channel.v1.Query/ChannelConsensusState height 242]" module=events\
2024-11-19 11:32:50 time="Nov 19 04:32:50.512" level=debug msg="ABCIQuery[path /cosmos.auth.v1beta1.Query/Account height 242]" module=events\
2024-11-19 11:32:50 time="Nov 19 04:32:50.513" level=debug msg="ABCIQuery[path /cosmos.auth.v1beta1.Query/Account height 242]" module=events\
2024-11-19 11:32:50 time="Nov 19 04:32:50.554" level=info msg="packet sent[sequence 3 src_port transfer src_channel channel-0 dst_port transfer dst_channel channel-0]" module=x/ibc/channel\
2024-11-19 11:32:50 time="Nov 19 04:32:50.554" level=info msg="IBC fungible token transfer[token urax amount 1000000 sender ethm18y5jx06c3z5lu3x0z4csd200kls2qz639y2knh receiver dym1un7ate426tmuzc6wj5gg4wljt28fem2c2mtjlu]" module=x/ibc-transfer\
2024-11-19 11:32:50 time="Nov 19 04:32:50.558" level=debug msg="ABCIQuery[path /cosmos.tx.v1beta1.Service/Simulate height 242]" module=events\
2024-11-19 11:32:50 time="Nov 19 04:32:50.620" level=debug msg="inserted new valid transaction[priority 0 tx 5133C948B340CC21E599A3552A765C9B23A6FF08847958E16890762C423957FA height 242 num_txs 1]"\
2024-11-19 11:32:50 time="Nov 19 04:32:50.621" level=debug msg="Gossiping transaction.[len 362]" module=p2p\
2024-11-19 11:32:54 time="Nov 19 04:32:54.863" level=info msg="Freezing node[err rollapp fork detected. please rollback to height previous to rollapp_revision_start_height.  local_block_height: 242 rollapp_revision_start_height: 243 local_revision: 0 rollapp_revision: 1]" module=block_manager\
2024-11-19 11:32:54 time="Nov 19 04:32:54.863" level=error msg="ErrGroup goroutine.[err context canceled]" module=block_manager\
2024-11-19 11:32:54 time="Nov 19 04:32:54.864" level=info msg="Freezing node[err context canceled]" module=block_manager\
2024-11-19 11:32:54 time="Nov 19 04:32:54.864" level=error msg="node is unhealthy[error rollapp fork detected. please rollback to height previous to rollapp_revision_start_height.  local_block_height: 242 rollapp_revision_start_height: 243 local_revision: 0 rollapp_revision: 1]"\
2024-11-19 11:32:54 time="Nov 19 04:32:54.864" level=info msg="Stopped block producer loop.[]" module=block_manager\
2024-11-19 11:32:54 time="Nov 19 04:32:54.864" level=error msg="ErrGroup goroutine.[err context canceled]" module=block_manager\
2024-11-19 11:32:54 time="Nov 19 04:32:54.864" level=error msg="ErrGroup goroutine.[err context canceled]" module=block_manager\
2024-11-19 11:32:54 time="Nov 19 04:32:54.864" level=error msg="ErrGroup goroutine.[err context canceled]" module=block_manager\
2024-11-19 11:32:56 time="Nov 19 04:32:56.123" level=info msg="Starting JSON-RPC server[address 0.0.0.0:8545]"\
2024-11-19 11:33:01 time="Nov 19 04:33:01.150" level=info msg="Starting JSON WebSocket server[address 0.0.0.0:8546]"\
2024-11-19 11:33:10 panic: runtime error: invalid memory address or nil pointer dereference\
2024-11-19 11:33:10 [signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x1667a0e]\
2024-11-19 11:33:10 \
2024-11-19 11:33:10 goroutine 3783 [running]:\
2024-11-19 11:33:10 github.com/dymensionxyz/dymint/types.(*Commit).ValidateWithHeader(0xc001f0c240, \{0x0, 0x0\}, 0xc001f0c008)\
2024-11-19 11:33:10     /go/pkg/mod/github.com/dymensionxyz/[email protected]/types/validation.go:136 +0x1ce\
2024-11-19 11:33:10 github.com/dymensionxyz/dymint/p2p.(*BlockData).Validate(0xc001f0c008, \{0x0, 0x0\})\
2024-11-19 11:33:10     /go/pkg/mod/github.com/dymensionxyz/[email protected]/p2p/block.go:62 +0x4b\

Fulllog:

seqA after.txt
seqB after.txt

[danwt]

Thanks @anhductn2001 . Any idea which commit this was at? (either rollapp or dymint).
There were some commits to this file on the 19th Nov.

It was probably this line and commit

dymint/types/validation.go

Line 136 in 2bb17cc

if !proposerPubKey.VerifySignature(abciHeaderBytes, c.Signatures[0]) {

Probably a nil deref on proposerPubKey from trying to validate extremely soon after starting the node

[anhductn2001]

Yea, Maybe I should add commit information next time, I think it will be commits close to the issue creation time as you predicted. @danwt