panic: assertion failed: Page expected to be: 5

[serathius]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

Panic discovered by Antithesis testing

https://linuxfoundation.antithesis.com/report/G-9rIjiZJiwodTEN5avQ7wgK/u_uFsWOwZSxS5mOmbEprwMUijNhsWdV6mfde_CT-y4k.html?auth=v2.public.eyJuYmYiOiIyMDI1LTA3LTAzVDA3OjMxOjUyLjQzMzQ2ODk3NFoiLCJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoidV91RnNXT3daU3hTNW1PbWJFcHJ3TVVpak5oc1dkVjZtZmRlX0NULXk0ay5odG1sIiwicmVwb3J0X2lkIjoiRy05cklqaVpKaXdvZFRFTjVhdlE3d2dLIn19fUHU0wnVRoDtfilwOCROUiDTtcOlIZkrVaddCqjorH3utgcIEPIzlsrMAJGXFC6NTZMneLqAWWU_lq-9prD_tQc

/cc @ahrtr @fuweid

What did you expect to happen?

No panic

How can we reproduce it (as minimally and precisely as possible)?

Contact @marcus-hodgson-antithesis to help reproduce it on Antithesis

Anything else we need to know?

No response

Etcd version (please run commands below)

main branch

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

{"level":"info","ts":"2025-07-02T17:53:12.524581Z","caller":"snap/db.go:65","msg":"saved database snapshot to disk","path":"/var/etcd/data/member/snap/0000000000000051.snap.db","bytes":24576,"size":"25 kB"}
{"level":"info","ts":"2025-07-02T17:53:12.524643Z","caller":"rafthttp/http.go:287","msg":"received and saved database snapshot","local-member-id":"dcb68c82481661be","remote-snapshot-sender-id":"99eab3685d8363a1","incoming-snapshot-index":81,"incoming-snapshot-size-bytes":24576,"incoming-snapshot-size":"25 kB","download-took":"1.602245032s"}
{"level":"info","ts":"2025-07-02T17:53:12.525182Z","logger":"raft","caller":"v3@v3.6.0/raft.go:897","msg":"dcb68c82481661be became follower at term 147"}
{"level":"info","ts":"2025-07-02T17:53:12.525209Z","logger":"raft","caller":"v3@v3.6.0/log.go:465","msg":"log [committed=79, applied=79, applying=79, unstable.offset=80, unstable.offsetInProgress=80, len(unstable.Entries)=0] starts to restore snapshot [index: 81, term: 147]"}
{"level":"info","ts":"2025-07-02T17:53:12.525245Z","logger":"raft","caller":"v3@v3.6.0/raft.go:1981","msg":"dcb68c82481661be switched to configuration voters=(3201069280892892151 11090874293180130209 15904053625288548798)"}
{"level":"info","ts":"2025-07-02T17:53:12.525340Z","logger":"raft","caller":"v3@v3.6.0/raft.go:1935","msg":"dcb68c82481661be [commit: 81, lastindex: 81, lastterm: 147] restored snapshot [index: 81, term: 147]"}
{"level":"info","ts":"2025-07-02T17:53:12.525381Z","logger":"raft","caller":"v3@v3.6.0/raft.go:1844","msg":"dcb68c82481661be [commit: 81] restored snapshot [index: 81, term: 147]"}
{"level":"info","ts":"2025-07-02T17:53:12.525408Z","logger":"raft","caller":"v3@v3.6.0/node.go:370","msg":"raft.node: dcb68c82481661be elected leader 99eab3685d8363a1 at term 147"}
{"level":"info","ts":"2025-07-02T17:53:12.525980Z","caller":"etcdserver/raft.go:276","msg":"applied incoming Raft snapshot","snapshot-index":81}
{"level":"info","ts":"2025-07-02T17:53:12.526036Z","caller":"etcdserver/server.go:1001","msg":"applying snapshot","current-snapshot-index":52,"current-applied-index":79,"incoming-leader-snapshot-index":81,"incoming-leader-snapshot-term":147}
{"level":"info","ts":"2025-07-02T17:53:12.526095Z","logger":"bbolt","caller":"backend/backend.go:203","msg":"Opening db file (/var/etcd/data/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: hashmap, ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 10737418240, PageSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0xc000020d00}"}
{"level":"info","ts":"2025-07-02T17:53:12.526699Z","logger":"bbolt","caller":"runtime/panic.go:792","msg":"Opening bbolt db (/var/etcd/data/member/snap/db) successfully"}
panic: assertion failed: Page expected to be: 5, but self identifies as 0
goroutine 8052 [running]:
go.etcd.io/bbolt/internal/common.Assert(...)
	go.etcd.io/bbolt@v1.4.2/internal/common/verify.go:65
go.etcd.io/bbolt/internal/common.(*Page).FastCheck(0x7f16ce802000, 0x5)
	go.etcd.io/bbolt@v1.4.2/internal/common/page.go:83 +0x1d9
go.etcd.io/bbolt.(*Tx).page(0xc0010a31d0?, 0xc0002992e8?)
	go.etcd.io/bbolt@v1.4.2/tx.go:598 +0x7b
go.etcd.io/bbolt.(*Tx).forEachPageInternal(0xc000c71340, {0xc0010a31d0, 0x1, 0xa}, 0xc0002993a0)
	go.etcd.io/bbolt@v1.4.2/tx.go:610 +0x5a
go.etcd.io/bbolt.(*Tx).forEachPage(...)
	go.etcd.io/bbolt@v1.4.2/tx.go:606
go.etcd.io/bbolt.(*Tx).checkInvariantProperties(0xc000c71340, 0x5, 0xc0001ee808?, 0xc00131e780?, {0x1526270, 0x1e93dc0}, 0xc001329570)
	go.etcd.io/bbolt@v1.4.2/tx_check.go:143 +0xb2
go.etcd.io/bbolt.(*Tx).recursivelyCheckBucket(0xc000c71340, 0xc000c71358, 0xc000299640, 0xc000299588, {0x1526270, 0x1e93dc0}, 0xc001329570)
	go.etcd.io/bbolt@v1.4.2/tx_check.go:130 +0x73
go.etcd.io/bbolt.(*DB).freepages(0xc00048cfc8)
	go.etcd.io/bbolt@v1.4.2/db.go:1251 +0x21b
go.etcd.io/bbolt.(*DB).loadFreelist.func1()
	go.etcd.io/bbolt@v1.4.2/db.go:422 +0xc9
sync.(*Once).doSlow(0xc000299708?, 0xc00048cfc8?)
	sync/once.go:78 +0xab
sync.(*Once).Do(...)
	sync/once.go:69
go.etcd.io/bbolt.(*DB).loadFreelist(0xc00048cfc8?)
	go.etcd.io/bbolt@v1.4.2/db.go:418 +0x3b
go.etcd.io/bbolt.Open({0xc0011f2600, 0x1d}, 0x180, 0xc00136a070)
	go.etcd.io/bbolt@v1.4.2/db.go:299 +0xb48
go.etcd.io/etcd/server/v3/storage/backend.newBackend({{0xc0011f2600, 0x1d}, 0x5f5e100, 0x2710, {0x12d2af4, 0x7}, 0x280000000, 0xc000213780, 0x0, 0x0, ...})
	server/storage/backend/backend.go:203 +0x21b
go.etcd.io/etcd/server/v3/storage/backend.New({{0xc0011f2600, 0x1d}, 0x5f5e100, 0x2710, {0x12d2af4, 0x7}, 0x280000000, 0xc000213780, 0x0, 0x0, ...})
	server/storage/backend/backend.go:167 +0x45
go.etcd.io/etcd/server/v3/storage.newBackend({{0xc00001203a, 0x5}, {{{0x1e93dc0, 0x0, 0x0}, 0x12a05f200, 0x77359400, 0x77359400, 0x165a0bc00, 0x0, ...}, ...}, ...}, ...)
	server/storage/backend.go:55 +0x478
go.etcd.io/etcd/server/v3/storage.OpenBackend.func1()
	server/storage/backend.go:76 +0x8b
created by go.etcd.io/etcd/server/v3/storage.OpenBackend in goroutine 171
	server/storage/backend.go:75 +0x16f

[ahrtr]

The db file (received snapshot) is somehow corrupted.

Can this be easily be reproduced? (I don't think so)

If the data was somehow corrupted/modified during the transportation, it might also cause issue. But if TLS is enabled, it's unlikely.

[serathius]

Can this be easily be reproduced?

Issue is was found on Antithesis platform which uses deterministic simulation testing. This means that it can reproduce it and with help of @marcus-hodgson-antithesis you can get access to debugger at moment of issue happening.

Previous reports:

• etcd panic: assertion failed: Page expected to be: 36312, but self identifies as 0 #18096
• static etcd container failed to start in kubeadm based k8s deployment #17772
• Panic happens when opening a boltdb bbolt#705
• Panic due to failed assertion bbolt#1011
• DB cannot be opened after node hard reset bbolt#562

[marcus-hodgson-antithesis]

@ahrtr Antithesis has the power to reproduce any issue previously found. Since our system is fully deterministic, we can replay the series of inputs that led to the panic to setup a system that is essentially frozen in time at the point of the panic.

Our product offers the multiverse debugger where you can rewind time / inject bash commands into specific containers / read & extract files at any point in time.

If you're interested, I can provide more information, but the docs do a great job at explaining what it's capable of. Pretty much anything you can think of!

[ahrtr]

@ahrtr Antithesis has the power to reproduce any issue previously found.

Can you convert these into steps which others can reproduce manually without Antithesis? If it isn't possible, please elaborate/explain the steps.

[fuweid]

Hi @marcus-hodgson-antithesis during runtime, is there any customized filesystem for etcd container? or just overlayfs for it.

There is about log [ 936.128] [ etcd2] [err] {"level":"info","ts":"2025-07-02T17:53:12.524581Z","caller":"snap/db.go:65","msg":"saved database snapshot to disk","path":"/var/etcd/data/member/snap/0000000000000051.snap.db","bytes":24576,"size":"25 kB"}

Is there any networking failpoint between etcd1 -> etcd2? if there is no filesystem level failpoint, it seems like there is something wrong for that snapshot. The networking failpoint could make more senses here. I can try to use Antithesis to reproduce it, if there is any available instruction.

[       935.963] [               etcd2] [err] {"level":"info","ts":"2025-07-02T17:53:12.360031Z","logger":"raft","caller":"v3@v3.6.0/raft.go:1175","msg":"dcb68c82481661be [term: 147] ignored a MsgAppResp message with lower term from 99eab3685d8363a1 [term: 146]"}
[       936.128] [               etcd2] [err] {"level":"info","ts":"2025-07-02T17:53:12.524581Z","caller":"snap/db.go:65","msg":"saved database snapshot to disk","path":"/var/etcd/data/member/snap/0000000000000051.snap.db","bytes":24576,"size":"25 kB"}
[       936.128] [               etcd2] [err] {"level":"info","ts":"2025-07-02T17:53:12.524643Z","caller":"rafthttp/http.go:287","msg":"received and saved database snapshot","local-member-id":"dcb68c82481661be","remote-snapshot-sender-id":"99eab3685d8363a1","incoming-snapshot-index":81,"incoming-snapshot-size-bytes":24576,"incoming-snapshot-size":"25 kB","download-took":"1.602245032s"}
[       936.128] [               etcd2] [err] {"level":"info","ts":"2025-07-02T17:53:12.525182Z","logger":"raft","caller":"v3@v3.6.0/raft.go:897","msg":"dcb68c82481661be became follower at term 147"}
[       936.128] [               etcd2] [err] {"level":"info","ts":"2025-07-02T17:53:12.525209Z","logger":"raft","caller":"v3@v3.6.0/log.go:465","msg":"log [committed=79, applied=79, applying=79, unstable.offset=80, unstable.offsetInProgress=80, len(unstable.Entries)=0] starts to restore snapshot [index: 81, term: 147]"}
[       936.128] [               etcd2] [err] {"level":"info","ts":"2025-07-02T17:53:12.525245Z","logger":"raft","caller":"v3@v3.6.0/raft.go:1981","msg":"dcb68c82481661be switched to configuration voters=(3201069280892892151 11090874293180130209 15904053625288548798)"}
[       936.129] [               etcd2] [err] {"level":"info","ts":"2025-07-02T17:53:12.525340Z","logger":"raft","caller":"v3@v3.6.0/raft.go:1935","msg":"dcb68c82481661be [commit: 81, lastindex: 81, lastterm: 147] restored snapshot [index: 81, term: 147]"}
[       936.129] [               etcd2] [err] {"level":"info","ts":"2025-07-02T17:53:12.525381Z","logger":"raft","caller":"v3@v3.6.0/raft.go:1844","msg":"dcb68c82481661be [commit: 81] restored snapshot [index: 81, term: 147]"}
[       936.129] [               etcd2] [err] {"level":"info","ts":"2025-07-02T17:53:12.525408Z","logger":"raft","caller":"v3@v3.6.0/node.go:370","msg":"raft.node: dcb68c82481661be elected leader 99eab3685d8363a1 at term 147"}
[       936.129] [               etcd2] [err] {"level":"info","ts":"2025-07-02T17:53:12.525980Z","caller":"etcdserver/raft.go:276","msg":"applied incoming Raft snapshot","snapshot-index":81}
[       936.129] [               etcd2] [err] {"level":"info","ts":"2025-07-02T17:53:12.526036Z","caller":"etcdserver/server.go:1001","msg":"applying snapshot","current-snapshot-index":52,"current-applied-index":79,"incoming-leader-snapshot-index":81,"incoming-leader-snapshot-term":147}
[       936.129] [               etcd2] [err] {"level":"info","ts":"2025-07-02T17:53:12.526095Z","logger":"bbolt","caller":"backend/backend.go:203","msg":"Opening db file (/var/etcd/data/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: hashmap, ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 10737418240, PageSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0xc000020d00}"}
[       936.130] [               etcd2] [err] {"level":"info","ts":"2025-07-02T17:53:12.526699Z","logger":"bbolt","caller":"runtime/panic.go:792","msg":"Opening bbolt db (/var/etcd/data/member/snap/db) successfully"}
[       936.133] [               etcd2] [err] panic: assertion failed: Page expected to be: 5, but self identifies as 0
[       936.133] [               etcd2] [err] 
[       936.133] [               etcd2] [err] goroutine 8052 [running]:
[       936.133] [               etcd2] [err] go.etcd.io/bbolt/internal/common.Assert(...)
[       936.133] [               etcd2] [err] 	go.etcd.io/bbolt@v1.4.2/internal/common/verify.go:65
[       936.133] [               etcd2] [err] go.etcd.io/bbolt/internal/common.(*Page).FastCheck(0x7f16ce802000, 0x5)
[       936.133] [               etcd2] [err] 	go.etcd.io/bbolt@v1.4.2/internal/common/page.go:83 +0x1d9
[       936.133] [               etcd2] [err] go.etcd.io/bbolt.(*Tx).page(0xc0010a31d0?, 0xc0002992e8?)
[       936.133] [               etcd2] [err] 	go.etcd.io/bbolt@v1.4.2/tx.go:598 +0x7b
[       936.133] [               etcd2] [err] go.etcd.io/bbolt.(*Tx).forEachPageInternal(0xc000c71340, {0xc0010a31d0, 0x1, 0xa}, 0xc0002993a0)
[       936.133] [               etcd2] [err] 	go.etcd.io/bbolt@v1.4.2/tx.go:610 +0x5a
[       936.133] [               etcd2] [err] go.etcd.io/bbolt.(*Tx).forEachPage(...)
[       936.133] [               etcd2] [err] 	go.etcd.io/bbolt@v1.4.2/tx.go:606
[       936.133] [               etcd2] [err] go.etcd.io/bbolt.(*Tx).checkInvariantProperties(0xc000c71340, 0x5, 0xc0001ee808?, 0xc00131e780?, {0x1526270, 0x1e93dc0}, 0xc001329570)
[       936.133] [               etcd2] [err] 	go.etcd.io/bbolt@v1.4.2/tx_check.go:143 +0xb2
[       936.133] [               etcd2] [err] go.etcd.io/bbolt.(*Tx).recursivelyCheckBucket(0xc000c71340, 0xc000c71358, 0xc000299640, 0xc000299588, {0x1526270, 0x1e93dc0}, 0xc001329570)
[       936.133] [               etcd2] [err] 	go.etcd.io/bbolt@v1.4.2/tx_check.go:130 +0x73
[       936.133] [               etcd2] [err] go.etcd.io/bbolt.(*DB).freepages(0xc00048cfc8)
[       936.133] [               etcd2] [err] 	go.etcd.io/bbolt@v1.4.2/db.go:1251 +0x21b
[       936.133] [               etcd2] [err] go.etcd.io/bbolt.(*DB).loadFreelist.func1()
[       936.133] [               etcd2] [err] 	go.etcd.io/bbolt@v1.4.2/db.go:422 +0xc9
[       936.133] [               etcd2] [err] sync.(*Once).doSlow(0xc000299708?, 0xc00048cfc8?)
[       936.133] [               etcd2] [err] 	sync/once.go:78 +0xab
[       936.133] [               etcd2] [err] sync.(*Once).Do(...)
[       936.133] [               etcd2] [err] 	sync/once.go:69
[       936.133] [               etcd2] [err] go.etcd.io/bbolt.(*DB).loadFreelist(0xc00048cfc8?)
[       936.133] [               etcd2] [err] 	go.etcd.io/bbolt@v1.4.2/db.go:418 +0x3b
[       936.133] [               etcd2] [err] go.etcd.io/bbolt.Open({0xc0011f2600, 0x1d}, 0x180, 0xc00136a070)
[       936.133] [               etcd2] [err] 	go.etcd.io/bbolt@v1.4.2/db.go:299 +0xb48
[       936.133] [               etcd2] [err] go.etcd.io/etcd/server/v3/storage/backend.newBackend({{0xc0011f2600, 0x1d}, 0x5f5e100, 0x2710, {0x12d2af4, 0x7}, 0x280000000, 0xc000213780, 0x0, 0x0, ...})
[       936.133] [               etcd2] [err] 	server/storage/backend/backend.go:203 +0x21b
[       936.133] [               etcd2] [err] go.etcd.io/etcd/server/v3/storage/backend.New({{0xc0011f2600, 0x1d}, 0x5f5e100, 0x2710, {0x12d2af4, 0x7}, 0x280000000, 0xc000213780, 0x0, 0x0, ...})
[       936.133] [               etcd2] [err] 	server/storage/backend/backend.go:167 +0x45
[       936.133] [               etcd2] [err] go.etcd.io/etcd/server/v3/storage.newBackend({{0xc00001203a, 0x5}, {{{0x1e93dc0, 0x0, 0x0}, 0x12a05f200, 0x77359400, 0x77359400, 0x165a0bc00, 0x0, ...}, ...}, ...}, ...)
[       936.133] [               etcd2] [err] 	server/storage/backend.go:55 +0x478
[       936.133] [               etcd2] [err] go.etcd.io/etcd/server/v3/storage.OpenBackend.func1()
[       936.133] [               etcd2] [err] 	server/storage/backend.go:76 +0x8b
[       936.133] [               etcd2] [err] created by go.etcd.io/etcd/server/v3/storage.OpenBackend in goroutine 171
[       936.133] [               etcd2] [err] 	server/storage/backend.go:75 +0x16f
[       936.139] [               etcd0] [err] {"level":"warn","ts":"2025-07-02T17:53:12.535732Z","caller":"rafthttp/stream.go:421","msg":"lost TCP stream

[marcus-hodgson-antithesis]

@fuweid By default, we use tmpfs to create a vm filesystem for the containers. I don't think we use overlayfs for this.

We currently do not have any direct filesystem failpoints for the containers, but we do have things like CPU maxing faults for specific containers that would simulate overload.

I've created a log viewer for this panic that has unfiltered logs. We usually don't expose these since they can become messy, but for the purposes of seeing faults that are happening, it might be useful. Here, we can see that right before the panic from etcd2, there was a CPU limit fault by 50% for etcd2. Could this information help figure out the root cause?

I could also run a debug report if you'd like?

[ahrtr]

I feel like this is an environment issue. Some pages might not be synced to disk successfully even the fdatasync system call returns success. We can double confirm this by checking the snapshot directly using bbolt command. If all data of a page is 0, then most likely the page doesn't get flushed /synced successfully.

Also is TLS enabled in the test?

etcd2] [err] {"level":"info","ts":"2025-07-02T17:53:12.526036Z","caller":"etcdserver/server.go:1001","msg":"applying snapshot","current-snapshot-index":52,"current-applied-index":79,"incoming-leader-snapshot-index":81,"incoming-leader-snapshot-term":147}

• The index is 81, but the term is 147. It means the network is super terrible (or intentionally injected too many failpoints) in the test environment.
• The current member's applied-index is 79, but it still received a snapshot with index 81. Did you set the --snapshot-catchup-entries to 1?

Anyway, @marcus-hodgson-antithesis can you please provide a summary on the rough test steps?

[marcus-hodgson-antithesis]

@ahrtr Antithesis neither disabled nor enables TLS connections. I'd have to defer to @serathius on what the connection type would be between the servers based on the docker-compose.

If there is a bash command using bbolt to confirm that, would you mind sending it to me? I can run it in etcd2 at a point just before the panic.

Apologies, but would you mind expanding a bit on the last question? Are you looking to kick off a test of your own? Are you looking for more information on the test setup that we are running with currently?

[serathius]

The current member's applied-index is 79, but it still received a snapshot with index 81. Did you set the --snapshot-catchup-entries to 1?

It's lowered to 100, same like in robustness tests ETCD_SNAPSHOT_CATCHUP_ENTRIES: 100.

[ahrtr]

Apologies, but would you mind expanding a bit on the last question? Are you looking to kick off a test of your own? Are you looking for more information on the test setup that we are running with currently?

Thanks for helping testing etcd. The problem is that there is some knowledge gap. Please don't assume everyone (including me) know how Antithesis works. Please try to explain the overall steps & environment & config, so that the community can understand more about the situation, and provide feedback and how to debug further.

I’ve seen more than 20 data corruption issues reported over the past 4 years. But I’ve never been able to reproduce any of them myself, and none of the reporters have been able to reproduce them easily either. So please try to summary the test as mentioned above, and we will help to analyze the issue.

If there is a bash command using bbolt to confirm that, would you mind sending it to me? I can run it in etcd2 at a point just before the panic.

You will have to do it after the panic, otherwise you don't know which page to look into. The command to check the page is as below. You can generate a binary by executing go build ./cmd/bbolt at the root directory of https://github.com/etcd-io/bbolt.

$ bbolt dump ~/tmp/etcd/default.etcd/member/snap/db  6
0018000 0000 0000 0000 0000 0000 0000 0000 0000
0018010 *
001bff0 0000 0000 0000 0000 0000 0000 0000 0000

It's lowered to 100, same like in robustness tests ETCD_SNAPSHOT_CATCHUP_ENTRIES: 100.

thx. But it's strange. Clearly 79 + 100 > 81.

[serathius]

Please don't assume everyone (including me) know how Antithesis works.

Sorry, my goal of filing this issue was to encourage collaboration with Antithesis to debug the issue, not cause confusion. The problem is that the only way to reproduce the issue is to use the debugger from Antithesis. Unfortunately at this moment only people from Antithesis have access as their platform doesn't support Github Auth.

The only way to do that would be to jump on a call with @marcus-hodgson-antithesis.

[ahrtr]

my goal of filing this issue was to encourage collaboration with Antithesis to debug the issue

Right, it definitely needs collaboration between Antithesis and the community. I'd like to have a summary of the test from Antithesis first. Afterwards, I will try to provide feedback and thought how to debug this further.

The only way to do that would be to jump on a call with @marcus-hodgson-antithesis.

I am OK to jump into a call. Probably we can extend our next community meeting for around 30 minutes, so 1 hour in total. If we have less topic to discuss, then we will have more time dedicated to this issue. But before that, I think it's still better to have a summary from Antithesis (and probably several back and forth discussion using github comments or slack messages) first, so that we will have more efficient discussion in the zoom meeting.

[serathius]

But before that, I think it's still better to have a summary from Antithesis

I agree, however they are not very familiar with etcd and we might not have complete knowledge how the faults injected are implemented we don't have access to engineering.

@marcus-hodgson-antithesis can you provide all the context you have for this particular panic?

[marcus-hodgson-antithesis]

@ahrtr of course!

We (Antithesis) provide an endpoint that you can call with a cURL command. This endpoint kicks off a test with parameters you are able to specify. Marek has built out a github action for this in .github/workflows

Some of these parameters specify "how long should we run this test for", "what images should I use to start my containers in this test", "what is the docker-compose I should use to connect these images together", etc.

So linking that to our testing with etcd, we've been spinning up a 3-node etcd server cluster and "a client" image. The "client" image contains the robustness tests, which are run and connect to the cluster during the test.

The test itself has basically 2 phases. (1) The setup phase brings up the containers, connects them over a network, and the client makes sure the nodes are reachable / available before moving on. (2) The exploration phase is where the work really happens. This is where the majority of the time is spent, where Antithesis injects faults, and the integrated robustness tests are running.

We are able to know what happened from the exploration test with the use of Antithesis SDK assertions & global properties. @serathius has defined these assertions in code, and an example would be "notify me if the watch validation ever fails". The global properties would say something like "did we ever see a panic". And these are reported on the triage reports that are sent back after the completion of the test.

Since our Antithesis environment is deterministic, we are able to replay any event. This means spinning up those same containers and following the instructions that led to the event of interest. We end up with the system state of the event, frozen in time. And we can do all sorts of things to debug. Our docs have great explanations for this