[tidb-x] After killing a tikv for some time, a tiflash wn repeatedly crashes

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、run tpch
2、kill one of tikv
3、after some time，a tiflash wn repeatedly crashes

2. What did you expect to see? (Required)

tiflash run normally

3. What did you see instead (Required)

After killing a tikv for some time, a tiflash wn repeatedly crashes
{"log":"2026-01-20T14:08:12.44876661+08:00 stderr F [2026/01/20 14:08:12.448 +08:00] [FATAL] [lib.rs:556] ["called Result::unwrap() on an Err value: Other(\"[components/kvengine/src/shard.rs:438]: errors is not empty: [DfsError(NoSuchKey(\\\"file 463686552158470393.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686552158470394.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686552158470389.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686667396710805.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686552158470387.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686705433804809.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686627747430465.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686552000921655.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686590510137720.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686552158470390.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686552158470388.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686552158470395.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686552158470391.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686552158470392.sst not exist,\\\"))]\")"] [backtrace=" 0: tikv_util::set_panic_hook::{{closure}}\n 1: <alloc::boxed::Box<F,A> as core::ops::function::Fn>::call\n at rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/alloc/src/boxed.rs:2029:9\n std::panicking::rust_panic_with_hook\n at rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panicking.rs:783:13\n 2: std::panicking::begin_panic_handler::{{closure}}\n at rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panicking.rs:657:13\n 3: std::sys_common::backtrace::__rust_end_short_backtrace\n at rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/sys_common/backtrace.rs:171:18\n 4: rust_begin_unwind\n at rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panicking.rs:645:5\n 5: core::panicking::panic_fmt\n at rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/core/src/panicking.rs:72:14\n 6: core::result::unwrap_failed\n at rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/core/src/result.rs:1649:5\n 7: proxy_ffi::cloud_helper::CloudHelper::get_shard\n 8: proxy_ffi::cloud_helper::CloudHelper::make_sst_reader\n 9: proxy_ffi::sst_reader_impls::ffi_make_sst_reader\n 10: _ZN2DB13MonoSSTReaderC1EPKNS_22TiFlashRaftProxyHelperENS_7SSTViewENSt3__110shared_ptrIKNS_15RegionRangeKeysEEERKNS6_INS_6LoggerEEE\n at workspace/source/tiflash/dbms/src/Storages/KVStore/FFI/SSTReader.cpp:114:13\n 11: _ZNSt3__110__function6__funcIZN2DB2DM26SSTFilesToBlockInputStreamC1ENS_10shared_ptrINS2_6RegionEEEmRKNS2_10SSTViewVecEPKNS2_22TiFlashRaftProxyHelperERNS2_10TMTContextEONS_8optionalINS3_16SSTScanSoftLimitEEENS5_INS2_16PreHandlingTrace4ItemEEEONS3_30SSTFilesToBlockInputStreamOptsEE3$0NS_9allocatorISP_EEFNS_10unique_ptrINS2_13MonoSSTReaderENS_14default_deleteIST_EEEESD_NS2_7SSTViewENS5_IKNS2_15RegionRangeKeysEEERKNS5_INS2_6LoggerEEEEEclEOSD_OSX_OS10_S14.387a1388c4b5c8fe1da21e49667395e6\n at usr/local/bin/../include/c++/v1/__memory/unique_ptr.h:689:30\n 12: _ZNSt3__111make_uniqueB8ue170006IN2DB14MultiSSTReaderINS1_13MonoSSTReaderENS1_7SSTViewEEEJRPKNS1_22TiFlashRaftProxyHelperENS1_16ColumnFamilyTypeERZNS1_2DM26SSTFilesToBlockInputStreamC1ENS_10shared_ptrINS1_6RegionEEEmRKNS1_10SSTViewVecES8_RNS1_10TMTContextEONS_8optionalINSB_16SSTScanSoftLimitEEENSD_INS1_16PreHandlingTrace4ItemEEEONSB_30SSTFilesToBlockInputStreamOptsEE3$0RNS_6vectorIS4_NS_9allocatorIS4_EEEERNSD_INS1_6LoggerEEENSD_IKNS1_15RegionRangeKeysEEEEEENS_11__unique_ifIT_E15__unique_singleEDpOT0\n at usr/local/bin/../include/c++/v1/__functional/function.h:517:16\n 13: _ZN2DB2DM26SSTFilesToBlockInputStreamC1ENSt3__110shared_ptrINS_6RegionEEEmRKNS_10SSTViewVecEPKNS_22TiFlashRaftProxyHelperERNS_10TMTContextEONS2_8optionalINS0_16SSTScanSoftLimitEEENS3_INS_16PreHandlingTrace4ItemEEEONS0_30SSTFilesToBlockInputStreamOptsE\n at workspace/source/tiflash/dbms/src/Storages/DeltaMerge/Decode/SSTFilesToBlockInputStream.cpp:100:27\n 14: _ZN2DB7KVStore22preHandleSSTsToDTFilesENSt3__110shared_ptrINS_6RegionEEENS_10SSTViewVecEmmNS_2DM18FileConvertJobTypeERNS_10TMTContextE\n at usr/local/bin/../include/c++/v1/__memory/construct_at.h:41:46\n 15: _ZN2DB7KVStore24preHandleSnapshotToFilesENSt3__110shared_ptrINS_6RegionEEENS_10SSTViewVecEmmNS1_8optionalImEERNS_10TMTContextE\n at workspace/source/tiflash/dbms/src/Storages/KVStore/MultiRaft/PrehandleSnapshot.cpp:292:34\n 16: PreHandleSnapshot\n at workspace/source/tiflash/dbms/src/Storages/KVStore/FFI/ProxyFFI.cpp:697:42\n 17: proxy_ffi::engine_store_helper_impls::::pre_handle_snapshot\n 18: engine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl\n 19: yatp::task::future::RawTask::poll\n 20: <yatp::task::future::Runner as yatp::pool::runner::Runner>::handle\n 21: std::sys_common::backtrace::__rust_begin_short_backtrace\n 22: core::ops::function::FnOnce::call_once{{vtable.shim}}\n 23: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once\n at rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/alloc/src/boxed.rs:2015:9\n <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once\n at rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/alloc/src/boxed.rs:2015:9\n std::sys::unix::thread::Thread::new::thread_start\n at rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/sys/unix/thread.rs:108:17\n 24: start_thread\n 25: __GI___clone\n"] [location=proxy_components/proxy_ffi/src/cloud_helper.rs:269] [thread_name=region-task-3]","container":"tiflash","namespace":"ha-test-tiflash-tps-8047826-1-764","pod":"tc-tiflash-0"}

4. What is your TiFlash version? (Required)

TiFlash
Release Version: v8.5.4-nextgen.202510.8
Edition: Enterprise
Git Commit Hash: b23fae4
Git Branch: HEAD
UTC Build Time: 2026-01-13 01:04:19
Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto next-gen hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake
Profile: RELWITHDEBINFO
Compiler: clang++ 17.0.6
Raft Proxy
Git Commit Hash: 2505f2f8d3061d8e61aa6f4ff4b91ade95a50785
Git Commit Branch: HEAD
UTC Build Time: ""
Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27)
Storage Engine: tiflash
Prometheus Prefix: tiflash_proxy_
Profile: release
Enable Features: "raftstore-proxy/external-jemalloc" portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure
2026-01-19T16:03:30.184+0800

[Lily2025]

/severity major

[JaySon-Huang]

This issue is caused by tiflash received a region snapshot but the sst inside it has already been removed from cloud storage.

## Error cause (current)
The first fatal error from `tc-tiflash-0` shows TiFlash crashing because it attempted to read SSTs that were missing in the backing DFS/S3 (NoSuchKey). This indicates snapshot pre-handle failed due to missing SST objects.

[2026/01/19 20:11:24.988 +08:00] [FATAL] [lib.rs:556] ["called `Result::unwrap()` on an `Err` value: Other(\"[components/kvengine/src/shard.rs:438]: errors is not empty: [DfsError(NoSuchKey(\\\"file 463683283471565174.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463683339098521612.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463683450365542456.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463683392379289601.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463683283471565173.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463683555039903745.sst not exist,\\\"))]\")"] [backtrace="..."] [location=proxy_components/proxy_ffi/src/cloud_helper.rs:269] [thread_name=region-task-3]


## TiKV shard/region mapping for 463683283471565174.sst
- tc-tikv-0: shard_id 1705, region `[1223:1705:196]` (compaction change set includes `463683283471565174`).
- tc-tikv-1: shard_id 1705, region `[1296:1705:196]` (compaction change set includes `463683283471565174`).
- tc-tikv-2: shard_id 1705, region `[1001:1705:196]` (compaction change set includes `463683283471565174`).

## Timeline (key events)
- 2026/01/19 16:29:13.840 +08:00: `tc-tikv-1` reads `463683283471565174.sst` (region `[1296:1705:196]`).
- 2026/01/19 16:29:14.122 +08:00: `tc-tikv-2` reads `463683283471565174.sst` (region `[1001:1705:196]`).
- 2026/01/19 16:29:14.226 +08:00: `tc-tikv-0` reads `463683283471565174.sst` (region `[1223:1705:196]`).
- 2026/01/19 18:32:59.993 +08:00: `tc-tikv-1` starts repeated `handle raft message err` (MsgUnreachable); last seen at 18:34:04.474 +08:00, lasting ~64.5s.
- 2026/01/19 18:34:04.420 +08:00: HA test starts killing `tc-tikv-1`.
- 2026/01/19 18:34:07.490 +08:00: `tc-tikv-1` restarts (`Welcome to TiKV`), with ~3s log gap after the last error.
- 2026/01/19 18:37:37.843 +08:00: `tc-tikv-0` compaction marks `463683283471565174.sst` in `bottom_deletes` (region `[1223:1705:196]`).
- 2026/01/19 18:37:38.372 +08:00: `tc-tikv-2` compaction marks `463683283471565174.sst` in `bottom_deletes` (region `[1001:1705:196]`).
- 2026/01/19 18:37:38.375 +08:00: `tc-tikv-1` compaction marks `463683283471565174.sst` in `bottom_deletes` (region `[1296:1705:196]`).
- 2026/01/19 18:39:04.422 +08:00: HA test ends kill (cleanup).
- 2026/01/19 10:43:52.687 UTC+0 (equiv 2026/01/19 18:43:52.687 +08:00): s3clean removed `463683283471565174.sst`.
- 2026/01/19 11:54:23.988 UTC+0 (equiv 2026/01/19 19:54:23.988 +08:00): s3clean permanently removed the same SST (removed at 2026-01-19 10:43:52.645 UTC).
- 2026/01/19 20:11:16.327 +08:00: TiFlash pre-apply snapshot for `region_id=1705` (closest 1705-related log before FATAL).
- 2026/01/19 20:11:24.988 +08:00: First TiFlash FATAL due to missing SSTs (including `463683283471565174.sst`).
- 2026/01/19 20:13:39.218 +08:00: TiFlash reports `region_id=1705` applying snapshot (post-FATAL).