Failed ASan Test: UniversalPageStorageServiceCheckpointTest.DumpFail

[solotzg]

Bug Report

1. Minimal reproduce step (Required)

https://ci.pingcap.net/blue/organizations/jenkins/tiflash-sanitizer-daily/detail/tiflash-sanitizer-daily/3183/pipeline/

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiFlash version? (Required)

master

[JaySon-Huang]

Only test logic affects

2026-02-05T19:29:42.907Z] �[0;33mNote: Google Test filter = UniversalPageStorageServiceCheckpointTest.DumpFail
[2026-02-05T19:29:42.907Z] �[m�[0;32m[==========] �[mRunning 1 test from 1 test case.
[2026-02-05T19:29:42.907Z] �[0;32m[----------] �[mGlobal test environment set-up.
[2026-02-05T19:29:42.907Z] �[0;32m[----------] �[m1 test from UniversalPageStorageServiceCheckpointTest
[2026-02-05T19:29:42.907Z] �[0;32m[ RUN      ] �[mUniversalPageStorageServiceCheckpointTest.DumpFail
[2026-02-05T19:29:42.907Z] [2026/02/05 19:29:41.801 +00:00] [WARN] [Context.cpp:1670] ["GlobalStoragePool has already been initialized."] [source=Context] [thread_id=1]
[2026-02-05T19:29:42.907Z] [2026/02/05 19:29:41.801 +00:00] [INFO] [PageStorageImpl.cpp:54] ["PageStorageImpl start. Config{ PageStorageConfig {blob_file_limit_size: 268435456, blob_spacemap_type: 2, blob_heavy_gc_valid_rate: 0.500, blob_heavy_gc_valid_rate_raft_data: 0.050, blob_block_alignment_bytes: 0, wal_roll_size: 2097152, wal_max_persisted_log_files: 4} }"] [source=__global__.log] [thread_id=1]
[2026-02-05T19:29:42.907Z] [2026/02/05 19:29:41.801 +00:00] [INFO] [PageStorageImpl.cpp:54] ["PageStorageImpl start. Config{ PageStorageConfig {blob_file_limit_size: 268435456, blob_spacemap_type: 2, blob_heavy_gc_valid_rate: 0.500, blob_heavy_gc_valid_rate_raft_data: 0.050, blob_block_alignment_bytes: 0, wal_roll_size: 2097152, wal_max_persisted_log_files: 4} }"] [source=__global__.data] [thread_id=1]
[2026-02-05T19:29:42.907Z] [2026/02/05 19:29:41.801 +00:00] [INFO] [PageStorageImpl.cpp:54] ["PageStorageImpl start. Config{ PageStorageConfig {blob_file_limit_size: 268435456, blob_spacemap_type: 2, blob_heavy_gc_valid_rate: 0.500, blob_heavy_gc_valid_rate_raft_data: 0.050, blob_block_alignment_bytes: 0, wal_roll_size: 2097152, wal_max_persisted_log_files: 4} }"] [source=__global__.meta] [thread_id=1]
[2026-02-05T19:29:42.907Z] [2026/02/05 19:29:41.801 +00:00] [INFO] [PageDirectoryFactory.cpp:68] ["PageDirectory restored, max_page_id=0 max_applied_ver=0"] [source=__global__.log] [thread_id=1]
[2026-02-05T19:29:42.907Z] [2026/02/05 19:29:41.801 +00:00] [INFO] [PageDirectoryFactory.cpp:68] ["PageDirectory restored, max_page_id=0 max_applied_ver=0"] [source=__global__.data] [thread_id=1]
[2026-02-05T19:29:42.907Z] [2026/02/05 19:29:41.801 +00:00] [INFO] [PageDirectoryFactory.cpp:68] ["PageDirectory restored, max_page_id=0 max_applied_ver=0"] [source=__global__.meta] [thread_id=1]
[2026-02-05T19:29:42.907Z] [2026/02/05 19:29:41.801 +00:00] [INFO] [StorageConfigParser.cpp:640] ["enable with StorageS3Config{endpoint= bucket=mockbucket root=tiflash_ut/ max_connections=4096 max_redirections=10 connection_timeout_ms=5000 request_timeout_ms=30000 access_key_id_size=0 secret_access_key_size=0 enable_http_pool=false enable_poco_client=true}"] [thread_id=1]
[2026-02-05T19:29:42.908Z] [2026/02/05 19:29:41.802 +00:00] [INFO] [BlobStore.cpp:130] ["Ignore not blob file, dir=/tiflash/tmp/UniversalPageStorageServiceCheckpointTest.DumpFail file=page err_msg="] [source=test.t] [thread_id=1]
[2026-02-05T19:29:42.908Z] [2026/02/05 19:29:41.802 +00:00] [INFO] [PageDirectoryFactory.cpp:68] ["PageDirectory restored, max_page_id=0 max_applied_ver=0"] [source=test.t] [thread_id=1]
[2026-02-05T19:29:42.908Z] [2026/02/05 19:29:41.802 +00:00] [DEBUG] [TiFlashTestEnv.cpp:288] ["Created bucket mockbucket"] [source="bucket=mockbucket root=tiflash_ut/"] [thread_id=1]
[2026-02-05T19:29:42.908Z] [2026/02/05 19:29:41.802 +00:00] [INFO] [BlobStat.cpp:147] ["Created a new BlobStat [blob_id=10] [capacity=268435456]"] [source=test.t] [thread_id=1]
[2026-02-05T19:29:42.908Z] [2026/02/05 19:29:41.802 +00:00] [INFO] [WALStore.cpp:143] ["Creating log file for writing, fullname=/tiflash/tmp/UniversalPageStorageServiceCheckpointTest.DumpFail/wal/log_1_0"] [source=test.t] [thread_id=1]
[2026-02-05T19:29:42.908Z] [2026/02/05 19:29:41.803 +00:00] [DEBUG] [S3Common.cpp:866] ["listPrefix page result, prefix=s2/manifest/ keys=0 total_keys=0"] [source="bucket=mockbucket root=tiflash_ut/"] [thread_id=1]
[2026-02-05T19:29:42.908Z] [2026/02/05 19:29:41.803 +00:00] [DEBUG] [S3Common.cpp:904] ["listPrefix done, prefix=s2/manifest/ total_keys=0 cost=0.00s"] [source="bucket=mockbucket root=tiflash_ut/"] [thread_id=1]
[2026-02-05T19:29:42.908Z] [2026/02/05 19:29:41.803 +00:00] [INFO] [S3LockLocalManager.cpp:114] ["restore from S3 finish, elapsed=0.000s last_upload_sequence=0 num_s3_entries=0 num_copied_entries=0 last_prefix={None}"] [thread_id=1]
[2026-02-05T19:29:42.908Z] [2026/02/05 19:29:41.803 +00:00] [INFO] [PageDirectory.cpp:2393] ["Dumped snapshot to edits, sequence=6 edit_size=3"] [source=test.t] [thread_id=1]
[2026-02-05T19:29:42.908Z] [2026/02/05 19:29:41.803 +00:00] [DEBUG] [PageDirectory.cpp:2354] ["After MVCC gc in memory, elapsed=0.000s general_lowest_seq=none delta_tree_only_seq=5. clean invalid_snapshot_nums=0 invalid_page_nums=0 invalid_raft_pages_nums=0 total_deref_counter=0 all_del_entries=0. Still exist, snapshot_nums=0 page_nums=3. Longest alive snapshot: {alive_time=0.000 seq=0 type=General tracing_id=} stale_snapshot_nums=0"] [source=test.t_1] [thread_id=1]
[2026-02-05T19:29:42.908Z] [2026/02/05 19:29:41.803 +00:00] [INFO] [PageDirectory.cpp:2393] ["Dumped snapshot to edits, sequence=5 edit_size=3"] [source=test.t_1] [thread_id=1]
[2026-02-05T19:29:42.908Z] [2026/02/05 19:29:41.808 +00:00] [DEBUG] [CPDataFileStat.cpp:191] ["CheckpointData stats {compaction=[] unchanged=[]} RemoteGCThreshold{min_age=3600 min_file_threshold=131072 valid_rate=0.50%}"] [source=UniPSService] [thread_id=1]
[2026-02-05T19:29:42.908Z] [2026/02/05 19:29:41.808 +00:00] [DEBUG] [CPFilesWriter.cpp:105] ["Prepare to dump 3 records, sequence=1, manifest_file_id=s2/manifest/mf_1"] [thread_id=1]
[2026-02-05T19:29:42.908Z] [2026/02/05 19:29:41.808 +00:00] [DEBUG] [CPFilesWriter.cpp:241] ["compact stats: {}"] [thread_id=1]
[2026-02-05T19:29:42.908Z] [2026/02/05 19:29:41.809 +00:00] [ERROR] [S3Common.cpp:699] ["S3 PutObject failed: , request_id= local_fname=/tiflash/tmp/checkpoint_upload_1/dat_1_0 bucket=mockbucket root=tiflash_ut/ key=s2/data/dat_1_0"] [source="bucket=mockbucket root=tiflash_ut/"] [thread_id=6]
[2026-02-05T19:29:42.908Z] [2026/02/05 19:29:41.810 +00:00] [ERROR] [S3Common.cpp:699] ["S3 PutObject failed: , request_id= local_fname=/tiflash/tmp/checkpoint_upload_1/dat_1_0 bucket=mockbucket root=tiflash_ut/ key=s2/data/dat_1_0"] [source="bucket=mockbucket root=tiflash_ut/"] [thread_id=6]
[2026-02-05T19:29:42.909Z] [2026/02/05 19:29:42.514 +00:00] [ERROR] [Exception.cpp:88] ["Code: 11000, e.displayText() = DB::Exception: S3 PutObject failed, local_fname=/tiflash/tmp/checkpoint_upload_1/dat_1_0 bucket=mockbucket root=tiflash_ut/ key=s2/data/dat_1_0 s3error=UNKNOWN s3exception_name= s3msg= request_id=, e.what() = DB::Exception, Stack trace:\n\n\n  0x55cc7cb98c15\t___interceptor_backtrace [gtests_dbms+124582933]\n                \t/tiflash-release-8.3/release-centos7-llvm/dockerfiles/misc/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:4355\n  0x55cca1519003\tStackTrace::StackTrace() [gtests_dbms+738525187]\n                \tdbms/src/Common/StackTrace.cpp:25\n  0x55cc9bd36396\tDB::Exception::Exception<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) [gtests_dbms+646370198]\n                \tdbms/src/Common/Exception.h:48\n  0x55cc9bd35513\tDB::Exception DB::S3::fromS3Error<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(Aws::S3::S3Error const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) [gtests_dbms+646366483]\n                \tdbms/src/Storages/S3/S3Common.h:58\n  0x55cc9bd167b6\tDB::S3::uploadFile(DB::S3::TiFlashS3Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::EncryptionPath const&, std::__1::shared_ptr<DB::FileProvider> const&, int) [gtests_dbms+646240182]\n                \tdbms/src/Storages/S3/S3Common.cpp:680\n  0x55cc9658439d\tstd::__1::__packaged_task_func<DB::DM::Remote::DataStoreS3::putCheckpointFiles(DB::PS::V3::LocalCheckpointFiles const&, unsigned long, unsigned long)::$_0, std::__1::allocator<DB::DM::Remote::DataStoreS3::putCheckpointFiles(DB::PS::V3::LocalCheckpointFiles const&, unsigned long, unsigned long)::$_0>, void ()>::operator()() [gtests_dbms+554415005]\n                \tdbms/src/Storages/DeltaMerge/Remote/DataStore/DataStoreS3.cpp:156\n  0x55cc7cf3f734\tstd::__1::packaged_task<void ()>::operator()() [gtests_dbms+128411444]\n                \t/usr/local/bin/../include/c++/v1/future:1891\n  0x55cca16393b5\tDB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::worker(std::__1::__list_iterator<DB::ThreadFromGlobalPoolImpl<false>, void*>) [gtests_dbms+739705781]\n                \t/usr/local/bin/../include/c++/v1/__functional/function.h:517\n  0x55cca1642c7c\tstd::__1::__function::__func<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), std::__1::allocator<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()>, void ()>::operator()() [gtests_dbms+739744892]\n                \tdbms/src/Common/UniThreadPool.cpp:169\n  0x55cca16340c2\tDB::ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) [gtests_dbms+739684546]\n                \t/usr/local/bin/../include/c++/v1/__functional/function.h:517\n  0x55cca163d423\tvoid* std::__1::__thread_proxy[abi:ue170006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>>(void*) [gtests_dbms+739722275]\n                \tdbms/src/Common/UniThreadPool.cpp:169\n  0x55cc7cbeca0b\tasan_thread_start(void*) [gtests_dbms+124926475]\n                \t/tiflash-release-8.3/release-centos7-llvm/dockerfiles/misc/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:225\n  0x7f9bd63e1ea5\tstart_thread [libpthread.so.0+32421]\n  0x7f9bd5cdab0d\tclone [libc.so.6+1043213]"] [source=DataStoreS3] [thread_id=1]
[2026-02-05T19:29:42.910Z] libc++abi: terminating due to uncaught exception of type Poco::FileNotFoundException: File not found
[2026-02-05T19:29:42.910Z] Received signal Aborted
[2026-02-05T19:29:42.910Z] 
[2026-02-05T19:29:42.910Z]   0x55cc7cb98c15	___interceptor_backtrace [gtests_dbms+124582933]
[2026-02-05T19:29:42.910Z]                 	/tiflash-release-8.3/release-centos7-llvm/dockerfiles/misc/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:4355
[2026-02-05T19:29:42.910Z]   0x55cca1519003	StackTrace::StackTrace() [gtests_dbms+738525187]
[2026-02-05T19:29:42.910Z]                 	dbms/src/Common/StackTrace.cpp:25
[2026-02-05T19:29:42.910Z]   0x55cc86dfd60d	fault_signal_handler(int) [gtests_dbms+294864397]
[2026-02-05T19:29:42.910Z]                 	dbms/src/TestUtils/gtests_dbms_main.cpp:40
[2026-02-05T19:29:42.910Z]   0x7f9bd5c12400	<unknown symbol> [libc.so.6+222208]
[2026-02-05T19:29:42.910Z]   0x7f9bd5c12387	raise [libc.so.6+222087]
[2026-02-05T19:29:42.910Z]   0x7f9bd5c13a78	abort [libc.so.6+227960]
[2026-02-05T19:29:42.910Z]   0x7f9bd6e19186	abort_message [libc++abi.so.1+229766]
[2026-02-05T19:29:42.910Z]   0x7f9bd6dfd12b	demangling_terminate_handler() [libc++abi.so.1+114987]
[2026-02-05T19:29:42.910Z]   0x7f9bd6e18473	std::__terminate(void (*)()) [libc++abi.so.1+226419]
[2026-02-05T19:29:42.910Z]   0x7f9bd6e18428	std::terminate() [libc++abi.so.1+226344]
[2026-02-05T19:29:42.910Z]   0x55cc7cc37ace	__clang_call_terminate [gtests_dbms+125233870]
[2026-02-05T19:29:42.911Z]   0x55cc9d3bc4e5	ext::ScopeGuard<DB::UniversalPageStorageService::uploadCheckpointImpl(metapb::Store const&, std::__1::shared_ptr<DB::S3::IS3LockClient> const&, std::__1::shared_ptr<DB::DM::Remote::IDataStore> const&, bool)::$_1>::~ScopeGuard() [gtests_dbms+669988069]
[2026-02-05T19:29:42.911Z]                 	libs/libcommon/include/ext/scope_guard.h:35
[2026-02-05T19:29:42.911Z]   0x55cc9d3bbafa	DB::UniversalPageStorageService::uploadCheckpointImpl(metapb::Store const&, std::__1::shared_ptr<DB::S3::IS3LockClient> const&, std::__1::shared_ptr<DB::DM::Remote::IDataStore> const&, bool) [gtests_dbms+669985530]
[2026-02-05T19:29:42.911Z]                 	dbms/src/Storages/Page/V3/Universal/UniversalPageStorageService.cpp:308
[2026-02-05T19:29:42.911Z]   0x55cc8632ff02	DB::PS::universal::tests::UniversalPageStorageServiceCheckpointTest_DumpFail_Test::TestBody() [gtests_dbms+283537154]
[2026-02-05T19:29:42.911Z]                 	dbms/src/Storages/Page/V3/Universal/tests/gtest_checkpoint.cpp:1293
[2026-02-05T19:29:42.911Z]   0x55cc96bdd6b2	void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) [gtests_dbms+561071794]
[2026-02-05T19:29:42.911Z]                 	contrib/googletest/googletest/src/gtest.cc:2443
[2026-02-05T19:29:42.911Z]   0x55cc96b7dcf6	testing::Test::Run() [gtests_dbms+560680182]
[2026-02-05T19:29:42.911Z]                 	contrib/googletest/googletest/src/gtest.cc:2517
[2026-02-05T19:29:42.911Z]   0x55cc96b8125b	testing::TestInfo::Run() [gtests_dbms+560693851]
[2026-02-05T19:29:42.911Z]                 	contrib/googletest/googletest/src/gtest.cc:2693
[2026-02-05T19:29:42.911Z]   0x55cc96b82910	testing::TestCase::Run() [gtests_dbms+560699664]
[2026-02-05T19:29:42.911Z]                 	contrib/googletest/googletest/src/gtest.cc:2811
[2026-02-05T19:29:42.911Z]   0x55cc96bade0b	testing::internal::UnitTestImpl::RunAllTests() [gtests_dbms+560877067]
[2026-02-05T19:29:42.911Z]                 	contrib/googletest/googletest/src/gtest.cc:5177
[2026-02-05T19:29:42.911Z]   0x55cc96bdfdf2	bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) [gtests_dbms+561081842]
[2026-02-05T19:29:42.911Z]                 	contrib/googletest/googletest/src/gtest.cc:2443
[2026-02-05T19:29:42.911Z]   0x55cc96bad0a7	testing::UnitTest::Run() [gtests_dbms+560873639]
[2026-02-05T19:29:42.911Z]                 	contrib/googletest/googletest/src/gtest.cc:4786
[2026-02-05T19:29:42.911Z]   0x55cc86e00166	main [gtests_dbms+294875494]
[2026-02-05T19:29:42.911Z]                 	contrib/googletest/googletest/include/gtest/gtest.h:2341
[2026-02-05T19:29:42.911Z]   0x7f9bd5bfe555	__libc_start_main [libc.so.6+140629]
[2026-02-05T19:29:42.911Z]   0x55cc7cb52ea9	<unknown symbol> [gtests_dbms+124296873]