[Bug]: During my CI test runs, the tests debug_restart_test, debug_unit_test, and debug_pysdk_testare failing intermittently.

[zpf121]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Version or Commit ID

fix_optimize_and_alter_index

Other environment information

Actual behavior and How to reproduce it

During my CI test runs, the tests debug_restart_test, debug_unit_test, and debug_pysdk_testare failing intermittently.

Expected behavior

No response

Additional information

No response

[zpf121]

debug_unit_test failure cause identified:

Expected: (correct_rate) >= (0.95), actual: 0.908953 vs 0.95

[  FAILED  ] HnswHandlerTest.test_parallel (6851 ms)

[zpf121]

debug_pysdk_test:

Txn ID: 27215, Text: optimize index, Begin TS: 34067, Commit TS: 18446744073709551615, KV Commit TS: 18446744073709551615, State: Started


[11:34:45.811] [410] [critical] Error: Aborted
[11:34:45.811] [410] [critical]    0# infinity::PrintStacktrace@infinity_core(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at /infinity/src/common/utility/exception_impl.cpp:47
   1# (anonymous namespace)::SignalHandler(int, siginfo_t*, void*) at /infinity/src/bin/infinity_main.cpp:160
   2#      at /build/glibc-B3wQXB/glibc-2.31/nptl/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0
   3# __GI_raise at ../sysdeps/unix/sysv/linux/raise.c:51
   4# __GI_abort at /build/glibc-B3wQXB/glibc-2.31/stdlib/abort.c:79
   5# (anonymous namespace)::TerminateHandler() at /infinity/src/bin/infinity_main.cpp:222
   6# __cxxabiv1::__terminate(void (*)()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
   7# std::terminate() at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:58
   8# __cxa_throw at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:98
   9# infinity::UnrecoverableError@infinity_core(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, unsigned int) at /infinity/src/common/utility/exception_impl.cpp:84
  10# infinity::BGTaskProcessor@infinity_core::Process() at /infinity/src/storage/background_process_impl.cpp:114
  11# infinity::BGTaskProcessor@infinity_core::Start()::$_0::operator()() const at /infinity/src/storage/background_process_impl.cpp:42
  12# void std::__invoke_impl<void, infinity::BGTaskProcessor@infinity_core::Start()::$_0>(std::__invoke_other, infinity::BGTaskProcessor@infinity_core::Start()::$_0&&) at /usr/lib/gcc/x86_64-pc-linux-gnu/15.2.0/../../../../include/c++/15.2.0/bits/invoke.h:63
  13# std::__invoke_result<infinity::BGTaskProcessor@infinity_core::Start()::$_0>::type std::__invoke<infinity::BGTaskProcessor@infinity_core::Start()::$_0>(infinity::BGTaskProcessor@infinity_core::Start()::$_0&&) at /usr/lib/gcc/x86_64-pc-linux-gnu/15.2.0/../../../../include/c++/15.2.0/bits/invoke.h:98
  14# void std::thread::_Invoker<std::tuple<infinity::BGTaskProcessor@infinity_core::Start()::$_0> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) at /usr/lib/gcc/x86_64-pc-linux-gnu/15.2.0/../../../../include/c++/15.2.0/bits/std_thread.h:303
  15# std::thread::_Invoker<std::tuple<infinity::BGTaskProcessor@infinity_core::Start()::$_0> >::operator()() at /usr/lib/gcc/x86_64-pc-linux-gnu/15.2.0/../../../../include/c++/15.2.0/bits/std_thread.h:310
  16# std::thread::_State_impl<std::thread::_Invoker<std::tuple<infinity::BGTaskProcessor@infinity_core::Start()::$_0> > >::_M_run() at /usr/lib/gcc/x86_64-pc-linux-gnu/15.2.0/../../../../include/c++/15.2.0/bits/std_thread.h:255
  17# execute_native_thread_routine at ../../../../../libstdc++-v3/src/c++11/thread.cc:104
  18# asan_thread_start(void*) at :0
  19# start_thread at /build/glibc-B3wQXB/glibc-2.31/nptl/pthread_create.c:477
  20# __clone at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
  21# 

[zpf121]

debug_restart_test:

FAILED python/restart_test/test_fulltext.py::TestFullText::test_fulltext_realtime[infinity_runner0]

[01:32:08.808] [16645] [trace] Replay wal cmd: Append: 
db name: default_db
db id: 1
table name: test_fulltext_gt
table id: 0
row range: [8413, +1] | row count: 1
column: Integer Varchar Varchar Varchar 
, txn id: 8429, commit ts: 16840
[01:32:08.809] [16645] [debug] GetTableMeta: txn_id: 8429 table_id: 0
[01:32:08.809] [16645] [trace] NewTxn: 8429 is committing, begin_ts:16839 committing ts: 16840
[01:32:08.809] [16645] [trace] Transaction commit bottom: 8429 start.
[01:32:08.809] [16645] [debug] NewTxn::CommitBottomAppend begin
[01:32:08.810] [16645] [trace] BlockVersion::LoadFromFile version, created: 221
[01:32:08.816] [16645] [trace] Request memory 65536
[01:32:08.816] [16645] [debug] CommitBottomAppend block 1, existing row cnt 221, new row cnt 1
[01:32:08.816] [16645] [trace] Request memory 32768
[01:32:08.816] [16645] [trace] Request memory 131072
[01:32:08.816] [16645] [critical] 
[01:32:08.816] [16645] [critical] Error: Read path: /var/infinity/data/db_1/tbl_0/seg_0/blk_1/col_1_out, error: IO error: File: /var/infinity/data/db_1/tbl_0/seg_0/blk_1/col_1_out open failed: No such file or directory@src/storage/buffer/file_worker/file_worker_impl.cpp:133
[01:32:09.249] [16661] [trace] THRIFT: Connect success, new session 1
[01:32:09.250] [16661] [error] [THRIFT ERROR]: Infinity is initing
[01:32:10.010] [16645] [critical]    0# infinity::PrintStacktrace@infinity_core(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at /infinity/src/common/utility/exception_impl.cpp:47
   1# infinity::UnrecoverableError@infinity_core(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, unsigned int) at /infinity/src/common/utility/exception_impl.cpp:81
   2# infinity::FileWorker@infinity_core::ReadFromFile(bool) at /infinity/src/storage/buffer/file_worker/file_worker_impl.cpp:133
   3# infinity::BufferObj@infinity_core::Load() at /infinity/src/storage/buffer/buffer_obj_impl.cpp:114
   4# infinity::VarBufferManager@infinity_core::GetInnerNoLock() at /infinity/src/storage/column_vector/var_buffer_impl.cpp:174
   5# infinity::VarBufferManager@infinity_core::TotalSize() at /infinity/src/storage/column_vector/var_buffer_impl.cpp:198
   6# infinity::NewTxn@infinity_core::AppendInColumn(infinity::ColumnMeta@infinity_core&, unsigned long, unsigned long, infinity::ColumnVector@infinity_core const&, unsigned long) at /infinity/src/storage/new_txn/new_txn_data_impl.cpp:951
   7# infinity::NewTxn@infinity_core::AppendInBlock(infinity::BlockMeta@infinity_core&, unsigned long, unsigned long, infinity::DataBlock@infinity_core const*, unsigned long) at /infinity/src/storage/new_txn/new_txn_data_impl.cpp:910
   8# infinity::NewTxn@infinity_core::CommitBottomAppend(infinity::WalCmdAppendV2@infinity_core*) at /infinity/src/storage/new_txn/new_txn_data_impl.cpp:1594
   9# infinity::NewTxn@infinity_core::CommitBottom() at /infinity/src/storage/new_txn/new_txn_impl.cpp:4055
  10# infinity::NewTxn@infinity_core::CommitReplay() at /infinity/src/storage/new_txn/new_txn_impl.cpp:1982
  11# infinity::NewTxnManager@infinity_core::CommitReplayTxn(infinity::NewTxn@infinity_core*) at /infinity/src/storage/new_txn/new_txn_manager_impl.cpp:331
  12# infinity::WalManager@infinity_core::ReplayWalEntries(std::vector<std::shared_ptr<infinity::WalEntry@infinity_core>, std::allocator<std::shared_ptr<infinity::WalEntry@infinity_core> > > const&) at /infinity/src/storage/wal/wal_manager_impl.cpp:706
  13# infinity::Storage@infinity_core::AdminToWriter() at /infinity/src/storage/storage_impl.cpp:292
  14# infinity::Storage@infinity_core::SetStorageMode(infinity::StorageMode@infinity_core) at /infinity/src/storage/storage_impl.cpp:744
  15# infinity::InfinityContext@infinity_core::ChangeServerRole(infinity::NodeRole, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned short) at /infinity/src/main/infinity_context_impl.cpp:132
  16# infinity::InfinityContext@infinity_core::InitPhase2(bool) at /infinity/src/main/infinity_context_impl.cpp:90
  17# main at /infinity/src/bin/infinity_main.cpp:285
  18# __libc_start_main at ../csu/libc-start.c:308
  19# _start at :0
  20# 

[01:32:10.011] [16645] [critical] 
[01:32:10.011] [16645] [critical] Error: TerminateHandler: Unhandled Exception: Read path: /var/infinity/data/db_1/tbl_0/seg_0/blk_1/col_1_out, error: IO error: File: /var/infinity/data/db_1/tbl_0/seg_0/blk_1/col_1_out open failed: No such file or directory@src/storage/buffer/file_worker/file_worker_impl.cpp:133
[01:32:10.011] [16645] [critical]    0# infinity::PrintStacktrace@infinity_core(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at /infinity/src/common/utility/exception_impl.cpp:47
   1# (anonymous namespace)::TerminateHandler() at /infinity/src/bin/infinity_main.cpp:221
   2# __cxxabiv1::__terminate(void (*)()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
   3# std::terminate() at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:58
   4# __cxa_throw at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:98
   5# infinity::UnrecoverableError@infinity_core(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, unsigned int) at /infinity/src/common/utility/exception_impl.cpp:84
   6# infinity::FileWorker@infinity_core::ReadFromFile(bool) at /infinity/src/storage/buffer/file_worker/file_worker_impl.cpp:133
   7# infinity::BufferObj@infinity_core::Load() at /infinity/src/storage/buffer/buffer_obj_impl.cpp:114
   8# infinity::VarBufferManager@infinity_core::GetInnerNoLock() at /infinity/src/storage/column_vector/var_buffer_impl.cpp:174
   9# infinity::VarBufferManager@infinity_core::TotalSize() at /infinity/src/storage/column_vector/var_buffer_impl.cpp:198
  10# infinity::NewTxn@infinity_core::AppendInColumn(infinity::ColumnMeta@infinity_core&, unsigned long, unsigned long, infinity::ColumnVector@infinity_core const&, unsigned long) at /infinity/src/storage/new_txn/new_txn_data_impl.cpp:951
  11# infinity::NewTxn@infinity_core::AppendInBlock(infinity::BlockMeta@infinity_core&, unsigned long, unsigned long, infinity::DataBlock@infinity_core const*, unsigned long) at /infinity/src/storage/new_txn/new_txn_data_impl.cpp:910
  12# infinity::NewTxn@infinity_core::CommitBottomAppend(infinity::WalCmdAppendV2@infinity_core*) at /infinity/src/storage/new_txn/new_txn_data_impl.cpp:1594
  13# infinity::NewTxn@infinity_core::CommitBottom() at /infinity/src/storage/new_txn/new_txn_impl.cpp:4055
  14# infinity::NewTxn@infinity_core::CommitReplay() at /infinity/src/storage/new_txn/new_txn_impl.cpp:1982
  15# infinity::NewTxnManager@infinity_core::CommitReplayTxn(infinity::NewTxn@infinity_core*) at /infinity/src/storage/new_txn/new_txn_manager_impl.cpp:331
  16# infinity::WalManager@infinity_core::ReplayWalEntries(std::vector<std::shared_ptr<infinity::WalEntry@infinity_core>, std::allocator<std::shared_ptr<infinity::WalEntry@infinity_core> > > const&) at /infinity/src/storage/wal/wal_manager_impl.cpp:706
  17# infinity::Storage@infinity_core::AdminToWriter() at /infinity/src/storage/storage_impl.cpp:292
  18# infinity::Storage@infinity_core::SetStorageMode(infinity::StorageMode@infinity_core) at /infinity/src/storage/storage_impl.cpp:744
  19# infinity::InfinityContext@infinity_core::ChangeServerRole(infinity::NodeRole, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned short) at /infinity/src/main/infinity_context_impl.cpp:132
  20# infinity::InfinityContext@infinity_core::InitPhase2(bool) at /infinity/src/main/infinity_context_impl.cpp:90
  21# main at /infinity/src/bin/infinity_main.cpp:285
  22# __libc_start_main at ../csu/libc-start.c:308
  23# _start at :0
  24# 

[01:32:10.011] [16645] [critical] 
[01:32:10.011] [16645] [critical] Error: Aborted
[01:32:10.012] [16645] [critical]    0# infinity::PrintStacktrace@infinity_core(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at /infinity/src/common/utility/exception_impl.cpp:47
   1# (anonymous namespace)::SignalHandler(int, siginfo_t*, void*) at /infinity/src/bin/infinity_main.cpp:160
   2#      at /build/glibc-B3wQXB/glibc-2.31/nptl/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0
   3# __GI_raise at ../sysdeps/unix/sysv/linux/raise.c:51
   4# __GI_abort at /build/glibc-B3wQXB/glibc-2.31/stdlib/abort.c:79
   5# (anonymous namespace)::TerminateHandler() at /infinity/src/bin/infinity_main.cpp:222
   6# __cxxabiv1::__terminate(void (*)()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
   7# std::terminate() at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:58
   8# __cxa_throw at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:98
   9# infinity::UnrecoverableError@infinity_core(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, unsigned int) at /infinity/src/common/utility/exception_impl.cpp:84
  10# infinity::FileWorker@infinity_core::ReadFromFile(bool) at /infinity/src/storage/buffer/file_worker/file_worker_impl.cpp:133
  11# infinity::BufferObj@infinity_core::Load() at /infinity/src/storage/buffer/buffer_obj_impl.cpp:114
  12# infinity::VarBufferManager@infinity_core::GetInnerNoLock() at /infinity/src/storage/column_vector/var_buffer_impl.cpp:174
  13# infinity::VarBufferManager@infinity_core::TotalSize() at /infinity/src/storage/column_vector/var_buffer_impl.cpp:198
  14# infinity::NewTxn@infinity_core::AppendInColumn(infinity::ColumnMeta@infinity_core&, unsigned long, unsigned long, infinity::ColumnVector@infinity_core const&, unsigned long) at /infinity/src/storage/new_txn/new_txn_data_impl.cpp:951
  15# infinity::NewTxn@infinity_core::AppendInBlock(infinity::BlockMeta@infinity_core&, unsigned long, unsigned long, infinity::DataBlock@infinity_core const*, unsigned long) at /infinity/src/storage/new_txn/new_txn_data_impl.cpp:910
  16# infinity::NewTxn@infinity_core::CommitBottomAppend(infinity::WalCmdAppendV2@infinity_core*) at /infinity/src/storage/new_txn/new_txn_data_impl.cpp:1594
  17# infinity::NewTxn@infinity_core::CommitBottom() at /infinity/src/storage/new_txn/new_txn_impl.cpp:4055
  18# infinity::NewTxn@infinity_core::CommitReplay() at /infinity/src/storage/new_txn/new_txn_impl.cpp:1982
  19# infinity::NewTxnManager@infinity_core::CommitReplayTxn(infinity::NewTxn@infinity_core*) at /infinity/src/storage/new_txn/new_txn_manager_impl.cpp:331
  20# infinity::WalManager@infinity_core::ReplayWalEntries(std::vector<std::shared_ptr<infinity::WalEntry@infinity_core>, std::allocator<std::shared_ptr<infinity::WalEntry@infinity_core> > > const&) at /infinity/src/storage/wal/wal_manager_impl.cpp:706
  21# infinity::Storage@infinity_core::AdminToWriter() at /infinity/src/storage/storage_impl.cpp:292
  22# infinity::Storage@infinity_core::SetStorageMode(infinity::StorageMode@infinity_core) at /infinity/src/storage/storage_impl.cpp:744
  23# infinity::InfinityContext@infinity_core::ChangeServerRole(infinity::NodeRole, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned short) at /infinity/src/main/infinity_context_impl.cpp:132
  24# infinity::InfinityContext@infinity_core::InitPhase2(bool) at /infinity/src/main/infinity_context_impl.cpp:90
  25# main at /infinity/src/bin/infinity_main.cpp:285
  26# __libc_start_main at ../csu/libc-start.c:308
  27# _start at :0
  28# 

Aborted