Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] inverted index load data core #45715

Open
3 tasks done
qidaye opened this issue Dec 20, 2024 · 0 comments
Open
3 tasks done

[Bug] inverted index load data core #45715

qidaye opened this issue Dec 20, 2024 · 0 comments

Comments

@qidaye
Copy link
Contributor

qidaye commented Dec 20, 2024 •
edited
Loading

Search before asking

  • I had searched in the issues and found no similar issues.

Version

2.0.16

What's Wrong?

When ram_dir is enabled, when fs close error, be will core

=================================================================
==1757461==ERROR: AddressSanitizer: heap-use-after-free on address 0x606000a838f0 at pc 0x55f8421c864e bp 0x7fbd138083a0 sp 0x7fbd13808398
READ of size 8 at 0x606000a838f0 thread T1687 (MemTableFlushTh)
    #0 0x55f8421c864d in lucene::util::mutex_thread::lock() /mnt/disk2/luen/develop/workspace/doris/be/src/clucene/src/shared/CLucene/config/threads.cpp:235:3
    #1 0x55f8421c897a in lucene::util::mutexGuard::mutexGuard(lucene::util::mutex_thread&) /mnt/disk2/luen/develop/workspace/doris/be/src/clucene/src/shared/CLucene/config/threads.cpp:291:11
    #2 0x55f841b20cc6 in lucene::store::SingleInstanceLock::release() /mnt/disk2/luen/develop/workspace/doris/be/src/clucene/src/core/CLucene/store/Lock.cpp:94:5
    #3 0x55f841fd376d in lucene::index::IndexWriter::deinit(bool) /mnt/disk2/luen/develop/workspace/doris/be/src/clucene/src/core/CLucene/index/IndexWriter.cpp:97:20
    #4 0x55f841fd4069 in lucene::index::IndexWriter::~IndexWriter() /mnt/disk2/luen/develop/workspace/doris/be/src/clucene/src/core/CLucene/index/IndexWriter.cpp:115:5
    #5 0x55f841fd42a8 in lucene::index::IndexWriter::~IndexWriter() /mnt/disk2/luen/develop/workspace/doris/be/src/clucene/src/core/CLucene/index/IndexWriter.cpp:114:29
    #6 0x55f81a386202 in std::default_delete<lucene::index::IndexWriter>::operator()(lucene::index::IndexWriter*) const /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2
    #7 0x55f81a381f28 in std::unique_ptr<lucene::index::IndexWriter, std::default_delete<lucene::index::IndexWriter>>::~unique_ptr() /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4
    #8 0x55f81a3b59bd in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)17>::~InvertedIndexColumnWriterImpl() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:93:5
    #9 0x55f81a3b5a18 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)17>::~InvertedIndexColumnWriterImpl() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:89:47
    #10 0x55f81a3202e2 in std::default_delete<doris::segment_v2::InvertedIndexColumnWriter>::operator()(doris::segment_v2::InvertedIndexColumnWriter*) const /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2
    #11 0x55f81a3110f8 in std::unique_ptr<doris::segment_v2::InvertedIndexColumnWriter, std::default_delete<doris::segment_v2::InvertedIndexColumnWriter>>::~unique_ptr() /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4
    #12 0x55f81a2f837b in doris::segment_v2::ScalarColumnWriter::~ScalarColumnWriter() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:454:1
    #13 0x55f81a2f8408 in doris::segment_v2::ScalarColumnWriter::~ScalarColumnWriter() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:446:43
    #14 0x55f81a2d7402 in std::default_delete<doris::segment_v2::ColumnWriter>::operator()(doris::segment_v2::ColumnWriter*) const /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2
    #15 0x55f81a2d72d8 in std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>::~unique_ptr() /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4
    #16 0x55f81a2d7234 in void std::destroy_at<std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>>(std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>*) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15
    #17 0x55f81a2d7214 in void std::_Destroy<std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>>(std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>*) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7
    #18 0x55f81a2d71e6 in void std::_Destroy_aux<false>::__destroy<std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>*>(std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>*, std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>*) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6
    #19 0x55f81a2d71ac in void std::_Destroy<std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>*>(std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>*, std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>*) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7
    #20 0x55f81a2d70a0 in void std::_Destroy<std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>*, std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>>(std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>*, std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>*, std::allocator<std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>>&) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7
    #21 0x55f81a2c850e in std::vector<std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>, std::allocator<std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>>>::~vector() /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680:2
    #22 0x55f81a2a5cb3 in doris::segment_v2::SegmentWriter::~SegmentWriter() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:109:1
    #23 0x55f819d9da2a in std::default_delete<doris::segment_v2::SegmentWriter>::operator()(doris::segment_v2::SegmentWriter*) const /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2
    #24 0x55f819d9130b in std::unique_ptr<doris::segment_v2::SegmentWriter, std::default_delete<doris::segment_v2::SegmentWriter>>::~unique_ptr() /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4
    #25 0x55f819db2c26 in doris::BetaRowsetWriter::flush_single_memtable(doris::vectorized::Block const*, long*, doris::FlushContext const*) /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/beta_rowset_writer.cpp:505:1
    #26 0x55f81a64fd05 in doris::MemTable::_do_flush() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/memtable.cpp:533:5
    #27 0x55f81a64e7f4 in doris::MemTable::flush() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/memtable.cpp:496:5
    #28 0x55f81a62dd84 in doris::FlushToken::_flush_memtable(doris::MemTable*, long) /mnt/disk2/luen/develop/workspace/doris/be/src/olap/memtable_flush_executor.cpp:116:26
    #29 0x55f81a63feee in doris::MemtableFlushTask::run() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/memtable_flush_executor.cpp:54:23
    #30 0x55f81b8ea036 in doris::ThreadPool::dispatch_thread() /mnt/disk2/luen/develop/workspace/doris/be/src/util/threadpool.cpp:533:24
    #31 0x55f81b90f3c3 in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14
    #32 0x55f81b90f29c in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
    #33 0x55f81b90f224 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::__call<void, 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:420:11
    #34 0x55f81b90f0cd in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::operator()<void>() /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:503:17
    #35 0x55f81b90efe4 in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
    #36 0x55f81b90ef84 in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
    #37 0x55f81b90ec2c in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>>::_M_invoke(std::_Any_data const&) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
    #38 0x55f81840e022 in std::function<void ()>::operator()() const /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
    #39 0x55f81b8bd2f6 in doris::Thread::supervise_thread(void*) /mnt/disk2/luen/develop/workspace/doris/be/src/util/thread.cpp:498:5
    #40 0x7fc4eaba81c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: f8eb096e42e51112f2ff7ff793cf931aba1d3ef2)
    #41 0x7fc4eb5978d2 in clone (/lib64/libc.so.6+0x398d2) (BuildId: 6ae06e3b90805d95009d52783e9b82fe5f92c5a5)

0x606000a838f0 is located 48 bytes inside of 56-byte region [0x606000a838c0,0x606000a838f8)
freed by thread T1687 (MemTableFlushTh) here:
    #0 0x55f818259d9d in operator delete(void*) (/mnt/disk2/luen/mydoris2/be/lib/doris_be+0x1a3d0d9d) (BuildId: cc86a6484331667c)
    #1 0x55f841b1c6e1 in lucene::store::SingleInstanceLockFactory::~SingleInstanceLockFactory() /mnt/disk2/luen/develop/workspace/doris/be/src/clucene/src/core/CLucene/store/LockFactory.cpp:51:1
    #2 0x55f81a03a422 in doris::segment_v2::DorisRAMCompoundDirectory::~DorisRAMCompoundDirectory() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/segment_v2/inverted_index_compound_directory.cpp:701:5
    #3 0x55f81a03a568 in doris::segment_v2::DorisRAMCompoundDirectory::~DorisRAMCompoundDirectory() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/segment_v2/inverted_index_compound_directory.cpp:700:57
    #4 0x55f81a386022 in std::default_delete<doris::segment_v2::DorisCompoundDirectory>::operator()(doris::segment_v2::DorisCompoundDirectory*) const /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2
    #5 0x55f81a381e18 in std::unique_ptr<doris::segment_v2::DorisCompoundDirectory, std::default_delete<doris::segment_v2::DorisCompoundDirectory>>::~unique_ptr() /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4
    #6 0x55f81a3b5949 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)17>::~InvertedIndexColumnWriterImpl() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:93:5
    #7 0x55f81a3b5a18 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)17>::~InvertedIndexColumnWriterImpl() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:89:47
    #8 0x55f81a3202e2 in std::default_delete<doris::segment_v2::InvertedIndexColumnWriter>::operator()(doris::segment_v2::InvertedIndexColumnWriter*) const /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2
    #9 0x55f81a3110f8 in std::unique_ptr<doris::segment_v2::InvertedIndexColumnWriter, std::default_delete<doris::segment_v2::InvertedIndexColumnWriter>>::~unique_ptr() /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4
    #10 0x55f81a2f837b in doris::segment_v2::ScalarColumnWriter::~ScalarColumnWriter() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:454:1
    #11 0x55f81a2f8408 in doris::segment_v2::ScalarColumnWriter::~ScalarColumnWriter() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:446:43
    #12 0x55f81a2d7402 in std::default_delete<doris::segment_v2::ColumnWriter>::operator()(doris::segment_v2::ColumnWriter*) const /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2
    #13 0x55f81a2d72d8 in std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>::~unique_ptr() /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4
    #14 0x55f81a2d7234 in void std::destroy_at<std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>>(std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>*) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:88:15
    #15 0x55f81a2d7214 in void std::_Destroy<std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>>(std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>*) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:138:7
    #16 0x55f81a2d71e6 in void std::_Destroy_aux<false>::__destroy<std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>*>(std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>*, std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>*) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:152:6
    #17 0x55f81a2d71ac in void std::_Destroy<std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>*>(std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>*, std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>*) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:184:7
    #18 0x55f81a2d70a0 in void std::_Destroy<std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>*, std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>>(std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>*, std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>*, std::allocator<std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>>&) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:746:7
    #19 0x55f81a2c850e in std::vector<std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>, std::allocator<std::unique_ptr<doris::segment_v2::ColumnWriter, std::default_delete<doris::segment_v2::ColumnWriter>>>>::~vector() /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680:2
    #20 0x55f81a2a5cb3 in doris::segment_v2::SegmentWriter::~SegmentWriter() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:109:1
    #21 0x55f819d9da2a in std::default_delete<doris::segment_v2::SegmentWriter>::operator()(doris::segment_v2::SegmentWriter*) const /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:85:2
    #22 0x55f819d9130b in std::unique_ptr<doris::segment_v2::SegmentWriter, std::default_delete<doris::segment_v2::SegmentWriter>>::~unique_ptr() /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:361:4
    #23 0x55f819db2c26 in doris::BetaRowsetWriter::flush_single_memtable(doris::vectorized::Block const*, long*, doris::FlushContext const*) /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/beta_rowset_writer.cpp:505:1
    #24 0x55f81a64fd05 in doris::MemTable::_do_flush() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/memtable.cpp:533:5
    #25 0x55f81a64e7f4 in doris::MemTable::flush() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/memtable.cpp:496:5
    #26 0x55f81a62dd84 in doris::FlushToken::_flush_memtable(doris::MemTable*, long) /mnt/disk2/luen/develop/workspace/doris/be/src/olap/memtable_flush_executor.cpp:116:26
    #27 0x55f81a63feee in doris::MemtableFlushTask::run() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/memtable_flush_executor.cpp:54:23
    #28 0x55f81b8ea036 in doris::ThreadPool::dispatch_thread() /mnt/disk2/luen/develop/workspace/doris/be/src/util/threadpool.cpp:533:24
    #29 0x55f81b90f3c3 in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14

previously allocated by thread T1687 (MemTableFlushTh) here:
    #0 0x55f81825953d in operator new(unsigned long) (/mnt/disk2/luen/mydoris2/be/lib/doris_be+0x1a3d053d) (BuildId: cc86a6484331667c)
    #1 0x55f81a03a6c3 in doris::segment_v2::DorisRAMCompoundDirectory::init(std::shared_ptr<doris::io::FileSystem> const&, char const*, bool, lucene::store::LockFactory*, std::shared_ptr<doris::io::FileSystem> const&, char const*) /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/segment_v2/inverted_index_compound_directory.cpp:724:46
    #2 0x55f81a02c371 in doris::segment_v2::DorisCompoundDirectoryFactory::getDirectory(std::shared_ptr<doris::io::FileSystem> const&, char const*, bool, bool, lucene::store::LockFactory*, std::shared_ptr<doris::io::FileSystem> const&, char const*) /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/segment_v2/inverted_index_compound_directory.cpp:912:10
    #3 0x55f81a3bb4b2 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)17>::init_fulltext_index() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:191:20
    #4 0x55f81a3b5597 in doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)17>::init() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:98:24
    #5 0x55f81a37547c in doris::segment_v2::InvertedIndexColumnWriter::create(doris::Field const*, std::unique_ptr<doris::segment_v2::InvertedIndexColumnWriter, std::default_delete<doris::segment_v2::InvertedIndexColumnWriter>>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, doris::TabletIndex const*, std::shared_ptr<doris::io::FileSystem> const&) /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:744:27
    #6 0x55f81a2f9813 in doris::segment_v2::ScalarColumnWriter::init() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:522:13
    #7 0x55f81a2beefe in auto doris::segment_v2::SegmentWriter::init(std::vector<unsigned int, std::allocator<unsigned int>> const&, bool, doris::FlushContext const*)::$_0::operator()<doris::TabletColumn>(unsigned int, doris::TabletColumn const&) const /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:256:9
    #8 0x55f81a2bce4e in doris::Status std::__invoke_impl<doris::Status, doris::segment_v2::SegmentWriter::init(std::vector<unsigned int, std::allocator<unsigned int>> const&, bool, doris::FlushContext const*)::$_0&, unsigned int, doris::TabletColumn const&>(std::__invoke_other, doris::segment_v2::SegmentWriter::init(std::vector<unsigned int, std::allocator<unsigned int>> const&, bool, doris::FlushContext const*)::$_0&, unsigned int&&, doris::TabletColumn const&) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
    #9 0x55f81a2bcd8f in std::enable_if<is_invocable_r_v<doris::Status, doris::segment_v2::SegmentWriter::init(std::vector<unsigned int, std::allocator<unsigned int>> const&, bool, doris::FlushContext const*)::$_0&, unsigned int, doris::TabletColumn const&>, doris::Status>::type std::__invoke_r<doris::Status, doris::segment_v2::SegmentWriter::init(std::vector<unsigned int, std::allocator<unsigned int>> const&, bool, doris::FlushContext const*)::$_0&, unsigned int, doris::TabletColumn const&>(doris::segment_v2::SegmentWriter::init(std::vector<unsigned int, std::allocator<unsigned int>> const&, bool, doris::FlushContext const*)::$_0&, unsigned int&&, doris::TabletColumn const&) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:114:9
    #10 0x55f81a2bcc3f in std::_Function_handler<doris::Status (unsigned int, doris::TabletColumn const&), doris::segment_v2::SegmentWriter::init(std::vector<unsigned int, std::allocator<unsigned int>> const&, bool, doris::FlushContext const*)::$_0>::_M_invoke(std::_Any_data const&, unsigned int&&, doris::TabletColumn const&) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
    #11 0x55f81a2c9b8a in std::function<doris::Status (unsigned int, doris::TabletColumn const&)>::operator()(unsigned int, doris::TabletColumn const&) const /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
    #12 0x55f81a2a7ae5 in doris::segment_v2::SegmentWriter::_create_writers(doris::TabletSchema const&, std::vector<unsigned int, std::allocator<unsigned int>> const&, std::function<doris::Status (unsigned int, doris::TabletColumn const&)>) /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:293:9
    #13 0x55f81a2a71c0 in doris::segment_v2::SegmentWriter::init(std::vector<unsigned int, std::allocator<unsigned int>> const&, bool, doris::FlushContext const*) /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:266:9
    #14 0x55f81a2a639d in doris::segment_v2::SegmentWriter::init(doris::FlushContext const*) /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:135:12
    #15 0x55f819db98b9 in doris::BetaRowsetWriter::_do_create_segment_writer(std::unique_ptr<doris::segment_v2::SegmentWriter, std::default_delete<doris::segment_v2::SegmentWriter>>*, bool, long, long, doris::FlushContext const*) /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/beta_rowset_writer.cpp:775:29
    #16 0x55f819da7503 in doris::BetaRowsetWriter::_create_segment_writer(std::unique_ptr<doris::segment_v2::SegmentWriter, std::default_delete<doris::segment_v2::SegmentWriter>>*, doris::FlushContext const*) /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/beta_rowset_writer.cpp:800:16
    #17 0x55f819db22f9 in doris::BetaRowsetWriter::flush_single_memtable(doris::vectorized::Block const*, long*, doris::FlushContext const*) /mnt/disk2/luen/develop/workspace/doris/be/src/olap/rowset/beta_rowset_writer.cpp:489:5
    #18 0x55f81a64fd05 in doris::MemTable::_do_flush() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/memtable.cpp:533:5
    #19 0x55f81a64e7f4 in doris::MemTable::flush() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/memtable.cpp:496:5
    #20 0x55f81a62dd84 in doris::FlushToken::_flush_memtable(doris::MemTable*, long) /mnt/disk2/luen/develop/workspace/doris/be/src/olap/memtable_flush_executor.cpp:116:26
    #21 0x55f81a63feee in doris::MemtableFlushTask::run() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/memtable_flush_executor.cpp:54:23
    #22 0x55f81b8ea036 in doris::ThreadPool::dispatch_thread() /mnt/disk2/luen/develop/workspace/doris/be/src/util/threadpool.cpp:533:24
    #23 0x55f81b90f3c3 in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14
    #24 0x55f81b90f29c in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
    #25 0x55f81b90f224 in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::__call<void, 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:420:11
    #26 0x55f81b90f0cd in void std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>::operator()<void>() /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:503:17
    #27 0x55f81b90efe4 in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
    #28 0x55f81b90ef84 in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>&) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
    #29 0x55f81b90ec2c in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::* (doris::ThreadPool*))()>>::_M_invoke(std::_Any_data const&) /mnt/disk2/luen/develop/tools/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9

Thread T1687 (MemTableFlushTh) created by T0 here:
    #0 0x55f818206caa in pthread_create (/mnt/disk2/luen/mydoris2/be/lib/doris_be+0x1a37dcaa) (BuildId: cc86a6484331667c)
    #1 0x55f81b8bc2a4 in doris::Thread::start_thread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::function<void ()> const&, unsigned long, scoped_refptr<doris::Thread>*) /mnt/disk2/luen/develop/workspace/doris/be/src/util/thread.cpp:452:15
    #2 0x55f81b8f457d in doris::Status doris::Thread::create<void (doris::ThreadPool::*)(), doris::ThreadPool*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, void (doris::ThreadPool::* const&)(), doris::ThreadPool* const&, scoped_refptr<doris::Thread>*) /mnt/disk2/luen/develop/workspace/doris/be/src/util/thread.h:56:16
    #3 0x55f81b8e77fd in doris::ThreadPool::create_thread() /mnt/disk2/luen/develop/workspace/doris/be/src/util/threadpool.cpp:601:12
    #4 0x55f81b8e72db in doris::ThreadPool::init() /mnt/disk2/luen/develop/workspace/doris/be/src/util/threadpool.cpp:259:25
    #5 0x55f818271786 in doris::Status doris::ThreadPoolBuilder::build<doris::ThreadPool>(std::unique_ptr<doris::ThreadPool, std::default_delete<doris::ThreadPool>>*) const /mnt/disk2/luen/develop/workspace/doris/be/src/util/threadpool.h:119:13
    #6 0x55f81a62e9e0 in doris::MemTableFlushExecutor::init(std::vector<doris::DataDir*, std::allocator<doris::DataDir*>> const&) /mnt/disk2/luen/develop/workspace/doris/be/src/olap/memtable_flush_executor.cpp:151:14
    #7 0x55f81a59912c in doris::StorageEngine::_open() /mnt/disk2/luen/develop/workspace/doris/be/src/olap/storage_engine.cpp:221:31
    #8 0x55f81a598214 in doris::StorageEngine::open(doris::EngineOptions const&, doris::StorageEngine**) /mnt/disk2/luen/develop/workspace/doris/be/src/olap/storage_engine.cpp:113:5
    #9 0x55f818261ed4 in main /mnt/disk2/luen/develop/workspace/doris/be/src/service/doris_main.cpp:483:15
    #10 0x7fc4eb5987e4 in __libc_start_main (/lib64/libc.so.6+0x3a7e4) (BuildId: 6ae06e3b90805d95009d52783e9b82fe5f92c5a5)

SUMMARY: AddressSanitizer: heap-use-after-free /mnt/disk2/luen/develop/workspace/doris/be/src/clucene/src/shared/CLucene/config/threads.cpp:235:3 in lucene::util::mutex_thread::lock()
Shadow bytes around the buggy address:
  0x606000a83600: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
  0x606000a83680: 00 00 00 00 00 00 00 00 fa fa fa fa 00 00 00 00
  0x606000a83700: 00 00 07 fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x606000a83780: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
  0x606000a83800: 00 00 00 00 00 00 00 05 fa fa fa fa fd fd fd fd
=>0x606000a83880: fd fd fd fd fa fa fa fa fd fd fd fd fd fd[fd]fa
  0x606000a83900: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
  0x606000a83980: 00 00 00 00 00 00 00 fa fa fa fa fa fd fd fd fd
  0x606000a83a00: fd fd fd fd fa fa fa fa 00 00 00 00 00 00 00 00
  0x606000a83a80: fa fa fa fa fd fd fd fd fd fd fd fa fa fa fa fa
  0x606000a83b00: fd fd fd fd fd fd fd fa fa fa fa fa 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==1757461==ABORTING

What You Expected?

load normal

How to Reproduce?

ASAN build.
./run-regression-test.sh --run -d fault_injection_p0 -s test_index_compound_directory_failure_injection

Anything Else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@qidaye