bitcoin-node segfaults when interrupted #123
Comments
|
(continued) Here is some logs. Logs for just starting and stopping `bitcoin-node`Logs for when the client constructs `Init`, creates a thread, and stops |
|
I am able to reproduce this pretty easily with a local branch (ryanofsky/bitcoin@e944fe0) running If I just stop the But if I press CTRL-C in the terminal which sends Could also reproduce this in GDB by running details
2024-12-04T02:38:33Z [ipc] {bitcoin-node-4009464/b-capnp-loop-4009469} IPC client: unexpected network disconnect.
[New Thread 0x7fffa97fa6c0 (LWP 4009654)]
terminate called after throwing an instance of 'kj::ExceptionImpl'
what(): /nix/store/0rvzw1dahhd1h8qyhyxmwgr2l745fn3a-capnproto-1.0.2/include/kj/memory.h:258: failed: expected ptr != nullptr; null Own<> dereference
stack: 555557d3fe3d 555555eb3520 55555662980c 55555661e849 555556545220 555556545115 555556545075 55555654502d 555556544fd5 555556544f25 555556544dbd 555557bd7204 555557bd49e8 5555565f1eef 5555565f1d91 5555565f1cf1 5555565f1ca9 5555565f1c51 5555565f1af9 7ffff78e8682 7ffff767d382 7ffff770000b
Thread 3 "b-capnp-loop" received signal SIGABRT, Aborted.
0x00007ffff767f07c in __pthread_kill_implementation () from /nix/store/1rm6sr6ixxzipv5358x0cmaw8rs84g2j-glibc-2.38-44/lib/libc.so.6
(gdb) bt
#0 0x00007ffff767f07c in __pthread_kill_implementation () from /nix/store/1rm6sr6ixxzipv5358x0cmaw8rs84g2j-glibc-2.38-44/lib/libc.so.6
#1 0x00007ffff762fe06 in raise () from /nix/store/1rm6sr6ixxzipv5358x0cmaw8rs84g2j-glibc-2.38-44/lib/libc.so.6
#2 0x00007ffff76188f5 in abort () from /nix/store/1rm6sr6ixxzipv5358x0cmaw8rs84g2j-glibc-2.38-44/lib/libc.so.6
#3 0x00007ffff78acc0b in __gnu_cxx::__verbose_terminate_handler() [clone .cold] () from /nix/store/agp6lqznayysqvqkx4k1ggr8n1rsyi8c-gcc-13.2.0-lib/lib/libstdc++.so.6
#4 0x00007ffff78bc21a in __cxxabiv1::__terminate(void (*)()) () from /nix/store/agp6lqznayysqvqkx4k1ggr8n1rsyi8c-gcc-13.2.0-lib/lib/libstdc++.so.6
#5 0x00007ffff78bc285 in std::terminate() () from /nix/store/agp6lqznayysqvqkx4k1ggr8n1rsyi8c-gcc-13.2.0-lib/lib/libstdc++.so.6
#6 0x00007ffff78bc4d7 in __cxa_throw () from /nix/store/agp6lqznayysqvqkx4k1ggr8n1rsyi8c-gcc-13.2.0-lib/lib/libstdc++.so.6
#7 0x00005555556cef57 in kj::ExceptionCallback::RootExceptionCallback::onFatalException(kj::Exception&&) ()
#8 0x0000555557d44765 in kj::throwFatalException(kj::Exception&&, unsigned int) ()
#9 0x0000555557d405a3 in kj::_::Debug::Fault::fatal() ()
#10 0x0000555557d3fe3e in kj::_::inlineRequireFailure(char const*, int, char const*, char const*, char const*) ()
#11 0x0000555555eb3521 in kj::Own<capnp::ClientHook, decltype(nullptr)>::operator->() (this=0x7fffe8019760) at /nix/store/0rvzw1dahhd1h8qyhyxmwgr2l745fn3a-capnproto-1.0.2/include/kj/memory.h:258
#12 0x000055555662980d in capnp::Capability::Client::newCall<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (this=0x7fffe8019760,
interfaceId=16225999917376910712, methodId=5, sizeHint=..., hints=...) at /nix/store/0rvzw1dahhd1h8qyhyxmwgr2l745fn3a-capnproto-1.0.2/include/capnp/capability.h:1113
#13 0x000055555661e84a in ipc::capnp::messages::ChainClient::Client::flushRequest (this=0x7fffe8019758, sizeHint=...) at ./build/src/ipc/./build/src/ipc/capnp/chain.capnp.c++:13499
#14 0x0000555556545221 in mp::clientInvoke<mp::ProxyClient<ipc::capnp::messages::WalletLoader>, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>> >(mp::ProxyClient<ipc::capnp::messages::WalletLoader>&, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>>&&)::{lambda()#1}::operator()() const (this=0x7ffffffd24a8) at /home/russ/work/mp/build/prefix/include/mp/proxy-types.h:1575
#15 0x0000555556545116 in std::__invoke_impl<void, mp::clientInvoke<mp::ProxyClient<ipc::capnp::messages::WalletLoader>, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>> >(mp::ProxyClient<ipc::capnp::messages::WalletLoader>&, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>>&&)::{lambda()#1}&>(std::__invoke_other, mp::clientInvoke<mp::ProxyClient<ipc::capnp::messages::WalletLoader>, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>> >(mp::ProxyClient<ipc::capnp::messages::WalletLoader>&, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>>&&)::{lambda()#1}&) (__f=...)
at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/bits/invoke.h:61
#16 0x0000555556545076 in std::__invoke<mp::clientInvoke<mp::ProxyClient<ipc::capnp::messages::WalletLoader>, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>> >(mp::ProxyClient<ipc::capnp::messages::WalletLoader>&, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>>&&)::{lambda()#1}&>(mp::ProxyClient<ipc::capnp::messages::WalletLoader>&&) (__fn=...)
at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/bits/invoke.h:96
#17 0x000055555654502e in std::reference_wrapper<mp::clientInvoke<mp::ProxyClient<ipc::capnp::messages::WalletLoader>, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>> >(mp::ProxyClient<ipc::capnp::messages::WalletLoader>&, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>>&&)::{lambda()#1}>::operator()<>() const (this=0x7ffffffd23e8)
at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/bits/refwrap.h:359
#18 0x0000555556544fd6 in std::__invoke_impl<void, std::reference_wrapper<mp::clientInvoke<mp::ProxyClient<ipc::capnp::messages::WalletLoader>, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>> >(mp::ProxyClient<ipc::capnp::messages::WalletLoader>&, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::* const&)(kj::Maybe<capn--Type <RET> for more, q to quit, c to continue without paging--c
p::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>>&&)::{lambda()#1}>&>(std::__invoke_other, std::reference_wrapper<mp::clientInvoke<mp::ProxyClient<ipc::capnp::messages::WalletLoader>, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>> >(mp::ProxyClient<ipc::capnp::messages::WalletLoader>&, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>>&&)::{lambda()#1}>&) (__f=...)
at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/bits/invoke.h:61
#19 0x0000555556544f26 in std::__invoke_r<void, std::reference_wrapper<mp::clientInvoke<mp::ProxyClient<ipc::capnp::messages::WalletLoader>, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>> >(mp::ProxyClient<ipc::capnp::messages::WalletLoader>&, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>>&&)::{lambda()#1}>&>(std::reference_wrapper<mp::clientInvoke<mp::ProxyClient<ipc::capnp::messages::WalletLoader>, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>> >(mp::ProxyClient<ipc::capnp::messages::WalletLoader>&, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>>&&)::{lambda()#1}>&) (__fn=...)
at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/bits/invoke.h:111
#20 0x0000555556544dbe in std::_Function_handler<void (), std::reference_wrapper<mp::clientInvoke<mp::ProxyClient<ipc::capnp::messages::WalletLoader>, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>> >(mp::ProxyClient<ipc::capnp::messages::WalletLoader>&, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>>&&)::{lambda()#1}> >::_M_invoke(std::_Any_data const&) (__functor=...)
at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/bits/std_function.h:290
#21 0x0000555557bd7205 in std::function<void ()>::operator()() const (this=0x7ffffffd23e8) at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/bits/std_function.h:591
#22 mp::Unlock<std::unique_lock<std::mutex>, std::function<void ()> const&>(std::unique_lock<std::mutex>&, std::function<void ()> const&) (lock=..., callback=...) at /home/russ/work/mp/include/mp/util.h:146
#23 0x0000555557bd49e9 in mp::EventLoop::loop (this=0x5555593b2840) at /home/russ/work/mp/src/mp/proxy.cpp:197
#24 0x00005555565f1ef0 in ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::{lambda()#1}::operator()() const (this=0x5555593b9578) at ./build/src/ipc/./src/ipc/capnp/protocol.cpp:91
#25 0x00005555565f1d92 in std::__invoke_impl<void, ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::{lambda()#1}>(std::__invoke_other, ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::{lambda()#1}&&) (__f=...) at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/bits/invoke.h:61
#26 0x00005555565f1cf2 in std::__invoke<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::{lambda()#1}>(ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::{lambda()#1}&&) (__fn=...) at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/bits/invoke.h:96
#27 0x00005555565f1caa in std::thread::_Invoker<std::tuple<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x5555593b9578)
at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/bits/std_thread.h:292
#28 0x00005555565f1c52 in std::thread::_Invoker<std::tuple<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::{lambda()#1}> >::operator()() (this=0x5555593b9578)
at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/bits/std_thread.h:299
#29 0x00005555565f1afa in std::thread::_State_impl<std::thread::_Invoker<std::tuple<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::{lambda()#1}> > >::_M_run() (this=0x5555593b9570)
at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/bits/std_thread.h:244
#30 0x00007ffff78e8683 in execute_native_thread_routine () from /nix/store/agp6lqznayysqvqkx4k1ggr8n1rsyi8c-gcc-13.2.0-lib/lib/libstdc++.so.6
#31 0x00007ffff767d383 in start_thread () from /nix/store/1rm6sr6ixxzipv5358x0cmaw8rs84g2j-glibc-2.38-44/lib/libc.so.6
#32 0x00007ffff770000c in clone3 () from /nix/store/1rm6sr6ixxzipv5358x0cmaw8rs84g2j-glibc-2.38-44/lib/libc.so.6Probably the simplest way to fix this would be to make the But additionally it would make sense to identify cause of the |
The cause seems to be a bug in the The crash happens because bitcoin-node main thread is trying to call Stack trace for the main thread at the time of the crash looks like: trace
#0 0x00007ffff7679ebe in __futex_abstimed_wait_common () from /nix/store/1rm6sr6ixxzipv5358x0cmaw8rs84g2j-glibc-2.38-44/lib/libc.so.6
#1 0x00007ffff767c720 in pthread_cond_wait@@GLIBC_2.3.2 () from /nix/store/1rm6sr6ixxzipv5358x0cmaw8rs84g2j-glibc-2.38-44/lib/libc.so.6
#2 0x0000555557bd4ebb in std::condition_variable::wait<mp::EventLoop::post(std::function<void ()> const&)::$_2>(std::unique_lock<std::mutex>&, mp::EventLoop::post(std::function<void ()> const&)::$_2) (
this=0x5555593b28b0, __lock=..., __p=...) at /nix/store/rqga421d43q40blrrgmiw820p01a4nba-gcc-13.2.0/include/c++/13.2.0/condition_variable:105
#3 mp::EventLoop::post(std::function<void ()> const&) (this=0x5555593b2840, fn=...) at /home/russ/work/mp/src/mp/proxy.cpp:229
#4 0x0000555556544a24 in mp::EventLoop::sync<mp::clientInvoke<mp::ProxyClient<ipc::capnp::messages::WalletLoader>, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>> >(mp::ProxyClient<ipc::capnp::messages::WalletLoader>&, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>>&&)::{lambda()#1}>(mp::ProxyClient<ipc::capnp::messages::WalletLoader>&&) (this=0x5555593b2840, callable=...)
at /home/russ/work/mp/build/prefix/include/mp/proxy-io.h:155
#5 0x00005555561df084 in mp::clientInvoke<mp::ProxyClient<ipc::capnp::messages::WalletLoader>, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::*)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>> >(mp::ProxyClient<ipc::capnp::messages::WalletLoader>&, capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::* const&)(kj::Maybe<capnp::MessageSize>), mp::ClientParam<mp::Accessor<mp::wallet_fields::Context, 17>>&&) (proxy_client=...,
get_request=@0x7ffffffd28a8: (capnp::Request<ipc::capnp::messages::ChainClient::FlushParams, ipc::capnp::messages::ChainClient::FlushResults> (ipc::capnp::messages::ChainClient::Client::*)(ipc::capnp::messages::ChainClient::Client * const, kj::Maybe<capnp::MessageSize>)) 0x55555661e7c0 <ipc::capnp::messages::ChainClient::Client::flushRequest(kj::Maybe<capnp::MessageSize>)>, fields=...)
at /home/russ/work/mp/build/prefix/include/mp/proxy-types.h:1574
#6 0x00005555561aeabe in mp::ProxyClient<ipc::capnp::messages::WalletLoader>::flush (this=0x7fffe8019750) at ./build/src/ipc/./build/src/ipc/capnp/wallet.capnp.proxy-client.c++:404
#7 0x0000555555706713 in Shutdown (node=...) at ./build/src/./src/init.cpp:290
#8 0x00005555556f6522 in main (argc=2, argv=0x7ffffffd2e68) at ./build/src/./src/bitcoind.cpp:283 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Originally posted by @darosior in bitcoin/bitcoin#10102 (comment)
Not sure if it's worth sharing here but hey what's an 1300th comment.
Running
bitcoin-nodefrom bitcoin/bitcoin#29409 with commits1045047a73a0b3ae15647ab5bf1829d27141b7cd..9ca113f326fc9ba2f661595283f4aef390df984dfrom this PR cherry-picked on top.Simply compiled with
cmake -B multiprocbuild/ -DWITH_MULTIPROCESS=ON && cmake --build multiprocbuild/ -j20. (Although i did have to apply the following diff to be able to compile on Debian stable.)Expand diff to compile on Debian stable
Just starting
bitcoin-nodeand stopping it with aSIGKILLresults in the following crash:The command i used to start it is
./multiprocbuild/src/bitcoin-node -regtest -ipcbind=unix -debug=ipc.With a simple hacked up client which constructs an
Initinterface and stops, the crash becomes (still only when stoppingbitcoin-node):Now if the client requested a thread before stopping, this becomes a segfault (still only when stopping
bitcoin-node):The text was updated successfully, but these errors were encountered: