[BOLT][AArch64] unsupported CFI opcode on Ubuntu 24.04 and AL2023

[salvatoredipietro]

Description

When attempting to use BOLT on Amazon Linux 2023 (AL2023) and Ubuntu24.04 using an AArch64 instance (AWS m7g.4xlarge), llvm-bolt command fails with "unsupported CFI opcode" error. This happen especially when I am trying to use long perf profile ( > 120 seconds).

Environment

• Operating System: Amazon Linux 2023
• Hardware: AWS m7g.4xlarge instance (ARM-based)
• BOLT version: d33a2c5
• Application used: PostgreSQL (build locally)

Steps to Reproduce

1. Compile Postgres 16.4:

export CFLAGS="-g -O2 -fstack-protector-strong -fno-omit-frame-pointer -moutline-atomics -fno-reorder-blocks-and-partition"
export CPPFLAGS="-D_FORTIFY_SOURCE=2"
export LDFLAGS="-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,--emit-relocs"

## Compile it
./configure --disable-rpath --enable-debug --enable-dtrace --enable-tap-tests --enable-thread-safety --with-gnu-ld --with-icu --with-tcl --with-perl --with-python --with-openssl --with-libxml --with-libxslt --prefix=\${HOME}/usr/  
make -j\$(nproc)
sudo make install

2. Build BOLT on Amazon Linux 2023:

# BOLT  Installation on AL2023
sudo yum install  -y perf cmake ninja-build 
git clone https://github.com/llvm/llvm-project.git  
mkdir build   &&  cd build  
cmake -G Ninja ../llvm-project/llvm -DLLVM_TARGETS_TO_BUILD="X86;AArch64" -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PROJECTS="bolt"
ninja bolt

3. Run the following command:

# Get perf information
sudo perf record -e cycles:u  -u postgres -o perf.data -a -- sleep 600      

# Run perf2bolt cmd
sudo ~/build/bin/perf2bolt -p perf.data -o perf.boltdata --nl ${HOME}/usr/bin/postgres

# Run BOLT
sudo ~/build/bin/llvm-bolt ${HOME}/usr/bin/postgres -o ${HOME}/usr/bin/postgres.bolt --data perf.boltdata --reorder-blocks ext-tsp --reorder-functions hfsort --split-functions --split-all-cold --split-eh --update-debug-sections --dyno-stats --print-profile-stats  

Error Message

BOLT-INFO: Starting stub-insertion pass
BOLT-INFO: Inserted 470 stubs in the hot area and 216 stubs in the cold area. Shared 0 times, iterated 3 times.
unsupported CFI opcode
UNREACHABLE executed at /home/ec2-user/llvm-project/bolt/lib/Core/BinaryFunction.cpp:2591!
 #0 0x0000000000e3d810 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/ec2-user/build/bin/llvm-bolt+0xe3d810)
 #1 0x0000000000e3b77c llvm::sys::RunSignalHandlers() (/home/ec2-user/build/bin/llvm-bolt+0xe3b77c)
 #2 0x0000000000e3b8a8 SignalHandler(int) Signals.cpp:0:0
 #3 0x0000ffffbe05d830 (linux-vdso.so.1+0x830)
 #4 0x0000ffffbda83554 __pthread_kill_implementation (/lib64/libc.so.6+0x92554)
 #5 0x0000ffffbda3a3e0 gsignal (/lib64/libc.so.6+0x493e0)
 #6 0x0000ffffbda21224 abort (/lib64/libc.so.6+0x30224)
 #7 0x0000000000dda54c (/home/ec2-user/build/bin/llvm-bolt+0xdda54c)
 #8 0x00000000015d5a44 llvm::bolt::(anonymous namespace)::CFISnapshot::advanceTo(int) BinaryFunction.cpp:0:0
 #9 0x00000000015d80b0 llvm::bolt::BinaryFunction::unwindCFIState(int, int, llvm::bolt::BinaryBasicBlock*, __gnu_cxx::__normal_iterator<llvm::MCInst*, std::vector<llvm::MCInst, std::allocator<llvm::MCInst>>>&) (/home/ec2-user/build/bin/llvm-bolt+0x15d80b0)
#10 0x00000000015da5e0 llvm::bolt::BinaryFunction::finalizeCFIState() (/home/ec2-user/build/bin/llvm-bolt+0x15da5e0)
#11 0x000000000147a15c std::_Function_handler<void (llvm::bolt::BinaryFunction&), llvm::bolt::FinalizeFunctions::runOnFunctions(llvm::bolt::BinaryContext&)::'lambda'(llvm::bolt::BinaryFunction&)>::_M_invoke(std::_Any_data const&, llvm::bolt::BinaryFunction&) BinaryPasses.cpp:0:0
#12 0x00000000016345c8 std::_Function_handler<void (), std::_Bind<llvm::bolt::ParallelUtilities::runOnEachFunction(llvm::bolt::BinaryContext&, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function<void (llvm::bolt::BinaryFunction&)>, std::function<bool (llvm::bolt::BinaryFunction const&)>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, bool, unsigned int)::'lambda'(std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>) (std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>)>>::_M_invoke(std::_Any_data const&) ParallelUtilities.cpp:0:0
#13 0x0000000000f1ff68 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<std::function<void ()>>>, void>>::_M_invoke(std::_Any_data const&) (/home/ec2-user/build/bin/llvm-bolt+0xf1ff68)
#14 0x0000000000f2074c std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) (/home/ec2-user/build/bin/llvm-bolt+0xf2074c)
#15 0x0000ffffbda86980 __pthread_once_slow (/lib64/libc.so.6+0x95980)
#16 0x0000000000f205bc std::__future_base::_Deferred_state<std::thread::_Invoker<std::tuple<std::function<void ()>>>, void>::_M_complete_async() (/home/ec2-user/build/bin/llvm-bolt+0xf205bc)
#17 0x0000000000f24eec std::_Function_handler<void (), std::shared_future<void> llvm::ThreadPoolInterface::asyncImpl<void>(std::function<void ()>, llvm::ThreadPoolTaskGroup*)::'lambda'()>::_M_invoke(std::_Any_data const&) (/home/ec2-user/build/bin/llvm-bolt+0xf24eec)
#18 0x00000000023f057c llvm::StdThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (/home/ec2-user/build/bin/llvm-bolt+0x23f057c)
#19 0x00000000023f1104 void* llvm::thread::ThreadProxy<std::tuple<llvm::StdThreadPool::grow(int)::'lambda'()>>(void*) ThreadPool.cpp:0:0
#20 0x0000ffffbda81934 start_thread (/lib64/libc.so.6+0x90934)
#21 0x0000ffffbda25e5c thread_start (/lib64/libc.so.6+0x34e5c)
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Aborted