libstdc++.so.6 is outdated and breaks RCall and others on recent Linux distros #34276
Comments
laborg
changed the title
|
My plan to address this is:
|
|
@staticfloat You already merged a couple PRs from your plan - how far are you away from completing this? Thanks |
|
Making good progress! The new CompilerSupportLibraries_jll enacts my plan, and I've been cranking away on #33973 which will ship it by default with Julia. |
|
I am not sure I understand how "always shipping the latest libstdc++" is gonna work, though? Suppose your fix is deployed in Julia 1.5, and suppose for the sake of the argument this is the new LTS. Then is the plan to keep releasing updates to 1.5.x promptly whenever a new libstdc++ is released, until it is no more the LTS version? And then everybody who updates GCC on their computer must remember to also update Julia at the same time, else things break down? Hmm, to be honest, I have a hard time seeing how that will be viable in practice. Quite possible I am missing part of the plan, though. Now, if all packages in Julia that build or link external code where to consequently use a C/C++ compiler installed via Julia, instead of a system compiler, that might work. But as soon as any external code is pulled in, there is a chance it requires a newer C++ library than what Julia has, no? |
|
You're correct in that there is a tension between Julia's precompiled binaries and system-compiled binaries. We have to work to balance two competing interests:
There are a lot of ways that something can be incompatible, and we do our best to deal with all of them. In the case of So the only corner case that is difficult in this scenario is when the user custom-compiles a library with their system compiler, and the system compiler is newer than what will ship with Julia. Luckily, both of these situations are rare; fewer and fewer users are compiling their own libraries for use in Julia (a trend we definitely want to encourage, as it reduces package installation errors dramatically) and for those that do, us shipping a more recent libstdc++ with Julia will allow a broader range of compiler support. We do not yet have a way to "update" the libraries that ship with Julia easily, but in the future, we may have a method by which a user can rebuild their system image and incorporate newer versions of the libraries that ship with Julia, thereby updating the libstdc++ that is shipped with Julia. That's speculative at this point though. |
laborg
changed the title
|
I wanted to add that this appeared in a completely different context. After an update of CImGui.jl my application segfaulted. Setting the environment variable ´LIBGL_DEBUG´ showed that this is again is due to the libstdc++ mismatch. @Gnimuc , this might be of interest for you. |
|
Is there a workaround for this? |
|
@ambiso The workaround is linked in the first post:
|
|
ubuntu Release 20.04 LTS (Focal Fossa) 64-bit
I had a crashed update on ubuntu 16.04 and had to clear system disc and start over. |
This workaround now segfaults for me on Julia 1.5.0 (official x64 binary) when running: with GLFW 3.3.2. Is there a roadmap to get this issue fixed? E.g. building with a non-legacy |
|
The segfault happens here: [#0] 0x7fffbc521d2d → llvm::TargetPassConfig::addPass(llvm::Pass*, bool, bool)()
[#1] 0x7fffbc524bf8 → llvm::TargetPassConfig::addRegAssignmentOptimized()()
[#2] 0x7fffbc524fbe → llvm::TargetPassConfig::addOptimizedRegAlloc()()
[#3] 0x7fffbc52521d → llvm::TargetPassConfig::addMachinePasses()()
[#4] 0x7fffbc283152 → mov rax, r12
[#5] 0x7fffbc287e41 → llvm::LLVMTargetMachine::addPassesToEmitFile(llvm::legacy::PassManagerBase&, llvm::raw_pwrite_stream&, llvm::raw_pwrite_stream*, llvm::CodeGenFileType, bool, llvm::MachineModuleInfoWrapperPass*)()
[#6] 0x7fffc1213c00 → pop rdx
[#7] 0x7fffc116cd5a → mov rdi, QWORD PTR [rbx+0x20]
[#8] 0x7fffc11ae03b → jmp 0x7fffc11adc97
[#9] 0x7fffc0e5d9c8 → xor eax, eaxinside of |
|
LLVM 10? |
|
Yes. Does Julia require a legacy version of LLVM? |
|
Julia 1.5.0 should not be using LLVM 10; that segfault sounds like you are somehow forcing Julia to use a different LLVM than the one it ships with, which will cause segfaults; LLVM is not compatible across versions. |
|
You're right according to $ ldd `which julia`
# ...
libLLVM-9jl.so => /usr/bin/../lib/julia/libLLVM-9jl.so
# ...I was thinking this could maybe be a misattribution of gdb but at least libLLVM-10 is loaded as can be seen from Here is the full information from gdb |
|
Are you setting |
|
No. See this long gdb log |
|
Perhaps GLFW or ModernGL is bringing in LLVM somehow. Can you run inside of |
|
Looks like you're right. In the beginning it's loading LLVM9But then after loading graphics drivers it seems to be loading LLVM-10 in line 718, possibly for shader compilation? |
|
Hmmmmm. Yeah, this is unfortunate. Can you open a new issue with this information, it's separate from this thread. |
|
Note that this is only the case when using |
|
It might be worthwhile to reopen this issue (and maybe search for a better fix than simply updating the libstdc++ version). It is again happening (e.g. on nightly). I am trying to use GLMakie which seems to depend on GLFW which faces this issue [1] on Ubuntum 22.04. Deleting the shipped libstdc++ fixes the problem. The nightly on which I observed this was installed with juliaup's fork [2]. It see someone is also asking whether the symlinking can be done by juliaup itself [3] 1: JuliaGL/GLFW.jl#198 I apologize if I have misunderstood the status of this issue though. I imagine there is a good reason to not fix it, but I did not immediately see it while skimming through the comments. |
|
I am a bit confused as to why these issues keep getting closed too. Yes, they will get fixed by upgrading Julia when Julia updates libstdc++, but will immediately break when the system libstdc++ is upgraded again. There are other options than trying to keep libstdc++ pointing at the newest out of the bundled version and the system version (which I still think is an approach that could be made to work). Another option is that Julia could standardise that all Julia itself and all binary packages to be built with an old glibc which is the manylinux from Python works https://github.com/pypa/manylinux . Since we generally have forward compatibility from build version => run version, libstdc++ could not be bundled at all but instead made a run time dependency. This approach can turn into a bit of a pain in terms of deprecation schedules --- it essentially means all libraries have to keep building on old versions of GCC. Another option is for special sub-point releases for Julia Linux binaries only to be made (automatically) as soon as new versions of libstdc++ become available, and users poked to upgrade to these whenever these types of errors happen. (Is there some automated way of catching these errors?) I guess this may be preferable to automating the symlink workaround. I do think the current situation is quite bad for some users. Users of rolling release Linux distributions are not so uncommon these days, and while they are likely to be able to do the Googling to work around things themselves, this is friction that might be avoided. |
This is already done and also misses the point of the issue entirely: problems happen when some system libraries built with recent versions of GCC are involved |
|
Hi @giordano. Thanks for the response. I do actually understand that part of the issue. I will admit I am not an expert on all the details. I don't completely understand why certain taboos exist, e.g. against implementing what is the standard workaround in juliaup. I hope you can be patient with me because I believe (perhaps naively) that we might be able to come to some reasonable solution together. Could you please tell me: if everything is built against a sufficiently old libstdc++, why is a new libstdc++ shipped, when we could instead assume that a newer libstdc++ is available from the system? |
Because that's a pretty wrong assumption. Have you ever tried to run software on a cluster which ships GCC 4.8? |
|
I too have trouble understanding the entirety of the problem and current solution. Currently I do need to delete a file in julia/lib in order to use a number of Julia libraries. I currently do not know which one of these is true:
I understand that a lot of work has already been poured into this (including by volunteers working for free) and I really appreciate it. I just hope I can understand this issue (that is quite frequently affecting me) better and this is the only venue where I can ask. |
|
Currently it's "2". I actually already opened a PR almost four months ago to get libstdc++ from GCC 12 in Julia (#45582), it's working everywhere apart from Windows, but I don't have a Windows machine nor have much experience with that system anyway. I've been asking for help, but no one helped me, I can't do miracles 🤷 What would be nice is a situation where we automatically detect system libstdc++ and decide whether to use that one or julia's one. The main problem is how to do this without
I don't think there is a technical solution to this problem, which is the reason why for the time being we need to live with "2". Side note, I don't think this is such a widespread as you believe. For one, I never run into this because my use of Julia doesn't involve loading system libraries. This happens for example when people need some video libraries, or R libraries like in the case reported at the top of this issue, I don't need any of them. |
|
RCall is currently broken on my system also, due to this bug. If you're here from a search engine looking for a workaround, copy the system E.g. (on archlinux with juliaup) |
I have not dealt with that particular scenario (more generally, I have managed to escape limitations of cluster environments using Singularity), but I understand the issue. It makes sense that Julia wants to support these very old environments, however, that should not get in the way of also supporting very new environments.
This is what I have been proposing as my primary proposal. I'm glad you agree that after consideration of the other options, this is the only one really left. You are correct that it is not always possible, but it should usually be possible at the point that Julia is installed. What I propose is that a hook could be included in the Julia distribution that will just emit a warning in case it can't write to Julia, and it's up to whichever installation process is used (e.g. juliaup) to decide whether it is appropriate to invoke it and when. Hopefully such a hook being bundled with Julia would give the juliaup folks some confidence that it's all kosher. At this point, I'm satisfied that this is the best solution available at the moment, and will come back with a proof-of-concept/stawman. Hopefully this will give us a better starting point that won't end up with us talking past each other. |
|
The best long-term solution here will likely be JuliaCI/julia-buildkite#205, which:
|
Okay, great! This is basically the "do it at startup approach" implied by giordano's "killing startup time" from above. In fact since it just requires 1 symbol value comparison and an extra (P.S. Ideally, this issue would still be reopened.) |
If the system libstdc++ is detected to be newer, load it. Otherwise, load the one that we ship. This improves compatibility with external shared libraries that the user might have on their system. Fixes JuliaLang#34276 Co-authored-by: Jameson Nash <[email protected]> Co-authored-by: Elliot Saba <[email protected]>
If the system libstdc++ is detected to be newer, load it. Otherwise, load the one that we ship. This improves compatibility with external shared libraries that the user might have on their system. Fixes JuliaLang#34276 Co-authored-by: Jameson Nash <[email protected]> Co-authored-by: Elliot Saba <[email protected]>
If the system libstdc++ is detected to be newer, load it. Otherwise, load the one that we ship. This improves compatibility with external shared libraries that the user might have on their system. Fixes JuliaLang#34276 Co-authored-by: Jameson Nash <[email protected]> Co-authored-by: Elliot Saba <[email protected]>
If the system libstdc++ is detected to be newer, load it. Otherwise, load the one that we ship. This improves compatibility with external shared libraries that the user might have on their system. Fixes JuliaLang#34276 Co-authored-by: Jameson Nash <[email protected]> Co-authored-by: Elliot Saba <[email protected]>
* Probe if system libstdc++ is newer than ours If the system libstdc++ is detected to be newer, load it. Otherwise, load the one that we ship. This improves compatibility with external shared libraries that the user might have on their system. Fixes #34276 Co-authored-by: Jameson Nash <[email protected]> Co-authored-by: Elliot Saba <[email protected]> * Addressed review comments. * Change error handling in wrapper functions Co-authored-by: Jameson Nash <[email protected]> * Call write_wrapper three times instead of snprintf Co-authored-by: Jameson Nash <[email protected]> * Apply suggestions from code review Co-authored-by: Jameson Nash <[email protected]> * Update cli/loader_lib.c Co-authored-by: Jameson Nash <[email protected]> * Reordered reading and waiting to avoid a deadlock. * Fixed obvious issues. * Only load libstdc++ preemptively on linux. * Update cli/loader_lib.c Co-authored-by: Jameson Nash <[email protected]> * Update cli/loader_lib.c Co-authored-by: Jameson Nash <[email protected]> * Specified path to bundled libstdc++ on the command line. * Removed whitespace. * Update cli/Makefile Co-authored-by: Jameson Nash <[email protected]> * Handled make install stringreplace. * Correctly quoted stringreplace. * Added -Wl,--enable-new-dtags to prevent DT_RPATH for transitive dependencies * Updated news entry. * Added comment about environment variable. * patched rpath for libgfortran and libLLVM. * Added explaination to Make.inc * Removed trailing space * Removed patchelf for libgfortran, now that BB has been fixed. * Fixed typos and comments Co-authored-by: Max Horn <[email protected]> Co-authored-by: Mosè Giordano <[email protected]> Co-authored-by: Jameson Nash <[email protected]> Co-authored-by: Elliot Saba <[email protected]> Co-authored-by: Max Horn <[email protected]>
* Probe if system libstdc++ is newer than ours If the system libstdc++ is detected to be newer, load it. Otherwise, load the one that we ship. This improves compatibility with external shared libraries that the user might have on their system. Fixes #34276 Co-authored-by: Jameson Nash <[email protected]> Co-authored-by: Elliot Saba <[email protected]> * Addressed review comments. * Change error handling in wrapper functions Co-authored-by: Jameson Nash <[email protected]> * Call write_wrapper three times instead of snprintf Co-authored-by: Jameson Nash <[email protected]> * Apply suggestions from code review Co-authored-by: Jameson Nash <[email protected]> * Update cli/loader_lib.c Co-authored-by: Jameson Nash <[email protected]> * Reordered reading and waiting to avoid a deadlock. * Fixed obvious issues. * Only load libstdc++ preemptively on linux. * Update cli/loader_lib.c Co-authored-by: Jameson Nash <[email protected]> * Update cli/loader_lib.c Co-authored-by: Jameson Nash <[email protected]> * Specified path to bundled libstdc++ on the command line. * Removed whitespace. * Update cli/Makefile Co-authored-by: Jameson Nash <[email protected]> * Handled make install stringreplace. * Correctly quoted stringreplace. * Added -Wl,--enable-new-dtags to prevent DT_RPATH for transitive dependencies * Updated news entry. * Added comment about environment variable. * patched rpath for libgfortran and libLLVM. * Added explaination to Make.inc * Removed trailing space * Removed patchelf for libgfortran, now that BB has been fixed. * Fixed typos and comments Co-authored-by: Max Horn <[email protected]> Co-authored-by: Mosè Giordano <[email protected]> Co-authored-by: Jameson Nash <[email protected]> Co-authored-by: Elliot Saba <[email protected]> Co-authored-by: Max Horn <[email protected]> (cherry picked from commit eb708d6)
As discussed on slack with and discourse ( https://discourse.julialang.org/t/glibcxx-3-4-26-not-found-in-rcall/29113/2 ) the problem of an outdated, bundled libstdc++ (e.g. Julia 1.3.1) compared to the libstdc++ provided by the OS (e.g. Ubuntu 19.10) is that it breaks RCall.jl when
Rcppis in play due to missing GLIBCXX symbols.I quote @staticfloat
An intermediate workaround is described here: https://discourse.julialang.org/t/glibcxx-3-4-26-not-found-in-rcall/29113/11?u=laborg
The text was updated successfully, but these errors were encountered: