Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

linker failure on macOS when building Miri: "linker stderr: ld: ignoring duplicate libraries" #136086

Closed
RalfJung opened this issue Jan 26, 2025 · 11 comments · Fixed by #136098
Closed
Labels
A-linkage Area: linking into static, shared libraries and binaries A-linkers Area: linkers... you gotta love linkers A-miri Area: The miri tool C-bug Category: This is a bug. L-linker_messages Lint: linker_messages T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@RalfJung
Copy link
Member

RalfJung commented Jan 26, 2025

Miri is failing to build on macOS:

 error: linker stderr: ld: ignoring duplicate libraries: '-liconv', '-lm'
         ld: search path '/Users/runner/work/miri/miri/target/release/build/libffi-sys-2c3f829423e2e4d5/out/libffi-root/lib64' not found
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[4](jemalloc.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[5](arena.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[6](background_thread.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[7](base.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[8](bin.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[9](bin_info.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[10](bitmap.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[11](buf_writer.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[12](cache_bin.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[13](ckh.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[14](counter.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[15](ctl.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[16](decay.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[17](div.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[18](ecache.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[19](edata.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[20](edata_cache.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[21](ehooks.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[22](emap.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[23](eset.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[24](exp_grow.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[25](extent.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[26](extent_dss.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[27](extent_mmap.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[28](fxp.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[29](san.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[30](san_bump.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[31](hook.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[32](hpa.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[33](hpa_hooks.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[34](hpdata.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[35](inspect.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[36](large.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[37](log.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[38](malloc_io.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[39](mutex.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[40](nstime.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[41](pa.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[42](pa_extra.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[43](pai.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[44](pac.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[45](pages.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[47](prof.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[48](prof_data.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[50](prof_recent.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[53](psset.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[54](rtree.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[55](safety_check.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[56](sc.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[57](sec.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[58](stats.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[59](sz.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[60](tcache.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[61](test_hooks.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[62](thread_event.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[63](ticker.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[64](tsd.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[65](witness.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/libtikv_jemalloc_sys-80d1ddce8807bb5c.rlib[66](zone.pic.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/liblibffi_sys-713e5fa4137b14c5.rlib[4](prep_cif.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/liblibffi_sys-713e5fa4137b14c5.rlib[5](types.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/liblibffi_sys-713e5fa4137b14c5.rlib[8](closures.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/liblibffi_sys-713e5fa4137b14c5.rlib[10](ffi.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
         ld: object file (/Users/runner/work/miri/miri/target/release/deps/liblibffi_sys-713e5fa4137b14c5.rlib[11](sysv.o)) was built for newer 'macOS' version (14.5) than being linked (11.0)
    |
    = note: `-D linker-messages` implied by `-D warnings`
    = help: to override `-D warnings` add `#[allow(linker_messages)]`
  error: could not compile `miri` (bin "miri") due to 1 previous error
  error: failed to compile `miri v0.1.0 (/Users/runner/work/miri/miri)`, intermediate artifacts can be found at `/Users/runner/work/miri/miri/target`.
  To reuse those artifacts with a future compilation, set the environment variable `CARGO_TARGET_DIR` to that path.
  Error: command exited with non-zero code `cargo +miri install --locked -Zroot-dir=/Users/runner/work/miri/miri --path /Users/runner/work/miri/miri --force --root /Users/runner/.rustup/toolchains/miri`: 101

Regression range: 814ebca...2f0ad2a

@rustbot rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Jan 26, 2025
@RalfJung
Copy link
Member Author

RalfJung commented Jan 26, 2025

Likely caused by #119286

Cc @jyn514 @bjorn3

These errors are entirely un-actionable from my perspective, so while I can add -A linker-messages that feels like just papering over possible issues without understanding anything.

@RalfJung RalfJung changed the title linker failure on macOS: "linker stderr: ld: ignoring duplicate libraries" linker failure on macOS when building Miri: "linker stderr: ld: ignoring duplicate libraries" Jan 26, 2025
@jieyouxu jieyouxu added A-linkage Area: linking into static, shared libraries and binaries T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. C-bug Category: This is a bug. A-miri Area: The miri tool A-linkers Area: linkers... you gotta love linkers labels Jan 26, 2025
@jyn514
Copy link
Member

jyn514 commented Jan 26, 2025

i am not sure what to tell you. those warnings were already there before, they were just unconditionally silenced instead of being silenced by a lint.

i saw these “duplicate library” warnings in CI when landing the PR and i did not investigate them because there were about a dozen other types of warnings too, and each takes me a couple weeks to track down: #135828

i still think giving you the option between using -A linker-messages and investigating yourself is better than unconditionally silencing the messages.

@rustbot label L-linker_messages

@rustbot rustbot added the L-linker_messages Lint: linker_messages label Jan 26, 2025
@onur-ozkan onur-ozkan removed the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Jan 26, 2025
@RalfJung
Copy link
Member Author

RalfJung commented Jan 26, 2025

I'm not convinced failing my build is the right call here; I asked for Rust warnings to be made errors (which are generally well-documented and understandable), but I didn't ask for inscrutable unactionable linker warnings that apparently do not reflect an actual problem to become hard errors.

@jyn514
Copy link
Member

jyn514 commented Jan 26, 2025

The other option, which I implemented initially, is to make this a hard warning instead of a lint. But that’s worse in other respects, because now you can’t allow it at all. So people will end up with unsilencable warnings when building locally.

I do not think it is a good idea to go back to silencing this unconditionally. In your case the warnings are not useful, but in other cases like #83436, the warnings are very useful and could have helped identify a very tricky bug earlier.

@jyn514
Copy link
Member

jyn514 commented Jan 26, 2025

it does seem like macOS LD is more chatty than other linkers, and if we can find some way to silence these particular warnings you’re getting for everyone, I would be OK with that. But I do not think we should silence all warnings unconditionally, not even if we scope that to macOS.

@jyn514
Copy link
Member

jyn514 commented Jan 26, 2025

it does seem like macOS LD is more chatty than other linkers

given this, and that @saethlin tells me he has seen this warning on almost every macOS linker error, I plan to do three things:

  • Make this lint allow by default on macOS only when using the system linker. Do this immediately so we don’t end up with a bunch of people ignoring the lint indefinitely. That gives us more time to:
  • extend rustc_codegen_ssa to filter out these chatty warnings that @RalfJung is seeing
  • Turn the lint level back up to warn

@Noratrieb
Copy link
Member

We've discussed this a bit more and came to the conclusion that we should make it fully allow by default for now and at least do a (GNU/Linux only of course) crater run to see some more warning patterns that we may want to filter out.
Then once we add more filters we can make it warn by default (and maybe repeat).

@RalfJung
Copy link
Member Author

RalfJung commented Jan 26, 2025 via email

bors added a commit to rust-lang-ci/rust that referenced this issue Jan 27, 2025
Downgrade `linker-warnings` to allow-by-default

This needs more time to bake before we turn it on. Turning it on early risks people silencing the warning indefinitely, before we have the chance to make it less noisy.

cc rust-lang#136096
fixes rust-lang#136086 (comment)

r? `@saethlin` cc `@Noratrieb` `@bjorn3`

`@rustbot` label A-linkage L-linker_messages
Zalathar added a commit to Zalathar/rust that referenced this issue Jan 27, 2025
Downgrade `linker-warnings` to allow-by-default

This needs more time to bake before we turn it on. Turning it on early risks people silencing the warning indefinitely, before we have the chance to make it less noisy.

cc rust-lang#136096
fixes rust-lang#136086 (comment)

r? `@saethlin` cc `@Noratrieb` `@bjorn3`

`@rustbot` label A-linkage L-linker_messages
@bors bors closed this as completed in 0cffe5c Jan 27, 2025
@RalfJung
Copy link
Member Author

RalfJung commented Jan 28, 2025

Builds with -D warnings (as is common on CI) will still fail when there is any linker output, right? So Miri still needs to explicitly silence the lint I think?

@saethlin
Copy link
Member

Experimentation locally indicates no. I tried the reproducer for #136109 on the nightly that just released and it now builds fine even with -Dwarnings. The lint must be specifically enabled to produce a warning now, with -Wlinker-messages or such.

@RalfJung
Copy link
Member Author

Ah right, it's not a "warning" any more so not affected by that flag.

BoxyUwU pushed a commit to BoxyUwU/rustc-dev-guide that referenced this issue Jan 28, 2025
Downgrade `linker-warnings` to allow-by-default

This needs more time to bake before we turn it on. Turning it on early risks people silencing the warning indefinitely, before we have the chance to make it less noisy.

cc rust-lang/rust#136096
fixes rust-lang/rust#136086 (comment)

r? `@saethlin` cc `@Noratrieb` `@bjorn3`

`@rustbot` label A-linkage L-linker_messages
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-linkage Area: linking into static, shared libraries and binaries A-linkers Area: linkers... you gotta love linkers A-miri Area: The miri tool C-bug Category: This is a bug. L-linker_messages Lint: linker_messages T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants