Cannot compile on MacOS: library not found for -lEnzyme-19

[lynaghk]

First, let me say that I'm extremely excited about having automatic differentiation like this in Rust --- thank you for your hard work on everything!

I'm trying to compile from a clean checkout of e47e0ba on my M1 Macbook Air (MacOS 12.7.2) and the build fails with library not found for -lEnzyme-19.

I did everything as per the book's instructions:

$ git clone --depth=1 [email protected]:EnzymeAD/rust.git rust-enzyme
Cloning into 'rust'...
remote: Enumerating objects: 51579, done.
remote: Counting objects: 100% (51579/51579), done.
remote: Compressing objects: 100% (43737/43737), done.
remote: Total 51579 (delta 6909), reused 28104 (delta 6251), pack-reused 0 (from 0)
Receiving objects: 100% (51579/51579), 32.69 MiB | 13.33 MiB/s, done.
Resolving deltas: 100% (6909/6909), done.
Updating files: 100% (49077/49077), done.
$ cd rust-enzyme   
$ ./configure --enable-llvm-link-shared --enable-llvm-plugins --enable-llvm-enzyme --release-channel=nightly --enable-llvm-assertions --enable-clang --enable-lld --enable-option-checking --enable-ninja --disable-docs

I've attached the full log as a txt file.

I see some references to AppleClang 14.0.0.14000029, which is interesting since on my PATH I have a much newer clang:

$ which clang
/opt/local/libexec/llvm-17/bin/clang
$ clang --version
clang version 17.0.6
Target: arm64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /opt/local/libexec/llvm-17/bin

I'm not super familiar with the dark magic of all these linker flags and environment variable stuff, so I figured I'd run it by y'all and document publicly in case anyone else runs into the same issue. Please let me know if you need any more info from my side. I also saw on some other issue threads that you may not have access to Mac hardware --- happy to hop on a call to pair if you prefer that to async discussion.

[lynaghk]

I revisited this with the latest commit, 2a23a80, but failed with the same error: library not found for -lEnzyme-19

I poked around and found a folder of built libraries, and running

export LIBRARY_PATH=/Users/dev/software/rust-enzyme/build/aarch64-apple-darwin/enzyme/lib

allowed the

./x.py build --stage 1 library

command to succeed. I then ran (as per book instructions)

rustup toolchain link enzyme build/host/stage1
rustup toolchain install nightly

but failed to successfully compile the rustbook samples (as of commit 0ffed11799d9ee4be1db84e01f88bdba2405be10):

18:40:53 $ cargo +enzyme test reverse
zsh: correct 'test' to 'tests' [nyae]? n
   Compiling samples v0.1.0 (/Users/dev/software/enzyme-rustbook/samples)
source_id: DefId(0:135 ~ mod[da47]::reverse::empty_return::f)
source_id: DefId(0:256 ~ mod[da47]::traits::volumetric::{impl#1}::psi)
source_id: DefId(0:146 ~ mod[da47]::reverse::active_return::f)
source_id: DefId(0:82 ~ mod[da47]::forward::dual_return::f)
source_id: DefId(0:71 ~ mod[da47]::forward::empty_return::f)
source_id: DefId(0:223 ~ mod[da47]::neohookean::psi)
source_id: DefId(0:107 ~ mod[da47]::reverse::square::square)
source_id: DefId(0:127 ~ mod[da47]::reverse::self_duplicated::{impl#0}::f)
source_id: DefId(0:231 ~ mod[da47]::neohookean::logdet)
source_id: DefId(0:96 ~ mod[da47]::forward::dual_only_return::f)
source_id: DefId(0:116 ~ mod[da47]::reverse::active_only::f)
source_id: DefId(0:160 ~ mod[da47]::reverse::forward_and_reverse::f)
source_id: DefId(0:171 ~ mod[da47]::second::forward_of_reverse::f)
source_id: DefId(0:116 ~ mod[da47]::reverse::active_only::f)
source_id: DefId(0:188 ~ mod[da47]::higher::df2)
source_id: DefId(0:176 ~ mod[da47]::second::forward_of_reverse::g)
source_id: DefId(0:160 ~ mod[da47]::reverse::forward_and_reverse::f)
source_id: DefId(0:96 ~ mod[da47]::forward::dual_only_return::f)
source_id: DefId(0:202 ~ mod[da47]::higher::f)
num_fnc_args: 2
input_activity.len(): 2
num_fnc_args: 1
input_activity.len(): 1
num_fnc_args: 2
input_activity.len(): 2
num_fnc_args: 2
input_activity.len(): 2
num_fnc_args: 2
input_activity.len(): 2
num_fnc_args: 1
input_activity.len(): 1
num_fnc_args: 1
input_activity.len(): 1
num_fnc_args: 2
input_activity.len(): 2
num_fnc_args: 1
input_activity.len(): 1
num_fnc_args: 1
input_activity.len(): 1
num_fnc_args: 1
input_activity.len(): 1
Attribute after last parameter!
ptr @preprocess_diffe_ZN3mod6higher1f17hb7deab8969c456c2E
; Function Attrs: mustprogress noinline sanitize_hwaddress willreturn memory(readwrite) uwtable
define internal void @preprocess_diffe_ZN3mod6higher1f17hb7deab8969c456c2E(ptr align 4 "enzyme_type"="{[-1]:Pointer, [-1,0]:Float@float, [-1,4]:Float@float}" %0, ptr align 4 "enzyme_type"="{[-1]:Pointer, [-1,0]:Float@float, [-1,4]:Float@float}" %1, ptr align 4 "enzyme_type"="{[-1]:Pointer, [-1,0]:Float@float}" %2, ptr align 4 "enzyme_type"="{[-1]:Pointer, [-1,0]:Float@float}" %3) unnamed_addr #110 !dbg !65781 {
    #dbg_value(ptr %0, !65783, !DIExpression(), !65785)
    #dbg_value(ptr %2, !65784, !DIExpression(), !65785)
  %5 = load float, ptr %0, align 4, !dbg !65786, !alias.scope !65787, !noalias !65790
  %6 = fmul float %5, %5, !dbg !65786
  %7 = getelementptr inbounds [2 x float], ptr %1, i64 0, i64 1, !dbg !65792
  %8 = getelementptr inbounds [2 x float], ptr %0, i64 0, i64 1, !dbg !65792
  %9 = load float, ptr %8, align 4, !dbg !65792, !alias.scope !65787, !noalias !65790
  %10 = fmul float %9, %5, !dbg !65792
  %11 = fadd float %6, %10, !dbg !65793
  store float %11, ptr %2, align 4, !dbg !65793, !alias.scope !65794, !noalias !65797
  %12 = load float, ptr %3, align 4, !dbg !65793, !alias.scope !65797, !noalias !65794
  store float 0.000000e+00, ptr %3, align 4, !dbg !65793, !alias.scope !65797, !noalias !65794
  %13 = fmul fast float %12, %5, !dbg !65792
  %14 = fmul fast float %12, %9, !dbg !65792
  %15 = load float, ptr %1, align 4, !dbg !65799, !alias.scope !65790, !noalias !65787
  %16 = fadd fast float %15, %14, !dbg !65799
  store float %16, ptr %1, align 4, !dbg !65799, !alias.scope !65790, !noalias !65787
  %17 = load float, ptr %7, align 4, !dbg !65792, !alias.scope !65790, !noalias !65787
  %18 = fadd fast float %17, %13, !dbg !65792
  store float %18, ptr %7, align 4, !dbg !65792, !alias.scope !65790, !noalias !65787
  %19 = fadd fast float %16, %13, !dbg !65800
  store float %19, ptr %1, align 4, !dbg !65800, !alias.scope !65790, !noalias !65787
  %20 = fadd fast float %19, %13, !dbg !65786
  store float %20, ptr %1, align 4, !dbg !65786, !alias.scope !65790, !noalias !65787
  ret void
}

rustc-LLVM ERROR: function failed verification (1)
error: could not compile `samples` (test "mod")

Caused by:
  process didn't exit successfully: `sccache /Users/dev/.rustup/toolchains/enzyme/bin/rustc --crate-name mod --edition=2021 samples/tests/mod.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=114 --emit=dep-info,link -C lto=fat -C debuginfo=2 -C split-debuginfo=unpacked --warn=unexpected_cfgs --test --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' --check-cfg 'cfg(broken)' -C metadata=9439086bb0fa6d2c -C extra-filename=-9439086bb0fa6d2c --out-dir /Users/dev/software/enzyme-rustbook/target/debug/deps -C incremental=/Users/dev/software/enzyme-rustbook/target/debug/incremental -L dependency=/Users/dev/software/enzyme-rustbook/target/debug/deps --extern samples=/Users/dev/software/enzyme-rustbook/target/debug/deps/libsamples-879e9c4c4c1fad88.rlib` (exit status: 101)
warning: build failed, waiting for other jobs to finish...

[ZuseZ4]

Hi, and thanks for investigating! The code for building and linking Enzyme is currently in the process of upstreaming and should land in upstream Rust within the next few hours, so I don't want to touch it right now.
rust-lang#129176

Once that happens I'll update the master branch of this fork to be built on the newest rustc. During upstreaming I got a few suggestions to improve the code quality, so maybe that will already be enough to fix your linker issue. Otherwise I'll make a follow up PR against rustc to fix this and let you know.

One of the tests failing also ins't surprising, I haven't checked for them in a while since I'm currently busy with upstreaming and since the Rust frontend isn't tested that well yet rustc/llvm updates might break rust-enzyme test. That should quickly get better once I have everything upstream and time to fix some of these bugs.

[DeliciousHair]

This is very, very cool-I'm so glad I decided to revisit this today!

Just wanted to chime in here as well to say that while the build seems to now work, the examples fail. I have created a (horrible) container to test things locally in isolation, the setup can be found here. Attempting to run the examples as instructed in the docs fails:

root@enzyme-dev:/opt/rustbook# cargo +enzyme test reverse
   Compiling serde_derive v1.0.209
   Compiling toml v0.8.19
   Compiling samples v0.1.0 (/opt/rustbook/samples)
   Compiling integration v0.1.0 (/opt/rustbook/integration)
source_id: DefId(0:33 ~ mod[037d]::examples::boxed::duplicated_active::sin)
warning: reading from a `Duplicated` const box is unsafe
 --> integration/tests/examples/boxed.rs:4:5
  |
4 |     fn sin(x: &Box<f32>) -> f32 {
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^

warning: reading from a `Duplicated` const ptr is unsafe
 --> integration/tests/examples/hessian_sin.rs:4:5
  |
4 |     fn sin(x: &Vec<f32>, y: &mut f32) {
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

source_id: DefId(0:52 ~ mod[037d]::examples::hessian_sin::vec::sin)
source_id: DefId(0:55 ~ mod[037d]::examples::hessian_sin::vec::jac2)
source_id: DefId(0:41 ~ mod[037d]::examples::foo::foo::f)
source_id: DefId(0:14 ~ mod[037d]::examples::array::reverse_duplicated_active::array)
source_id: DefId(0:188 ~ mod[d6d9]::higher::df2)
source_id: DefId(0:116 ~ mod[d6d9]::reverse::active_only::f)
source_id: DefId(0:107 ~ mod[d6d9]::reverse::square::square)
source_id: DefId(0:160 ~ mod[d6d9]::reverse::forward_and_reverse::f)
source_id: DefId(0:171 ~ mod[d6d9]::second::forward_of_reverse::f)
source_id: DefId(0:223 ~ mod[d6d9]::neohookean::psi)
source_id: DefId(0:146 ~ mod[d6d9]::reverse::active_return::f)
source_id: DefId(0:176 ~ mod[d6d9]::second::forward_of_reverse::g)
source_id: DefId(0:96 ~ mod[d6d9]::forward::dual_only_return::f)
source_id: DefId(0:135 ~ mod[d6d9]::reverse::empty_return::f)
source_id: DefId(0:96 ~ mod[d6d9]::forward::dual_only_return::f)
source_id: DefId(0:127 ~ mod[d6d9]::reverse::self_duplicated::{impl#0}::f)
source_id: DefId(0:160 ~ mod[d6d9]::reverse::forward_and_reverse::f)
source_id: DefId(0:202 ~ mod[d6d9]::higher::f)
source_id: DefId(0:116 ~ mod[d6d9]::reverse::active_only::f)
source_id: DefId(0:82 ~ mod[d6d9]::forward::dual_return::f)
source_id: DefId(0:256 ~ mod[d6d9]::traits::volumetric::{impl#1}::psi)
source_id: DefId(0:71 ~ mod[d6d9]::forward::empty_return::f)
source_id: DefId(0:231 ~ mod[d6d9]::neohookean::logdet)
num_fnc_args: 2
input_activity.len(): 2
num_fnc_args: 1
input_activity.len(): 1
Attribute after last parameter!
ptr @preprocess_diffe_ZN3mod8examples11hessian_sin3vec3sin17h6e915a53835e0aa3E
; Function Attrs: mustprogress noinline nonlazybind sanitize_hwaddress willreturn memory(readwrite) uwtable
define internal void @preprocess_diffe_ZN3mod8examples11hessian_sin3vec3sin17h6e915a53835e0aa3E(ptr align 8 "enzyme_type"="{[-1]:Pointer, [-1,0]:Integer, [-1,1]:Integer, [-1,2]:Integer, [-1,3]:Integer, [-1,4]:Integer, [-1,5]:Integer, [-1,6]:Integer, [-1,7]:Integer, [-1,8]:Pointer, [-1,8,-1]:Float@float, [-1,16]:Integer, [-1,17]:Integer, [-1,18]:Integer, [-1,19]:Integer, [-1,20]:Integer, [-1,21]:Integer, [-1,22]:Integer, [-1,23]:Integer}" %0, ptr align 8 "enzyme_type"="{[-1]:Pointer, [-1,0]:Integer, [-1,1]:Integer, [-1,2]:Integer, [-1,3]:Integer, [-1,4]:Integer, [-1,5]:Integer, [-1,6]:Integer, [-1,7]:Integer, [-1,8]:Pointer, [-1,8,-1]:Float@float, [-1,16]:Integer, [-1,17]:Integer, [-1,18]:Integer, [-1,19]:Integer, [-1,20]:Integer, [-1,21]:Integer, [-1,22]:Integer, [-1,23]:Integer}" %1, ptr align 4 "enzyme_type"="{[-1]:Pointer, [-1,-1]:Float@float}" %2, ptr align 4 "enzyme_type"="{[-1]:Pointer, [-1,-1]:Float@float}" %3) unnamed_addr #120 !dbg !67602 {
    #dbg_value(ptr %0, !67604, !DIExpression(), !67606)
    #dbg_value(ptr %2, !67605, !DIExpression(), !67606)
  %5 = call { { ptr, ptr, i64 }, { ptr, ptr }, { ptr, ptr } } @"augmented__ZN94_$LT$$RF$alloc..vec..Vec$LT$T$C$A$GT$$u20$as$u20$core..iter..traits..collect..IntoIterator$GT$9into_iter17hee9dc0c1f21bf252E"(ptr align 8 %0, ptr align 8 %1) #127, !dbg !67607
  %6 = extractvalue { { ptr, ptr, i64 }, { ptr, ptr }, { ptr, ptr } } %5, 0, !dbg !67607
  %7 = extractvalue { { ptr, ptr, i64 }, { ptr, ptr }, { ptr, ptr } } %5, 1, !dbg !67607
  %8 = extractvalue { { ptr, ptr, i64 }, { ptr, ptr }, { ptr, ptr } } %5, 2, !dbg !67607
  %9 = extractvalue { ptr, ptr } %7, 0, !dbg !67607
  %10 = extractvalue { ptr, ptr } %8, 0, !dbg !67607
  %11 = extractvalue { ptr, ptr } %7, 1, !dbg !67607
  %12 = extractvalue { ptr, ptr } %8, 1, !dbg !67607
  %13 = call { { ptr, ptr }, { ptr, ptr } } @augmented__ZN4core4iter6traits8iterator8Iterator3map17h2597bb794fa22e09E(ptr %9, ptr %10, ptr %11, ptr %12) #127, !dbg !67607
  %14 = extractvalue { { ptr, ptr }, { ptr, ptr } } %13, 0, !dbg !67607
  %15 = extractvalue { { ptr, ptr }, { ptr, ptr } } %13, 1, !dbg !67607
  %16 = extractvalue { ptr, ptr } %14, 0, !dbg !67607
  %17 = extractvalue { ptr, ptr } %15, 0, !dbg !67607
  %18 = extractvalue { ptr, ptr } %14, 1, !dbg !67607
  %19 = extractvalue { ptr, ptr } %15, 1, !dbg !67607
  %20 = call { { ptr, ptr, ptr, ptr }, float } @augmented__ZN4core4iter6traits8iterator8Iterator3sum17h1083f5b751d6484bE(ptr %16, ptr %17, ptr %18, ptr %19) #127, !dbg !67607
  %21 = extractvalue { { ptr, ptr, ptr, ptr }, float } %20, 0, !dbg !67607
  %22 = extractvalue { { ptr, ptr, ptr, ptr }, float } %20, 1, !dbg !67607
  store float %22, ptr %2, align 4, !dbg !67608, !alias.scope !67609, !noalias !67612
  %23 = load float, ptr %3, align 4, !dbg !67608, !alias.scope !67612, !noalias !67609
  store float 0.000000e+00, ptr %3, align 4, !dbg !67608, !alias.scope !67612, !noalias !67609
  call void @diffe_ZN4core4iter6traits8iterator8Iterator3sum17h1083f5b751d6484bE(ptr %16, ptr %17, ptr %18, ptr %19, float %23, { ptr, ptr, ptr, ptr } %21) #127, !dbg !67607
  call void @diffe_ZN4core4iter6traits8iterator8Iterator3map17h2597bb794fa22e09E(ptr %9, ptr %10, ptr %11, ptr %12) #127, !dbg !67607
  call void @"diffe_ZN94_$LT$$RF$alloc..vec..Vec$LT$T$C$A$GT$$u20$as$u20$core..iter..traits..collect..IntoIterator$GT$9into_iter17hee9dc0c1f21bf252E"(ptr align 8 %0, ptr align 8 %1, { ptr, ptr, i64 } %6) #127, !dbg !67607
  ret void
}

rustc-LLVM ERROR: function failed verification (1)
warning: `integration` (test "mod") generated 2 warnings
error: could not compile `integration` (test "mod"); 2 warnings emitted

Caused by:
  process didn't exit successfully: `/root/.rustup/toolchains/enzyme/bin/rustc --crate-name mod --edition=2021 integration/tests/mod.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=190 --emit=dep-info,link -C lto=fat -C debuginfo=2 --warn=unexpected_cfgs --test --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' --check-cfg 'cfg(broken)' -C metadata=ff80f42173bded00 -C extra-filename=-ff80f42173bded00 --out-dir /opt/rustbook/target/debug/deps -C incremental=/opt/rustbook/target/debug/incremental -L dependency=/opt/rustbook/target/debug/deps --extern samples=/opt/rustbook/target/debug/deps/libsamples-7b0e9c21ffcd6df6.rlib` (exit status: 101)
warning: build failed, waiting for other jobs to finish...
num_fnc_args: 1
input_activity.len(): 1
num_fnc_args: 2
input_activity.len(): 2
num_fnc_args: 2
input_activity.len(): 2
Attribute after last parameter!
ptr @preprocess_diffe_ZN3mod6higher1f17h0b43a8a690cc6365E
; Function Attrs: mustprogress noinline nonlazybind sanitize_hwaddress willreturn memory(readwrite) uwtable
define internal void @preprocess_diffe_ZN3mod6higher1f17h0b43a8a690cc6365E(ptr align 4 "enzyme_type"="{[-1]:Pointer, [-1,0]:Float@float, [-1,4]:Float@float}" %0, ptr align 4 "enzyme_type"="{[-1]:Pointer, [-1,0]:Float@float, [-1,4]:Float@float}" %1, ptr align 4 "enzyme_type"="{[-1]:Pointer, [-1,0]:Float@float}" %2, ptr align 4 "enzyme_type"="{[-1]:Pointer, [-1,0]:Float@float}" %3) unnamed_addr #120 !dbg !66404 {
    #dbg_value(ptr %0, !66406, !DIExpression(), !66408)
    #dbg_value(ptr %2, !66407, !DIExpression(), !66408)
  %5 = load float, ptr %0, align 4, !dbg !66409, !alias.scope !66410, !noalias !66413
  %6 = fmul float %5, %5, !dbg !66409
  %7 = getelementptr inbounds [2 x float], ptr %1, i64 0, i64 1, !dbg !66415
  %8 = getelementptr inbounds [2 x float], ptr %0, i64 0, i64 1, !dbg !66415
  %9 = load float, ptr %8, align 4, !dbg !66415, !alias.scope !66410, !noalias !66413
  %10 = fmul float %9, %5, !dbg !66415
  %11 = fadd float %6, %10, !dbg !66416
  store float %11, ptr %2, align 4, !dbg !66416, !alias.scope !66417, !noalias !66420
  %12 = load float, ptr %3, align 4, !dbg !66416, !alias.scope !66420, !noalias !66417
  store float 0.000000e+00, ptr %3, align 4, !dbg !66416, !alias.scope !66420, !noalias !66417
  %13 = fmul fast float %12, %5, !dbg !66415
  %14 = fmul fast float %12, %9, !dbg !66415
  %15 = load float, ptr %1, align 4, !dbg !66422, !alias.scope !66413, !noalias !66410
  %16 = fadd fast float %15, %14, !dbg !66422
  store float %16, ptr %1, align 4, !dbg !66422, !alias.scope !66413, !noalias !66410
  %17 = load float, ptr %7, align 4, !dbg !66415, !alias.scope !66413, !noalias !66410
  %18 = fadd fast float %17, %13, !dbg !66415
  store float %18, ptr %7, align 4, !dbg !66415, !alias.scope !66413, !noalias !66410
  %19 = fadd fast float %16, %13, !dbg !66423
  store float %19, ptr %1, align 4, !dbg !66423, !alias.scope !66413, !noalias !66410
  %20 = fadd fast float %19, %13, !dbg !66409
  store float %20, ptr %1, align 4, !dbg !66409, !alias.scope !66413, !noalias !66410
  ret void
}

rustc-LLVM ERROR: function failed verification (1)
error: could not compile `samples` (test "mod")

Caused by:
  process didn't exit successfully: `/root/.rustup/toolchains/enzyme/bin/rustc --crate-name mod --edition=2021 samples/tests/mod.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=190 --emit=dep-info,link -C lto=fat -C debuginfo=2 --warn=unexpected_cfgs --test --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' --check-cfg 'cfg(broken)' -C metadata=0456e5dc8b148851 -C extra-filename=-0456e5dc8b148851 --out-dir /opt/rustbook/target/debug/deps -C incremental=/opt/rustbook/target/debug/incremental -L dependency=/opt/rustbook/target/debug/deps --extern samples=/opt/rustbook/target/debug/deps/libsamples-7b0e9c21ffcd6df6.rlib` (exit status: 101)

[ZuseZ4]

I just rebased the master branch on the latest upstream rustc (which now includes the first autodiff code).
That should bring a few code quality improvements.
I've also marked one or two tests in the rustbook as broken, so now cargo +enzyme test should not result in ice's anymore.

Please feel free to test again. If it's still failing without your manual export, I'll look into fixing it in rustc @lynaghk

[lynaghk]

Thanks @ZuseZ4! I just did a clean checkout and build (enzyme_rust_log.txt), but ran into the same linker error. The same

export LIBRARY_PATH=/Users/dev/software/rust-enzyme/build/aarch64-apple-darwin/enzyme/lib

allowed the build to succeed. Unfortunately, there are still errors when trying to build the tests for the book (book_log.txt).

[ZuseZ4]

@lynaghk Someone from the rust compiler team wrote a fix for a similar linking issue, which I've already merged into master (36f5855).
Can you please try again and let me know if it also solves the MacOS linking issue?

[lynaghk]

I just tried the latest e3c1881 but unfortunately have the same failure.

  = note: ld: library not found for -lEnzyme-19

Full log.txt.