CUDA_ERROR_UNSUPPORTED_PTX_VERSION when loading is_u32_f32

[super-fun-surf]

Running executable with candle on Ubuntu 22 we get an error when trying to generate an image

DriverError(CUDA_ERROR_UNSUPPORTED_PTX_VERSION, \"the provided PTX was compiled with an unsupported toolchain.\") when loading is_u32_f32

Our compiling machine is Debian 12
Driver Version: 550.54.14 CUDA Version: 12.4 |
running on a NVIDIA RTX 4000 Ada

The machine that is failing to run the executable is
Ubuntu 22
Driver Version: 550.107.02 CUDA Version: 12.4
NVIDIA A100 80GB PCIe

[LaurentMazare]

No real clue what is going on here. Could you try compiling on the box that runs the executable to see if it fixes the issue?
You could try adjusting the compute cap used to compile the ptx via CUDA_COMPUTE_CAP=... though I'm unsure whether this would explain such an error message.

[super-fun-surf]

Ok so compiling on the A100 works great. It must be a cuda mismatch problem.
I did try using the CUDA_COMPUTE_CAP=80 when compiling on my desktop ADA4000 and then running that on the A100 but couldnt get it to wokr first try. need to explore that more. I will report more. thank you.