Trying to build the image with a machine with 16GB of RAM is impossible.

[tamastarjanyi]

Hi,
I have an issue on H100. For some reason I am getting:

2024-06-13T22:11:30.932570Z  WARN text_embeddings_backend_candle: backends/candle/src/[lib.rs:75](http://lib.rs:75/): Could not find a compatible CUDA device on host: CUDA is not available
2024-06-13T22:11:30.932593Z  WARN text_embeddings_backend_candle: backends/candle/src/[lib.rs:76](http://lib.rs:76/): Using CPU instead

I wanted to add some debugging to see the issue around:

pub fn compatible_compute_cap() -> Result<bool, anyhow::Error> {
    let compile_compute_cap = get_compile_compute_cap()?;
    let runtime_compute_cap = get_runtime_compute_cap()?;
    println!("Compile compute {}", compile_compute_cap);
    println!("Runtime compute {}", runtime_compute_cap);
    Ok(compute_cap_matching(
        runtime_compute_cap,
        compile_compute_cap,
    ))
}

And then wanted to build the image with:

runtime_compute_cap=90
docker build . -f Dockerfile-cuda --build-arg CUDA_COMPUTE_CAP=$runtime_compute_cap

But it ends up creating many-many cicc process in parallel each taking up 2-4 GB of memory causing OOM.

1.) Is there a way to limit parallel cicc processes? This would be great to be able to build it on smaller machines.

2.) Any idea about the CUDA issue?
I have tried both ghcr.io/huggingface/text-embeddings-inference:1.2 and ghcr.io/huggingface/text-embeddings-inference:hopper-1.2 images but both shows the same error despite nvidia-smi --query-gpu=compute_cap --format=csv | sed -n '2p' | sed 's/\.//g' gives 90

[OlivierDehaene]

How do you run the docker container? Do you use the nvidia runtime?
Regarding the docker build, you can limit parallelism for cuda kernels compilation with --build-arg RAYON_NUM_THREADS=4

[tamastarjanyi]

Hi,

Thanks for the answer!
I was using the guide from here.

runtime_compute_cap=90
docker build . -f Dockerfile-cuda --build-arg CUDA_COMPUTE_CAP=$runtime_compute_cap

Now I made a minor change in Dockerfile-cuda and added:

ARG RAYON_NUM_THREADS
ARG CARGO_BUILD_JOBS
ARG CARGO_BUILD_INCREMENTAL

And now using

docker build . -f Dockerfile-cuda --build-arg CUDA_COMPUTE_CAP=$runtime_compute_cap --build-arg RAYON_NUM_THREADS=1 --build-arg CARGO_BUILD_JOBS=1 --build-arg CARGO_BUILD_INCREMENTAL=true

Without CARGO_BUILD_JOBS I had 8 parallel jobs (8 cores in my machine) each running 4 cicc causing OOM again. But now with 1 job and 1 cicc per each compile is working fine just very slow. But thanks for the tip! This is tunable for me now.

About the other issue, I have found the problem. Unfortunately the current error handling suppressing the real error.

compute_cap.rs

pub fn get_runtime_compute_cap() -> Result<usize, anyhow::Error> {
    driver::result::init().context("CUDA is not available")?; // **This was success**
    let device = CudaDevice::new(0).context("CUDA is not available")?; // TODO: This failed with CUDA_ERROR_LAUNCH_FAILED but sent back "CUDA is not available"
    let major = device
        .attribute(CU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MAJOR)
        .context("Could not retrieve device compute capability major")?;
    let minor = device
        .attribute(CU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MINOR)
        .context("Could not retrieve device compute capability minor")?;
    Ok((major * 10 + minor) as usize)

and
backends/candle/src/lib.rs

 let device = if candle::utils::cuda_is_available() {
            #[cfg(feature = "cuda")]
            match compatible_compute_cap() {
                Ok(true) => Device::new_cuda(0),
                Ok(false) => {
                    return Err(BackendError::Start(format!(
                        "Runtime compute cap {} is not compatible with compile time compute cap {}",
                        get_runtime_compute_cap().unwrap(),
                        get_compile_compute_cap().unwrap()
                    )))
                }
                Err(err) => {
                    tracing::warn!("Could not find a compatible CUDA device on host: {err}"); // TODO: Misleading error about runtime vs compite time compatibility
                    tracing::warn!("Using CPU instead");
                    Ok(Device::Cpu)
                }
            }

This line let device = CudaDevice::new(0).context("CUDA is not available")?; run into a CUDA_ERROR_LAUNCH_FAILED but reported
but at application level it reports Could not find a compatible CUDA device on host: CUDA is not available

If I may suggest it would be great to get the real cuda errors back in any case.
The real error in my case was as mentioned above a CUDA_ERROR_LAUNCH_FAILED suppressed by this code with a Could not find a compatible CUDA device on host: CUDA is not available.
After getting the real error it was easier to fix this and now I have embedding server running and using GPU.

[OlivierDehaene]

I agree.
I will change the error message to include the err sent by the driver.

[OlivierDehaene]

Fixed by #300

[tamastarjanyi]

Thank you so much. <3

[nfairbank]

Is anyone else still finding this extremely difficult to build? I've tried all of the above and still get hung on compiling parts of:

cargo build --verbose --release --bin text-embeddings-router -F candle-cuda -F static-linking -F http --no-default-features && sccache -s;
It's irregular where it stops, but it seems like it will reach the last dozen or so things that it's supposed to compile there, then quit. I've messed around with the build args etc with no luck.

If anyone has a good build solution please let me know!