My general advice / tips for compiling mistral.rs in WSL2

[joshuasundance-swca]

also posted as a gist

some of this advice is also helpful for just getting mistral.rs docker images to work with WSL2 + GPU backend

let me know if you have any questions :)

---

This document is a quick-and-dirty guide to compiling mistral.rs in WSL2, but it probably contains useful information for anyone trying to build CUDA applications in WSL2.

mistral.rs: Blazingly fast LLM inference.

https://github.com/EricLBuehler/mistral.rs

REQUIREMENTS

• CUDA-capable NVIDIA GPU
  • https://developer.nvidia.com/cuda-gpus
• Windows 11 or Windows 10 >= 21H2. Windows 10 21H1 will not allow WSL or Docker to use the GPU.
  • https://learn.microsoft.com/en-us/windows/ai/directml/gpu-cuda-in-wsl
• WSL2

  • Install WSL2 and/or run wsl --update after installing the latest Windows updates. I'm using Ubuntu.

  • https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#system-requirements

  • Compatible kernel, gcc, and glibc

  • EXAMPLE: Ubuntu 20.04.z (z <= 6) LTS

    • Requires at least kernel 5.15.0-67, gcc 9.40, and glibc 2.31

    $ uname -a
    
    Linux 5.15.153.1-microsoft-standard-WSL2 #1 SMP Fri Mar 29 23:14:13 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
    

    $ gcc --version
    
    gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
    Copyright (C) 2019 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions.  There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
    

    $ ldd --version
    
    ldd (Ubuntu GLIBC 2.31-0ubuntu9.9) 2.31
    Copyright (C) 2020 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions.  There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
    Written by Roland McGrath and Ulrich Drepper.
    

STEP 1: CUDA in Windows

Get cuda working in Windows. Details are beyond the scope of this document.

1. Download the latest driver for your GPU
2. Download the latest CUDA Toolkit
3. Download the latest cuDNN

STEP 2: CUDA in WSL

Get cuda working in WSL without overwriting the driver inherited from Windows

WARNING:

From https://docs.nvidia.com/cuda/wsl-user-guide/index.html

Once a Windows NVIDIA GPU driver is installed on the system, CUDA becomes available within WSL 2.
The CUDA driver installed on Windows host will be stubbed inside the WSL 2 as libcuda.so,
therefore users must not install any NVIDIA GPU Linux driver within WSL 2.
One has to be very careful here as the default CUDA Toolkit comes packaged with a driver,
and it is easy to overwrite the WSL 2 NVIDIA driver with the default installation.
We recommend developers to use a separate CUDA Toolkit for WSL 2 (Ubuntu)
available from the CUDA Toolkit Downloads page to avoid this overwriting.
This WSL-Ubuntu CUDA toolkit installer will not overwrite the NVIDIA driver
that was already mapped into the WSL 2 environment.
To learn how to compile CUDA applications, please read the CUDA documentation for Linux.

Installation

Follow the instructions at https://docs.nvidia.com/cuda/wsl-user-guide/index.html#cuda-support-for-wsl-2 (section 3, Option 1)

Test your installation

$ nvidia-smi

Mon Jul  8 10:09:17 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.42.03              Driver Version: 555.85         CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A4000 Laptop GPU    On  |   00000000:01:00.0 Off |                  N/A |
| N/A   59C    P0             35W /  110W |       0MiB /   8192MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

$ nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Jun__6_02:18:23_PDT_2024
Cuda compilation tools, release 12.5, V12.5.82
Build cuda_12.5.r12.5/compiler.34385749_0

STEP 3: Get ready to build & run mistral.rs

1. Install stuff you'll need to compile mistral.rs:

   sudo apt-get install libssl-dev pkg-config build-essential
   

2. Alter environment variables in ~/.bashrc

   • Adding the CUDA bin directory to your PATH makes sure you can run the nvcc compiler and stuff
   • LD_LIBRARY_PATH makes sure you can actually run the binaries after compiling them
     • without this line you may get the following error: DriverError(CUDA_ERROR_NO_DEVICE, "no CUDA-capable device is detected")
     • You may want to try step 4 first, as it may be safer to avoid altering LD_LIBRARY_PATH if you don't need to

   Alter the following lines for your version of CUDA and then add them to the end of ~/.bashrc

   # change as needed for your version of CUDA
   export PATH="/usr/local/cuda-12.5/bin:$PATH"
   
   # you may not need to add this line if your installation is fresh and clean
   # apparently altering this path can cause problems in some contexts?
   # I think this is safe because we're just adding the dir, not overwriting the var completely
   # export LD_LIBRARY_PATH="/usr/lib/wsl/lib:$LD_LIBRARY_PATH"
   

STEP 4: Build & run mistral.rs

1. Restart your computer to make sure all the changes from above are applied.
2. Follow the installation instructions for mistral.rs, ie

   curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
   source $HOME/.cargo/env
   git clone https://github.com/EricLBuehler/mistral.rs.git
   cd mistral.rs && git pull
   cargo build --release --features "cuda cudnn flash-attn"
   

3. Test your build however you'd like, eg

   cargo run --release --features "cuda cudnn flash-attn" -- --isq Q4K -i plain -m microsoft/Phi-3-mini-128k-instruct -a phi3