WIS on WSL with GPU #118

mrrudy · 2023-09-18T10:03:42Z

mrrudy
Sep 18, 2023

Hi,
was able to setup WIS in WSL but only on CPU. I have GPU available through libcuda (tested with NVIDIA NGC TensorFlow container). Can't figure out how to force WIS container to use libcuda as it rightfully detects there is no GPU present and switches to CPU mode.

kind regards

Answered by mrrudy

Sep 18, 2023

Solved it!
I really don't know yet why but... I've asked myself what I'm I'm doing differently and it was using sudo for docker. I thought it does not make a difference and I'm lazy so I'll add my user later to the docker group and use sudo for now. Still don't know why but running it from user works:

willow-inference-server-wis-1    | [2023-09-18 11:58:01 +0000] [89] [INFO] Willow Inference Server is starting... Please wait.
willow-inference-server-wis-1    | [2023-09-18 11:58:01 +0000] [89] [INFO] CUDA: Detected 1 device(s)
willow-inference-server-wis-1    | [2023-09-18 11:58:01 +0000] [89] [INFO] CUDA: Device 0 name: NVIDIA GeForce GTX 1080
willow-inference-server-wis-1    | [2023-09-1…

View full answer

kristiankielhofner · 2023-09-18T10:14:33Z

kristiankielhofner
Sep 18, 2023
Maintainer

Can you provide the output of nvidia-smi from inside WSL?

Are you running Ubuntu WSL? If so, did you run the ./deps/ubuntu.sh script?

0 replies

paulgrove · 2023-09-18T10:53:00Z

paulgrove
Sep 18, 2023

I know this doesn't help much, but I can confirm that I run WIS in WSL2 on the GPU @mrrudy - it does work.

0 replies

mrrudy · 2023-09-18T11:00:05Z

mrrudy
Sep 18, 2023
Author

$ nvidia-smi
Mon Sep 18 12:33:13 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.07             Driver Version: 537.34       CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1080        On  | 00000000:01:00.0  On |                  N/A |
| 31%   39C    P8               9W / 180W |   1977MiB /  8192MiB |     27%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A        29      G   /Xwayland                                 N/A      |
+---------------------------------------------------------------------------------------+

Yes, run the dep script during deployment, re-run to confirm:

sudo ./deps/ubuntu.sh
Get:1 file:/var/cuda-repo-wsl-ubuntu-12-2-local  InRelease [1572 B]
Get:1 file:/var/cuda-repo-wsl-ubuntu-12-2-local  InRelease [1572 B]
Hit:2 https://download.docker.com/linux/ubuntu jammy InRelease
Hit:3 https://nvidia.github.io/libnvidia-container/stable/deb/amd64  InRelease
Hit:4 https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/amd64  InRelease
Hit:5 https://nvidia.github.io/nvidia-container-runtime/stable/ubuntu18.04/amd64  InRelease
Hit:6 https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64  InRelease
Hit:7 http://security.ubuntu.com/ubuntu jammy-security InRelease
Hit:8 http://archive.ubuntu.com/ubuntu jammy InRelease
Hit:9 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:10 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
nvidia-container-toolkit is already the newest version (1.14.1-1).
0 upgraded, 0 newly installed, 0 to remove and 49 not upgraded.
INFO[0000] Loading config from /etc/docker/daemon.json
INFO[0000] Wrote updated config to /etc/docker/daemon.json
INFO[0000] It is recommended that docker daemon be restarted.

cleaned containers (docker rm $(docker ps -aq))

and re-run install:

sudo ./utils.sh install
Using configuration overrides from .env file
NVIDIA GPU Support not detected - using CPU
[+] Building 2.6s (13/13) FINISHED
...

sudo ./utils.sh run
Using configuration overrides from .env file
NVIDIA GPU Support not detected - using CPU
[+] Running 2/0
 ✔ Container willow-inference-server-wis-1    Created                                                                                                  0.0s
 ✔ Container willow-inference-server-nginx-1  Created                                                                                                  0.0s
Attaching to willow-inference-server-nginx-1, willow-inference-server-wis-1
willow-inference-server-wis-1    |
willow-inference-server-wis-1    | =====================
willow-inference-server-wis-1    | == NVIDIA TensorRT ==
willow-inference-server-wis-1    | =====================
...
willow-inference-server-wis-1    | [2023-09-18 10:54:56 +0000] [62] [INFO] Willow Inference Server is starting with custom settings... Please wait.
willow-inference-server-wis-1    | [2023-09-18 10:54:56 +0000] [62] [INFO] CUDA: Not found - using CPU with 12 cores
willow-inference-server-wis-1    | [2023-09-18 10:54:56 +0000] [62] [INFO] Started server process [62]
willow-inference-server-wis-1    | [2023-09-18 10:54:56 +0000] [62] [INFO] Waiting for application startup.
willow-inference-server-wis-1    | [2023-09-18 10:54:56 +0000] [62] [INFO] CTRANSLATE: Supported compute types for device cpu are {'float32', 'int8', 'int8_float32', 'int16'}- using configured int8
willow-inference-server-wis-1    | [2023-09-18 10:54:56 +0000] [62] [INFO] Loading Whisper models...
willow-inference-server-wis-1    | [2023-09-18 10:55:02 +0000] [62] [INFO] Loading TTS models...
willow-inference-server-wis-1    | [2023-09-18 10:55:04 +0000] [62] [INFO] Skipping warm_models for CPU
willow-inference-server-wis-1    | [2023-09-18 10:55:04 +0000] [62] [INFO] Willow Inference Server is ready for requests!
willow-inference-server-wis-1    | [2023-09-18 10:55:04 +0000] [62] [INFO] Application startup complete.

happy to debug more, I can wipe the WSL completely if it helps.

0 replies

kristiankielhofner · 2023-09-18T11:09:35Z

kristiankielhofner
Sep 18, 2023
Maintainer

Can you try running the Nvidia "hellow world"?

docker run --rm --gpus all nvidia/cuda:12.2.0-runtime-ubuntu22.04 nvidia-smi

0 replies

mrrudy · 2023-09-18T11:19:29Z

mrrudy
Sep 18, 2023
Author

sudo docker run --rm --gpus all nvidia/cuda:12.2.0-runtime-ubuntu22.04 nvidia-smi
[sudo] password for rudy:
Unable to find image 'nvidia/cuda:12.2.0-runtime-ubuntu22.04' locally
12.2.0-runtime-ubuntu22.04: Pulling from nvidia/cuda
6b851dcae6ca: Pull complete
8f5f0e71700a: Pull complete
fac7ce4a13c3: Pull complete
1af9bee222cb: Pull complete
d47e0a26d15c: Pull complete
5d029ef24527: Pull complete
07875bb7068a: Pull complete
da99fb0c01a6: Pull complete
65a5bb1ab8c6: Pull complete
Digest: sha256:0f4f33b31222af3e4d66aea2af0a04368a44160af815b6ae7bfe36f5b536ab99
Status: Downloaded newer image for nvidia/cuda:12.2.0-runtime-ubuntu22.04

==========
== CUDA ==
==========

CUDA Version 12.2.0

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

Mon Sep 18 11:16:03 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.07             Driver Version: 537.34       CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1080        On  | 00000000:01:00.0  On |                  N/A |
| 31%   37C    P8              11W / 180W |   1794MiB /  8192MiB |      5%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A        32      G   /Xwayland                                 N/A      |
+---------------------------------------------------------------------------------------+

10 replies

kristiankielhofner Sep 18, 2023
Maintainer

Very, very interesting... Can you try with our container you built:

docker run --rm --gpus all willow-inference-server:latest nvidia-smi

mrrudy Sep 18, 2023
Author

indeed. I even wrote for the sanity check this 1-liner
if command -v nvidia-smi &> /dev/null; then echo "yes GPU"; else echo "no GPU"; fi
with expected result:
yes GPU

I did wipe the WSL and used only our scripts to ensure no interference from me. the only out of ordinary I got was
/sbin/ldconfig.real: /usr/lib/wsl/lib/libcuda.so.1 is not a symbolic link
during installation but that should not be an issue.

and your request:

sudo docker run --rm --gpus all willow-inference-server:latest  nvidia-smi

=====================
== NVIDIA TensorRT ==
=====================

NVIDIA Release 22.12 (build 49968236)
NVIDIA TensorRT Version 8.5.1
Copyright (c) 2016-2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Container image Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

https://developer.nvidia.com/tensorrt

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

To install Python sample dependencies, run /opt/tensorrt/python/python_setup.sh

To install the open-source samples corresponding to this TensorRT release version
run /opt/tensorrt/install_opensource.sh.  To build the open source parsers,
plugins, and samples for current top-of-tree on master or a different branch,
run /opt/tensorrt/install_opensource.sh -b <branch>
See https://github.com/NVIDIA/TensorRT for more information.

Mon Sep 18 11:42:00 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.07             Driver Version: 537.34       CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1080        On  | 00000000:01:00.0  On |                  N/A |
| 32%   36C    P8              10W / 180W |   1851MiB /  8192MiB |      7%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A        32      G   /Xwayland                                 N/A      |
+---------------------------------------------------------------------------------------+

mrrudy Sep 18, 2023
Author

Solved it!
I really don't know yet why but... I've asked myself what I'm I'm doing differently and it was using sudo for docker. I thought it does not make a difference and I'm lazy so I'll add my user later to the docker group and use sudo for now. Still don't know why but running it from user works:

willow-inference-server-wis-1    | [2023-09-18 11:58:01 +0000] [89] [INFO] Willow Inference Server is starting... Please wait.
willow-inference-server-wis-1    | [2023-09-18 11:58:01 +0000] [89] [INFO] CUDA: Detected 1 device(s)
willow-inference-server-wis-1    | [2023-09-18 11:58:01 +0000] [89] [INFO] CUDA: Device 0 name: NVIDIA GeForce GTX 1080
willow-inference-server-wis-1    | [2023-09-18 11:58:01 +0000] [89] [INFO] CUDA: Device 0 capability: 61
willow-inference-server-wis-1    | [2023-09-18 11:58:01 +0000] [89] [INFO] CUDA: Device 0 total memory: 8589803520 bytes
willow-inference-server-wis-1    | [2023-09-18 11:58:01 +0000] [89] [INFO] CUDA: Device 0 free memory: 7550795776 bytes
willow-inference-server-wis-1    | [2023-09-18 11:58:01 +0000] [89] [WARNING] CUDA: Device 0 has low memory, disabling chunking support
willow-inference-server-wis-1    | [2023-09-18 11:58:01 +0000] [89] [WARNING] CUDA: Device 0 is pre-Volta, forcing int8
willow-inference-server-wis-1    | [2023-09-18 11:58:02 +0000] [89] [INFO] Started server process [89]
willow-inference-server-wis-1    | [2023-09-18 11:58:02 +0000] [89] [INFO] Waiting for application startup.

Happy to debug further as this is strange as ... well out of ordinary. I run dozens of docker images on proxmox lxc's and always as root as those are unprivileged.

Answer selected by mrrudy

kristiankielhofner Sep 18, 2023
Maintainer

Interesting, but glad you got it up! Let us know how things go.

Also, the current main "rough edge" with WIS on WSL for most users is dealing with the Windows Firewall and port forwarding to WSL. If you'd be willing to contribute a Powershell script or similar to manage that we would really appreciate it!

mrrudy Sep 18, 2023
Author

ofcourse. quick and dirty one, powershell needs to be run as admin:

$wslIP = wsl ip addr show eth0 | wsl grep -Po '(?<=inet )\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}' 

$ports2forward = 19001, 19000

foreach ($port in $ports2forward) {
    Write-Output "Creating forwarding rule for port: $port of IP: $wslIP"
    netsh interface portproxy add v4tov4 listenport=$port listenaddress=0.0.0.0 connectport=$port connectaddress=$wslIP 
    Write-Output "Opening windows firewall port: $port"
    netsh advfirewall firewall add rule name= "Open Port $port for wis on wsl" dir=in action=allow protocol=TCP localport=$port 
}

It will work for only one, default WSL distro, also ports are manual. If you tell me how to get the ports (now I can get it from docker but wondering how to make it future-proof) I can introduce it. Is there a thread I should discuss it? I do not think it should be part of the main build but lets see whats your opinion.

mikey60 Nov 4, 2023

The following is a PowerShell Script(network.ps1) that I found on the internet. It also needs to be run as an administrator, but will ask to be run as administrator if run as a user. The desired ports also need to edited in the script. The output seems to be a little more verbose.

If (-NOT ([Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole] "Administrator")) {   
  $arguments = "& '" + $myinvocation.mycommand.definition + "'"
  Start-Process powershell -Verb runAs -ArgumentList $arguments
  Break
}

$remoteport = bash.exe -c "ifconfig eth0 | grep 'inet '"
$found = $remoteport -match '\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}';

if ($found) {
  $remoteport = $matches[0];
}
else {
  Write-Output "IP address could not be found";
  exit;
}

$ports = @(1883,12101,19000,8501,8502,443);

for ($i = 0; $i -lt $ports.length; $i++) {
  $port = $ports[$i];
  Invoke-Expression "netsh interface portproxy delete v4tov4 listenport=$port";
  Invoke-Expression "netsh advfirewall firewall delete rule name=$port";

  Invoke-Expression "netsh interface portproxy add v4tov4 listenport=$port connectport=$port connectaddress=$remoteport";
  Invoke-Expression "netsh advfirewall firewall add rule name=$port dir=in action=allow protocol=TCP localport=$port";
}

Invoke-Expression "netsh interface portproxy show v4tov4";

kristiankielhofner Nov 6, 2023
Maintainer

The last time I tested this it seemed that the forwarded ports don't persist across reboots. Can you confirm this is still the case currently with this approach? I imagine that would be the next wrinkle that will trip up Windows users. I don't have the knowledge/experience to figure out how to handle this reboot issue.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIS on WSL with GPU #118

{{title}}

Replies: 5 comments 10 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

WIS on WSL with GPU #118

mrrudy Sep 18, 2023

Replies: 5 comments · 10 replies

kristiankielhofner Sep 18, 2023 Maintainer

paulgrove Sep 18, 2023

mrrudy Sep 18, 2023 Author

kristiankielhofner Sep 18, 2023 Maintainer

mrrudy Sep 18, 2023 Author

kristiankielhofner Sep 18, 2023 Maintainer

mrrudy Sep 18, 2023 Author

mrrudy Sep 18, 2023 Author

kristiankielhofner Sep 18, 2023 Maintainer

mrrudy Sep 18, 2023 Author

mikey60 Nov 4, 2023

kristiankielhofner Nov 6, 2023 Maintainer

mrrudy
Sep 18, 2023

Replies: 5 comments 10 replies

kristiankielhofner
Sep 18, 2023
Maintainer

paulgrove
Sep 18, 2023

mrrudy
Sep 18, 2023
Author

kristiankielhofner
Sep 18, 2023
Maintainer

mrrudy
Sep 18, 2023
Author

kristiankielhofner Sep 18, 2023
Maintainer

mrrudy Sep 18, 2023
Author

mrrudy Sep 18, 2023
Author

kristiankielhofner Sep 18, 2023
Maintainer

mrrudy Sep 18, 2023
Author

kristiankielhofner Nov 6, 2023
Maintainer