tgi server launch fails with latest-rocm docker image.

[gurpreet-dhami]

System Info

text-generation-launcher 2.2.1-dev0

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

This one works
docker run --rm -it --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -p 8080:80 --device=/dev/kfd --device=/dev/dri --group-add video --ipc=host --shm-size 256g -v $PWD:/data --env PYTORCH_TUNABLEOP_ENABLED=0 --env HUGGINGFACE_HUB_CACHE=/data --env ROCM_USE_FLASH_ATTN_V2_TRITON=0 ghcr.io/huggingface/text-generation-inference:2.2.0-rocm --model-id=teknium/OpenHermes-2.5-Mistral-7B

This one fails
docker run --rm -it --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -p 8080:80 --device=/dev/kfd --device=/dev/dri --group-add video --ipc=host --shm-size 256g -v $PWD:/data --env PYTORCH_TUNABLEOP_ENABLED=0 --env HUGGINGFACE_HUB_CACHE=/data --env ROCM_USE_FLASH_ATTN_V2_TRITON=0 ghcr.io/huggingface/text-generation-inference:latest-rocm --model-id=teknium/OpenHermes-2.5-Mistral-7B

from text_generation_server.layers.attention.flashinfer import (

File "/opt/conda/lib/python3.11/site-packages/text_generation_server/layers/attention/flashinfer.py", line 5, in

import flashinfer

ModuleNotFoundError: No module named 'flashinfer'

2024-09-13T18:58:43.898755Z ERROR shard-manager: text_generation_launcher: Shard complete standard error output:

Expected behavior

It should launch the server successfully.