[Bug] Seeing random output with nvidia/Llama-3.1-Nemotron-70B-Reward

[pgimenes]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

I am trying to run the basic setup example with the nvidia/Llama-3.1-Nemotron-70B-Reward on a machine with 8 A6000s, but the observed output is random.

Generated output:

{"text":"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"","meta_info":{"prompt_tokens":6,"completion_tokens":16,"completion_tokens_wo_jump_forward":16,"cached_tokens":1,"finish_reason"{"type":"length","length":16},"id":"a3101b373bcd4fb8beab6f42e74efd53"}

Reproduction

Server side:

python -m sglang.launch_server --model-path nvidia/Llama-3.1-Nemotron-70B-Reward-HF --port 30000 --tp 8

Client side:

curl http://localhost:30000/generate \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Once upon a time,",
    "sampling_params": {
      "max_new_tokens": 16,
      "temperature": 0
    }
  }'

Environment

Python: 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA RTX A6000
GPU 0,1,2,3,4,5,6,7 Compute Capability: 8.6
CUDA_HOME: /usr
NVCC: Cuda compilation tools, release 11.5, V11.5.119
CUDA Driver Version: 550.54.15
PyTorch: 2.4.0+cu121
sglang: 0.3.5
flashinfer: 0.1.6+cu121torch2.4
triton: 3.0.0
transformers: 4.46.1
requests: 2.32.3
tqdm: 4.66.6
numpy: 1.26.4
aiohttp: 3.10.10
fastapi: 0.115.4
hf_transfer: 0.1.8
huggingface_hub: 0.26.2
interegular: 0.3.3
packaging: 24.1
PIL: 10.4.0
psutil: 6.1.0
pydantic: 2.9.2
uvicorn: 0.32.0
uvloop: 0.21.0
zmq: 26.2.0
vllm: 0.6.3.post1
multipart: 0.0.17
openai: 1.54.1
anthropic: 0.39.0
NVIDIA Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X SYS SYS SYS SYS SYS SYS SYS SYS 0-63,128-191 0 N/A
GPU1 SYS X SYS SYS SYS SYS SYS SYS SYS 0-63,128-191 0 N/A
GPU2 SYS SYS X SYS SYS SYS SYS SYS SYS 0-63,128-191 0 N/A
GPU3 SYS SYS SYS X SYS SYS SYS SYS SYS 0-63,128-191 0 N/A
GPU4 SYS SYS SYS SYS X SYS SYS SYS SYS 64-127,192-255 1 N/A
GPU5 SYS SYS SYS SYS SYS X SYS SYS SYS 64-127,192-255 1 N/A
GPU6 SYS SYS SYS SYS SYS SYS X SYS PHB 64-127,192-255 1 N/A
GPU7 SYS SYS SYS SYS SYS SYS SYS X SYS 64-127,192-255 1 N/A
NIC0 SYS SYS SYS SYS SYS SYS PHB SYS X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0

ulimit soft: 4096