how do I enable NVLink / peer transfers?

[recallmenot]

Hi,
111163e (#2470) should have enabled peer access to use the NVLink bridge.
However, I can not get this to work at all, getting the same token/s as with just PCIe (3.0x4 link so the bridge should be doing something).
I've noticed that in ggml/src/ggml-cuda/ggml-cuda.cu, ggml_cuda_set_peer_access() is never reached as ggml_backend_buft_is_cuda_split() in ggml_cuda_compute_forward() returns 0.

The GPUs:

nvidia-smi topo -p2p r
 	GPU0	GPU1	
 GPU0	X	OK	
 GPU1	OK	X	

nvidia-smi nvlink -s      
GPU 0: NVIDIA GeForce RTX 3090 (UUID: GPU-...)
	 Link 0: 14.062 GB/s
	 Link 1: 14.062 GB/s
	 Link 2: 14.062 GB/s
	 Link 3: 14.062 GB/s
GPU 1: NVIDIA GeForce RTX 3090 (UUID: GPU-...)
	 Link 0: 14.062 GB/s
	 Link 1: 14.062 GB/s
	 Link 2: 14.062 GB/s
	 Link 3: 14.062 GB/s

Of course watch -n 1 nvidia-smi nvlink --getthroughput d shows the links completely idle.
If I run cuda-samples' p2pBandwidthLatencyTest, I get

Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1 
     0 833.33   2.80 
     1   2.79 834.22 
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1 
     0 791.94  52.79 
     1  52.82 836.01 
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1 
     0 818.24   4.12 
     1   3.83 839.60 
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1 
     0 818.71 101.46 
     1 101.84 839.15 

So it's safe to assume my NVLink bridge works fine?

[recallmenot]

maybe, as the author of the NVLink patch, @JohannesGaessler can chime in?

[JohannesGaessler]

The tensor parallel support in llama.cpp is very rudimentary and unoptimized, you can try enabling it with --split-mode row.

[recallmenot]

Thank you very much!
So I did some testing with NVLink.

--split-mode row \

ggml/src/ggml-cuda/ggml-cuda.cu unmodified
18.5 t/s
ggml_cuda_set_peer_access() called but #ifdef NDEBUG prevents peer transfer setup

--split-mode row \

insert at the top of ggml/src/ggml-cuda/ggml-cuda.cu:

#define NDEBUG 1

22.5 t/s
only here, nvidia-smi nvlink --getthroughput d shows link activity

What surprised me is that this is slower than the default layer mode:

--split-mode layer \

26.4 t/s
Here, ggml_cuda_set_peer_access() is never called.

Are these results in line with what you'd expect?

So in other words, NVLink is not useful for LLM inference in the current state of row mode?
How come, I've seen posts claiming a 2x speedup with NVLink.
Or has layer mode just become that good?

model used: 32b R1 Qwen Q6
gpus: 2x 3090
other parameters:

--n-gpu-layers 65 \
--ctx-size 40000 \

building with:

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j $(nproc)