UCX cuda latency issue

[lijh5]

1. numactl -N 0 ib_read_lat -d mlx5_0 --use_cuda=0 -a -c RC
   bytes #iterations t_min[usec] t_max[usec] t_typical[usec] t_avg[usec]
   2 1000 3.25 9.45 3.31 3.32
   4 1000 3.22 8.11 3.32 3.32
   8 1000 3.25 3.43 3.32 3.32
   16 1000 3.27 3.41 3.34 3.33
   32 1000 3.24 3.42 3.32 3.32
   64 1000 3.26 3.47 3.33 3.33
   128 1000 3.27 3.47 3.35 3.34
   256 1000 3.30 3.49 3.37 3.38
   512 1000 3.30 3.65 3.39 3.39
   1024 1000 3.35 3.76 3.43 3.43
   2048 1000 3.41 3.67 3.49 3.49
   4096 1000 3.51 3.72 3.61 3.61
   8192 1000 3.64 6.33 3.72 3.73
   16384 1000 3.98 4.26 4.05 4.05
   32768 1000 4.31 4.53 4.42 4.41
   65536 1000 5.09 5.24 5.16 5.15
   131072 1000 7.28 7.59 7.35 7.34
   262144 1000 10.18 10.34 10.27 10.26
   524288 1000 16.01 16.19 16.09 16.09
   1048576 1000 27.67 33.20 27.75 27.76
   2097152 1000 51.00 51.17 51.07 51.07
   4194304 1000 97.63 98.23 97.71 97.71

2. mpirun -x UCX_TLS=rc,cuda -x UCX_ZCOPY_THRESH=auto -x UCX_RNDV_THRESH=auto -x UCX_MEMTYPE_CACHE=n numactl -N 0 osu_latency -d CUDA D D
   Size Latency (us)
   1 8.75
   2 8.91
   4 9.11
   8 9.28
   16 9.50
   32 9.67
   64 9.83
   128 9.84
   256 9.94
   512 9.66
   1024 9.46
   2048 9.59
   4096 9.98
   8192 10.54
   16384 17.85
   32768 32.04
   65536 7.15
   131072 8.51
   262144 11.72
   524288 17.04
   1048576 27.72
   2097152 48.90
   4194304 91.22

The latency ucx of small packages is much worse, while the delay of large packages is about the same.
May I ask if there are any parameters that can optimize the small packet latency of osu_latency?
Use hpcx-v2.17.1 ucx 1.16

Thank you.

[ivankochin]

Try to set UCX_RNDV_THRESH=0 and UCX_RNDV_SCHEME=get_zcopy

[lijh5]

OSU MPI-CUDA Latency Test v7.2
Send Buffer on DEVICE (D) and Receive Buffer on DEVICE (D)
Size Latency (us)
Datatype: MPI_CHAR.
1 12.28
2 12.55
4 13.07
8 13.18
16 13.36
32 13.30
64 20.70
128 20.27
256 5.92
512 5.98
1024 5.92
2048 5.98
4096 6.05
8192 6.23
16384 6.76
32768 7.00
65536 7.47
131072 8.77
262144 11.92
524288 17.23
1048576 27.82
2097152 48.99
4194304 91.31

There are still performance issues！

[ivankochin]

Potential reason is that ib_read_lat measures latency for RDMA-read operation only, but osu_latency uses RNDV protocol which includes sending control message with metainformation for performing RMA operation, RMA operation itself and ACK message. On small sizes these two additional messages can create significant overhead.

Please try to measure ucx_perftest -t get (see ucx_perftest -h for usage)

[lijh5]

Only in osu_latency -d CUDA D D mode can the latency of small packets be high, while in -d CUDA H H mode, the latency of small packets is consistent with perftest(ib_write_bw).
The reason I analyzed may be that UCX does not use the GPU Direct RDMA method when processing small packets, but instead copies them to the CPU before sending them, or receives them from the CPU before copying them to the GPU.
So I tried the following parameters to force ucx to use GPU Direct RDMA when processing small packets:
UCX_ZCOPY_THRESH=1 -x UCX_RNDV_THRESH=1 -x UCX_RNDV_SCHEME=put_zcopy
But the effect is still not ideal...
(I'm sorry, I don't have any GPU nodes at the moment to verify the performance of ucx_perftest, but I think there will also be the same problem in ucx_perftest, how to ensure that small packets are transmitted using GPU Direct RDMA instead of CPU participating in copying?)

[ivankochin]

To understand which protocol is in use you can measure benchmark with setting UCX_PROTO_INFO=y (pls share the output so I can check it too).

[lijh5]

What I mentioned earlier was all tested with UCX_PROTO_INFO=y enabled, using version ucx1.16. There is currently no GPU node, and I will continue testing when I have the environment.
Thank you very much for your answer!