Regarding the issue of GPU communication

[lijh5]

May I ask what the function of parameter UCX_IB_GPU_DIRECT_RDMA is? In the UCX code, I only found the processing logic for UCX_IB_GPU_DIRECT_RDMA!=0
In uct_ib_md_open_common:
if (md_config->enable_gpudirect_rdma != UCS_NO) {
/* check if GDR driver is loaded /
uct_ib_check_gpudirect_driver(md, "/sys/kernel/mm/memory_peers/nv_mem/version", UCS_MEMORY_TYPE_CUDA);
/ check if ROCM KFD driver is loaded /
uct_ib_check_gpudirect_driver(md, "/dev/kfd", UCS_MEMORY_TYPE_ROCM);
/ Check for dma-buf support */
uct_ib_md_check_dmabuf(md);
}

I have conducted some experiments and found that setting UCX_IB_GPU_DIRECT_RDMA=0 does indeed cause the network card to no longer directly manipulate GPU memory data, but instead copy it to the CPU through cuda_copy before executing the RDMA operation.

May I ask where else can the ucx source code determine UCX_IB_GPU_DIRECT_RDMA=0, to decide whether to use GPU Direct RDMA?
Thank you.

[yosefe]

the default behavior is to enable GPU direct if nv_peer_memory driver is loaded, unless UCX_IB_GPU_DIRECT_RDMA is set to 0