How to Properly Utilize RDMA in UCX?

[AVA022]

I am exploring the best practices for utilizing RDMA in UCX and have a couple of questions regarding its implementation:

1. Are the put/get communication functions the only ones that support RDMA, or can functions like ucp_tag_send_nbx and active messages also invoke RDMA under the right hardware conditions?

2. When using ucp_put_nbx for communication, what is the recommended way to notify the remote side upon completion of the operation?

Thank you for your assistance and guidance!

[yosefe]

Hi,

1. Tag and active messages can also utilize RDMA, mostly for large messages (using rendezvous protocols)
2. Usually one of the following methods is used:
   • ucp_put_nbx(data); ucp_worker_fence(); ucp_put_nbx(flag)
   • ucp_put_nbx(data); ucp_ep_flush/worker_flush(); wait for flush completion; ucm_am_nbx(signal)

There are suggestions to add put-with-signal API to signal remote completion, but it wasn't implemented so far.

[AVA022]

Thank you for your timely response. I have one more question:

• in a scenario with only one server and one client, does the active message guarantee that the order of data received is consistent with the order of sending？

[yosefe]

Yes, active messages and tags have send/recv ordering guarantee

[AVA022]

thank you for your reply :)