Data copying in AM fragments receive handlers

[huzhijiang]

Hi,
According to

ucx/src/ucp/core/ucp_am.c

Line 1491 in f3f911f

first_rdesc = ucs_malloc(total_length + sizeof(ucp_recv_desc_t) +

it seems if we send AM data in fragments, the receiver side needs to allocate a contig buffer to hold all of them. This could be a potential perfomance bottleneck. Sending data in a single piece will not leading to this copying on the receiver side but sender side multi-rail function will not work either(see

ucx/src/ucp/proto/proto_am.inl

Line 423 in f3f911f

req->send.lane = ucp_ep_get_am_lane(ep);

).

Do we have any other optimal way (can be AM or TAG, HW offload...) to not only avoid fragments copy on receiver side , but also enable multi-rail on sender side?

[evgeny-leksikov]

@huzhijiang rndv protocols should enable multi-rail by default and utilize zcopy rdma capability, you can enable it by env var UCX_RNDV_THRESH=<msg_size> or control per message by UCP_AM_SEND_FLAG_RNDV from your code.

[huzhijiang]

Hi @evgeny-leksikov ,
Thanks for the info.