Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UCP: Enable ppln protos with cuda buffers by default #10104

Open
wants to merge 13 commits into
base: master
Choose a base branch
from

Conversation

brminich
Copy link
Contributor

What

Enable using cuda staging buffers for pipeline protocols by default.

Why ?

Using ppln protocols with cuda staging buffers provides better performance for the following cases:

  • sending data from cuda-managed memory to cuda memory
  • sending data between different nodes where MNNVLK is available
  • etc

How ?

Rename RNDV_FRAG_MEM_TYPE to RNDV_FRAG_MEM_TYPES and allow to specify a list of memory types. Then initialize RNDV_RECEIVE protocol variants (rtr_mtype and get_mtype) with these memory types. RNDV_SEND protocol has to use the mem type which corresponds to the rkey from received RTR, because:

  • We use different fragment sizes for different mem_types, so rtr_mtype and put_mtype sizes will be different which breaks the protocol
  • Currently there is no strong use-case to support ppln protocols with different mem_type fragments being used by peers

@brminich brminich marked this pull request as ready for review September 18, 2024 19:06
@brminich brminich force-pushed the ucp/dev_mem_ppln_by_default branch 2 times, most recently from 055e489 to 2f4b402 Compare September 20, 2024 12:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant