UCT/CUDA: Update cuda_copy perf estimates for Grace-Hopper #10155
Conversation
SeyedMir
force-pushed
the
cuda-estimate-perf-conf
branch
from
9f8dc24
to
f6964ee
Compare
| perf_attr->bandwidth.shared = zcopy ? iface->config.bw.h2d : | ||
| iface->config.bw.h2d * 0.95; | ||
| } else if ((src_mem_type == UCS_MEMORY_TYPE_CUDA) && | ||
| (dst_mem_type == UCS_MEMORY_TYPE_HOST)) { | ||
| perf_attr->bandwidth.shared = (zcopy ? 11660.0 : 9320.0) * | ||
| UCS_MBYTE; | ||
| perf_attr->bandwidth.shared = zcopy ? iface->config.bw.d2h : | ||
| iface->config.bw.d2h * 0.95; |
There was a problem hiding this comment.
Why bcopy BW is slower than zcopy one? BTW 11660 * 0.95 is not equal to 9320. Maybe we need to introduce to different env variables like BCOPY_BW and ZCOPY_BW to control this values accurately. Or if we are OK with changing performance in common case, maybe better just not to distinguish bcopy/zcopy perf and set one value in both cases?
There was a problem hiding this comment.
It's actually not zcopy vs. bcopy; it's zcopy vs. short. Unlike zcopy, put/get short operations invoke cuStreamSynchronize per operation. Therefore, we want to advertise a slightly lower bw for the short vs zcopy operation for cuda_copy.
I'm not sure what 9320 represents. Why do you want it to be equal to 9320?
There was a problem hiding this comment.
Thanks for the explanation. I am thinking about whether we need this difference to be made by this way because each change to performance estimation without proper performance testing can lead to unforeseen degradation in some cases. So if we want to tune performance for GH systems only I would like to leave performance for other platforms untouched. Or if we are OK to change performance on all platforms in that PR, I am wondering whether this 5% difference really matters or we can follow this KISS principle and set the same values for both zcopy and short cases.
| ucs_offsetof(uct_cuda_copy_iface_config_t, bw.d2h)}, | ||
| {"d2d", "device to device bandwidth", | ||
| ucs_offsetof(uct_cuda_copy_iface_config_t, bw.d2d)}, | ||
| {"other", "any other src-dest memory types bandwidth", |
There was a problem hiding this comment.
Minor
| {"other", "any other src-dest memory types bandwidth", | |
| {"other", "any other memory types combinations bandwidth", |
| ucs_offsetof(uct_cuda_copy_iface_config_t, bandwidth), UCS_CONFIG_TYPE_BW}, | ||
| /* TODO: 1. Add separate keys for shared and dedicated bandwidth | ||
| 2. Remove the "other" key (use pref_loc for managed memory) */ | ||
| {"BW", "h2d:8300MBs,d2h:11660MBs,d2d:320GBs,other:10000MBs", |
There was a problem hiding this comment.
Is that possible to remove other and define value for other memory types combinations through value without label?
| {"BW", "h2d:8300MBs,d2h:11660MBs,d2d:320GBs,other:10000MBs", | |
| {"BW", "10000MBs,h2d:8300MBs,d2h:11660MBs,d2d:320GBs", |
What
Update cuda_copy perf estimates for Grace-Hopper
Why ?
The bandwidth and latency values will be different for PCIe versus C2C links that connect CPU and GPU.
How ?
Update the cuda_copy bw config and UCX_CUDA_COPY_BW.