Add asynchronous concurrent execution #3687
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
neon60
force-pushed
the
async-doc
branch
2 times, most recently
from
1484d67 to
f81588d
Compare
neon60
requested review from
chrispaquot,
gandryey,
saleelk,
mangupta and
rakesroy
as code owners
WIP WIP WIP WIP WIP
neon60
force-pushed
the
async-doc
branch
3 times, most recently
from
a8fd499 to
fd5af51
Compare
randyh62
approved these changes
Dec 11, 2024
| Asynchronous concurrent execution | ||
| ******************************************************************************* | ||
|
|
||
| Asynchronous concurrent execution important for efficient parallelism and |
There was a problem hiding this comment.
Suggested change
| Asynchronous concurrent execution important for efficient parallelism and | |
| Asynchronous concurrent execution is important for efficient parallelism and |
| Asynchronous concurrent execution important for efficient parallelism and | ||
| resource utilization, with techniques such as overlapping computation and data | ||
| transfer, managing concurrent kernel execution with streams on single or | ||
| multiple devices or using HIP graphs. |
There was a problem hiding this comment.
Suggested change
| multiple devices or using HIP graphs. | |
| multiple devices, or using HIP graphs. |
| data allocation/freeing all happen in the context of device streams. | ||
|
|
||
| Streams are FIFO buffers of commands to execute in order on a given device. | ||
| Commands which enqueue tasks on a stream all return promptly and the command is |
There was a problem hiding this comment.
Suggested change
| Commands which enqueue tasks on a stream all return promptly and the command is | |
| Commands which enqueue tasks on a stream all return promptly and the task is |
|
|
||
| Streams are FIFO buffers of commands to execute in order on a given device. | ||
| Commands which enqueue tasks on a stream all return promptly and the command is | ||
| executed asynchronously. Multiple streams may point to the same device and |
There was a problem hiding this comment.
Suggested change
| executed asynchronously. Multiple streams may point to the same device and | |
| executed asynchronously. Multiple streams can point to the same device and |
| Commands which enqueue tasks on a stream all return promptly and the command is | ||
| executed asynchronously. Multiple streams may point to the same device and | ||
| those streams may be fed from multiple concurrent host-side threads. Execution | ||
| on multiple streams may be concurrent but isn't required to be. |
There was a problem hiding this comment.
Suggested change
| on multiple streams may be concurrent but isn't required to be. | |
| on multiple streams might be concurrent but isn't required to be. |
| contention for shared resources. This is because multiple kernels may attempt | ||
| to access the same GPU resources simultaneously, leading to delays. | ||
|
|
||
| Asynchronous kernel execution is beneficial only under specific conditions It |
There was a problem hiding this comment.
Suggested change
| Asynchronous kernel execution is beneficial only under specific conditions It | |
| Asynchronous kernel execution is beneficial only under specific conditions. It |
| or from the GPU concurrently with kernel execution. Applications can query this | ||
| capability by checking the ``asyncEngineCount`` device property. Devices with | ||
| an ``asyncEngineCount`` greater than zero support concurrent data transfers. | ||
| Additionally, if host memory is involved in the copy, it should be page-locked |
There was a problem hiding this comment.
Is there a reference we can provide such as Memory Management or something?
|
|
||
| It is also possible to perform intra-device copies simultaneously with kernel | ||
| execution on devices that support the ``concurrentKernels`` device property | ||
| and/or with copies to or from the device (for devices that support the |
There was a problem hiding this comment.
Suggested change
| and/or with copies to or from the device (for devices that support the | |
| and/or with copies to or from the device (for devices that support the |
Are copies to or from the device intra-device copies?
| called, control is not returned to the host thread before the device has | ||
| completed the requested task. The behavior of the host thread—whether to yield, | ||
| block, or spin—can be specified using :cpp:func:`hipSetDeviceFlags` with | ||
| specific flags. Understanding when to use synchronous calls is important for |
There was a problem hiding this comment.
Suggested change
| specific flags. Understanding when to use synchronous calls is important for | |
| appropriate flags. Understanding when to use synchronous calls is important for |
| By creating an event with :cpp:func:`hipEventCreate` and recording it with | ||
| :cpp:func:`hipEventRecord`, developers can synchronize operations across | ||
| streams, ensuring correct task execution order. :cpp:func:`hipEventSynchronize` | ||
| allows waiting for an event to complete before proceeding with the next |
There was a problem hiding this comment.
Suggested change
| allows waiting for an event to complete before proceeding with the next | |
| lets the application wait for an event to complete before proceeding with the next |
randyh62
reviewed
Dec 11, 2024
| sequences of kernels and memory operations as a single graph, they simplify | ||
| complex workflows and enhance performance, particularly for applications with | ||
| intricate dependencies and multiple execution stages. | ||
|
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.