TEZ-4580: Slow preemption of new containers when re-use is enabled #374
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
🎊 +1 overall
This message was automatically generated. |
himanshu-mishra
commented
Sep 27, 2024
| if (delayedContainers.remove(container)) { | ||
| LOG.info("Removed {} from delayed containers", container.getContainer().getId()); | ||
| } else { | ||
| LOG.warn("Unknown container {} sent for removal. Ignoring.", container.getContainer().getId()); |
There was a problem hiding this comment.
A valid case here is when a new container is allocated - added to delayedContainers, it is polled (removed) from queue but if there is no pending task request releaseContainer() is invoked. Please suggest if the log level should be changed in info or for both the newly added logs should be at debug.
There was a problem hiding this comment.
nit: I think we should change the log level to debug and print the only if LOG.isDebugEnabled() is true,
Rest of the code LGTM .
There was a problem hiding this comment.
not sure about the log levels, let's keep the simple remove path on debug, as looks like the happy preemption path:
LOG.debug("Removed {} from delayed containers", container.getContainer().getId());
isDebugEnabled is not necessary as long as we use the {} formatting (to prevent unnecessary string creation)
There was a problem hiding this comment.
Agreed.
Thank you @abstractdog for the review ! :)
prabhjyotsingh
added a commit
to acceldata-io/tez
that referenced
this pull request
Nov 14, 2024
prabhjyotsingh
added a commit
to acceldata-io/tez
that referenced
this pull request
Nov 20, 2024
…nmerged) apache#374 (cherry picked from commit aea86ca)
prabhjyotsingh
added a commit
to acceldata-io/tez
that referenced
this pull request
Nov 20, 2024
…enabled (unmerged) apache#374 (#24) (cherry picked from commit aea86ca)
shubhluck
pushed a commit
to acceldata-io/tez
that referenced
this pull request
Nov 21, 2024
…enabled (unmerged) apache#374 (#24) (cherry picked from commit aea86ca)
|
@abstractdog can you please take a look when you are free. Thanks in advance! :) |
abstractdog
reviewed
Dec 21, 2024
| @@ -2163,6 +2166,17 @@ void addDelayedContainer(Container container, | |||
| } | |||
| } | |||
|
|
|||
| void removeDelayedContainer(HeldContainer container) { | |||
| if (container != null) { | |||
There was a problem hiding this comment.
this null check is not needed, as it's called from a block
if (delayedContainer != null) {
|
thanks @himanshu-mishra for taking care of this according to the description of TEZ-1742: after checking and trying out the unit test and the fix here, it seems like this patch what's exactly fixes the abovementioned problem apart from the assertion error fix, the unit test indeed proves that releaseAssignedContainer is called as many times as needed according to the percentage config in a single round, so this looks good to me only left a minor comment regarding a null check |
|
Thanks @simhadri-g , @abstractdog for review. I have incorporated feedback as following:
Also adding a note that as the the changes fixes slow preemption, it leads to performance gain mentioned in TEZ-1742 when container re-use is enabled, which is enabled by default in Tez. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
@himanshu-mishra : minor checkstyle left, can you please address: |
|
🎊 +1 overall
This message was automatically generated. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When container reuse is enabled, preemption of lower priority containers that are not yet assigned to task, takes long time as they are released one at a time, and not the number of containers based when tez.am.preemption.percentage is high added in https://issues.apache.org/jira/browse/TEZ-1742.
Further investigation lead to following conclusion:
Warn log / Assertion error thrown because in preemptIfNeeded(), when releasing new containers, the loop counter is being decremented with each
releaseUnassignedContainers, leading to looping only half number of times. By using another counter, assertion passes because of condition method returns with checkif (numPendingRequestsToService < 1) {.In releaseContainer(), the container is not getting removed from
delayedContainersqueue and only fromheldContainersmap, hence same container is being picked up for release in every iteration till next cycle ofDelayedContainerManagerfinds out that the container is not inheldContainersand skips it with logSkipping delayed container as container is no longer running, containerId=...This change adds a method in
DelayedContainerManagerto allow removal of delayed container and invokes it in releaseContainer method, which so far only removed it fromheldContainersmap.