Add robust trial status polling (#4750) #4756
Closed
+153
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@bernardbeckerman has exported this pull request. If you are a Meta employee, you can view the originating Diff in D90532278. |
meta-cla
bot
added
the
CLA Signed
Summary: Adds a robust poll_trial_status method to Orchestrator that: 1. If runner.poll_trial_status fails, falls back to polling trials one-at-a-time. 2. Each trial for which individual status polling fails is marked as abandoned. This is important for runners whose status-polling logic can undergo sporadic exceptions, in order to avoid failing the entire orchestrator process. Alternatives considered: * **Institute a PollStatusResult analogous to [MetricFetchResult](https://fburl.com/code/yyp9j66m).** Decided against because I don't think this logic belongs in runner.poll_trial_status, since it's good to have this centralized instead of replicated across all runners' poll_trial_status methods. Also this would require migration across all runner.poll_trial_status methods. * **Add a new Runner method robust_poll_trial_status (D90390842)**. It could make sense to keep retry logic closer to where polling happens, but realistically it's only used here. Also considered making this configurable, e.g., via a boolean flag `OrchestratorOptions.should_poll_status_robustly`, but opted for an opinionated solution that avoids config bloat. Differential Revision: D90532278
bernardbeckerman
force-pushed
the
export-D90532278
branch
from
6d24ce2 to
8ae3ef3
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4756 +/- ##
=======================================
Coverage 96.72% 96.72%
=======================================
Files 583 583
Lines 60860 60914 +54
=======================================
+ Hits 58866 58921 +55
+ Misses 1994 1993 -1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
This pull request has been merged in f575efe. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Adds a robust wrapper method for poll_trial_status that:
This is important for runners whose status-polling logic can undergo sporadic exceptions, in order to avoid failing the entire orchestrator process.
Alternatives considered:
Also considered making this configurable, but opted for an opinionated solution that avoids config bloat.
Differential Revision: D90532278