fix(engine): circuit breaker for deterministic failure cycles in main loop #2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…main loop The engine already had failure signature tracking for loop_restart edges, but the main runLoop had no equivalent protection. When a provider auth token expires mid-run, every stage fails with failure_class=deterministic. The per-node retry system (executeWithRetry) correctly blocks same-node retries, but conditional edge routing (e.g., check -> implement on outcome=fail) still follows fail edges, creating an infinite cycle of 1-2 second failures that never terminates. This adds loopFailureSignatures tracking to runLoop. After each stage, if the outcome is a deterministic failure, the failure signature is recorded. If the same signature repeats N times (default 3, configurable via loop_restart_signature_limit), the run aborts with a clear error. The counter resets on any successful stage. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
danshapiro
approved these changes
Feb 9, 2026
Owner
danshapiro
left a comment
danshapiro
left a comment
There was a problem hiding this comment.
Thorough, well-tested fix that follows established codebase patterns. Clean implementation of the circuit breaker for deterministic failure cycles. Merging now with two optional follow-up items to handle in a subsequent commit.
danshapiro
pushed a commit
that referenced
this pull request
Feb 9, 2026
… correctness The circuit breaker for deterministic failure cycles (PR #2) tracked failure signatures in-memory but did not persist them to checkpoint.json. This meant a resumed run would restart the counter at zero, potentially allowing 2×limit failures before the breaker tripped. Changes: - checkpoint() now serializes loopFailureSignatures to cp.Extra["loop_failure_signatures"] alongside the existing restart_failure_signatures - resume.go restores loopFailureSignatures via restoreLoopFailureSignatures(), mirroring the existing restoreRestartFailureSignatures() pattern - Added explanatory comment in loopRestart() documenting the intentional non-reset of loopFailureSignatures across loop restarts (if the same deterministic failure persists after a restart, the counter should keep accumulating) - Added tests for round-trip serialization including populated maps, nil checkpoints, missing keys, and empty-key filtering Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
loopFailureSignaturestracking torunLoop()to detect and abort infinite cycles caused by repeated deterministic failures (e.g., expired provider auth tokens)loop_restart_signature_limitgraph attribute (default 3) as the thresholdFixes #1
Details
The engine already had failure signature tracking for
loop_restartedges (restartFailureSignatures), but the mainrunLoophad no equivalent protection. When every stage fails withfailure_class=deterministic, conditional edge routing (e.g.,check -> implement [condition="outcome=fail"]) keeps cycling forever.This mirrors the existing
loopRestartcircuit breaker but applies it to the main loop.Test plan
TestRun_DeterministicFailureCycle_AbortsInfiniteLoop— graph with a conditional retry cycle where all nodes exit 1; verifies the run aborts after 3 repeated signaturesTestRun_DeterministicFailure_SingleRouteToRecovery_StillWorks— single deterministic failure routes to a recovery node and completes successfully (no false positive)TestWaitWithIdleWatchdog_ContextCancelKillsProcessGroup)🤖 Generated with Claude Code