Clarify evaluation of check values reported concurrently from multiple sources

[fjeldstad]

Code of Conduct

• I have read and agree to the GitHub Docs project's Code of Conduct

What article on docs.github.com is affected?

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/collaborating-on-repositories-with-code-quality-features/about-status-checks

What part(s) of the article would you like to see updated?

I think there is a need for a new section that explains how multiple sources of values for a given check are evaluated. In particular, it is very important for users to understand that the branch protection feature will not wait for all running workflows to complete execution before setting the outcome of a specific check.

If I would write the section myself, it could be something along these lines:

Evaluation of concurrently reported check values

It is possible for multiple sources to report values for the same check. A common example would be GitHub Actions workflows with a job named corresponding to a required check.
For example, let's say a branch has a required check called test. If there are multiple workflows each with a job named test, they will all eventually contribute to the
final value of the check. If one of them fails, the check will be marked as failed, as you would expect. But the check engine will not wait for all workflows to finish before
evaluating the check value - it will aggregate values as they are reported; a check might be first marked as successful by a quick workflow just to later be changed
to failed by a slower workflow. In other words, there can be a time window when a pull request is able to merge even if it would be prevented from doing so by other pending workflows.
This behavior is especially important to be aware of if using the auto-merge feature as pull requests will be merged as soon as the first successful check value is reported.

Additional information

No response

[welcome]

Thanks for opening this issue. A GitHub docs team member should be by to give feedback soon. In the meantime, please check out the contributing guidelines.

[nguyenalex836]

@fjeldstad Thank you for raising this issue! I'll get this triaged for review ✨ Our team will provide feedback regarding the best next steps for this issue - thanks for your patience! 💛

[github-actions]

Thanks for opening an issue! We've triaged this issue for technical review by a subject matter expert 👀

[nguyenalex836]

@fjeldstad Thank you for your patience while our SME team reviewed this issue! 💛 Our SME team observed the following -

I tried to set up a repro for this and wasn't able to observe the same behavior, but maybe I'm misunderstanding their setup. My setup was:

• Two required workflows, one that succeeds and another that fails after a pause. They both have a single job with the same name
• Open a PR that triggers the required workflows
• Observe the commit status

My expectation based on their claim: The commit status goes green when the successful workflow finishes, then turns red after the failing workflow finishes
My observation: The commit status remains yellow (in progress) until all the checks are done, then turns red

I also tried adding an approval gate to the failing workflow to get it to wait indefinitely, but I still observed the expected behavior

As we are unable to reproduce this issue, can you reach out to our excellent support team? They are best equipped to reproduce the error you are seeing, which we will need to do before we make changes to the docs 💛

If after discussing with our support team there should be changes made to the docs, please let us know! ✨

[fjeldstad]

The repro sounds well thought-through, but did you also try enabling auto-merge? In my scenario, the PR is merged immediately when the check turns green (when the first workflow finishes). It does not wait for the slower workflow (but the status is eventually reported as red). tors 26 sep. 2024 kl. 18:54 skrev Alex Nguyen ***@***.***>:

…

@fjeldstad <https://github.com/fjeldstad> Thank you for your patience while our SME team reviewed this issue! 💛 Our SME team observed the following - I tried to set up a repro for this and wasn't able to observe the same behavior, but maybe I'm misunderstanding their setup. My setup was: - Two required workflows, one that succeeds and another that fails after a pause. They both have a single job with the same name - Open a PR that triggers the required workflows - Observe the commit status *My expectation based on their claim:* The commit status goes green when the successful workflow finishes, then turns red after the failing workflow finishes *My observation:* The commit status remains yellow (in progress) until all the checks are done, then turns red I also tried adding an approval gate to the failing workflow to get it to wait indefinitely, but I still observed the expected behavior As we are unable to reproduce this issue, can you reach out to our excellent support team <https://support.github.com/request>? They are best equipped to reproduce the error you are seeing, which we will need to do before we make changes to the docs 💛 If after discussing with our support team there should be changes made to the docs, please let us know! ✨ — Reply to this email directly, view it on GitHub <#34568 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG5EA4Y2ZODMA7DLQU2SRDZYQ36BAVCNFSM6AAAAABODRM7OGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZXGQ3TGNZWHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[nguyenalex836]

@fjeldstad Thank you for letting us know! Would by chance be able to point to a public repo that exhibits this behavior (that we can use for testing)? If there's a setup you've personally put together to test this, we're happy to utilize that as well! 💛

[fjeldstad]

I’m sorry but I don’t have that available, this was found in a private repo. It sounds like the setup you tested with should be the same, it’s just auto-merge that is missing. tors 26 sep. 2024 kl. 23:30 skrev Alex Nguyen ***@***.***>:

…

@fjeldstad <https://github.com/fjeldstad> Thank you for letting us know! Would by chance be able to point to a public repo that exhibits this behavior (that we can use for testing)? If there's a setup you've personally put together to test this, we're happy to utilize that as well! 💛 — Reply to this email directly, view it on GitHub <#34568 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG5EAZB3OKR5XOZFE6K6DLZYR4H3AVCNFSM6AAAAABODRM7OGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZXHE3TGNJYG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[nguyenalex836]

@fjeldstad That's no problem, thank you for your patience while our team reviewed further!

While we are currently undergoing a repo freeze this week, internally we will be adding a note to the doc once the freeze is over along the lines of:

... to have uniquely named job names across multiple workflows if you’re using branch protections. If there is a conflict there is no waiting of any sort and it is ambiguous

We decided this would be the best course of action after additional discussion 💛 Thank you for your willingness to contribute to our docs, and raising a flag on this issue! 🙇