[hydra-github-bridge] rare race condition from queue that results in crash

[dermetfan]

https://input-output-rnd.slack.com/archives/CG1FBSDMM/p1705942473824639

It seems a notification wasn't delivered to github:

• hydra: https://ci.iog.io/jobset/IntersectMBO-ouroboros-network/pullrequest-4784
• github: https://github.com/IntersectMBO/ouroboros-network/runs/20731619501

Jan 22 14:47:12 ci1 hydra-github-bridge-intersect-start[249727]: Build Started (2523945): IntersectMBO-ouroboros-network:merge-queue-4784 aarch64-darwin.required(All jobs required to pass CI) Just ("IntersectMBO","ouroboros-network","bbc93eeef6bd166d133b7d42f025775274aea89>
Jan 22 14:47:12 ci1 hydra-github-bridge-intersect-start[249727]: CheckRun {owner = "IntersectMBO", repo = "ouroboros-network", payload = CheckRunPayload {name = "ci/hydra-build:aarch64-darwin.required", headSha = "bbc93eeef6bd166d133b7d42f025775274aea896", detailsUrl = Jus>
Jan 22 14:47:12 ci1 hydra-github-bridge-intersect-start[249727]: -> {"owner":"IntersectMBO","payload":{"completedAt":null,"conclusion":null,"detailsUrl":"https://ci.iog.io/build/2523945","externalId":"2523945","headSha":"bbc93eeef6bd166d133b7d42f025775274aea896","name":"ci>
Jan 22 14:47:12 ci1 hydra-github-bridge-intersect-start[249727]: statusHandler:AsyncCancelled
Jan 22 14:47:13 ci1 hydra-github-bridge-intersect-start[249727]: hydra-github-bridge: /var/lib/hydra/hydra-github-bridge/intersect/queue-2024-01-22+14:47:12.84253527.store: removeLink: does not exist (No such file or directory)
Jan 22 14:47:13 ci1 systemd[1]: hydra-github-bridge-intersect.service: Main process exited, code=exited, status=1/FAILURE
Jan 22 14:47:13 ci1 systemd[1]: hydra-github-bridge-intersect.service: Failed with result 'exit-code'.
Jan 22 14:47:13 ci1 systemd[1]: hydra-github-bridge-intersect.service: Consumed 1min 56.102s CPU time, received 43.3M IP traffic, sent 34.7M IP traffic.
Jan 22 14:47:23 ci1 systemd[1]: hydra-github-bridge-intersect.service: Scheduled restart job, restart counter is at 16.
Jan 22 14:47:23 ci1 systemd[1]: Started hydra-github-bridge-intersect.service.

[dermetfan]

Again: https://github.com/IntersectMBO/ouroboros-network/runs/20741619434

Jan 22 18:51:31 ci1 hydra-github-bridge-intersect-start[181714]: Build Finished (2526294): IntersectMBO-ouroboros-network:merge-queue-4784 x86_64-darwin.required(All jobs required to pass CI)>
Jan 22 18:51:31 ci1 hydra-github-bridge-intersect-start[181714]: CheckRun {owner = "IntersectMBO", repo = "ouroboros-network", payload = CheckRunPayload {name = "ci/hydra-build:x86_64-darwin.>
Jan 22 18:51:31 ci1 hydra-github-bridge-intersect-start[181714]: -> {"owner":"IntersectMBO","payload":{"completedAt":"2024-01-22T18:51:29Z","conclusion":"success","detailsUrl":"https://ci.iog>
Jan 22 18:51:31 ci1 hydra-github-bridge-intersect-start[181714]: statusHandler:AsyncCancelled
Jan 22 18:51:31 ci1 hydra-github-bridge-intersect-start[181714]: hydra-github-bridge: /var/lib/hydra/hydra-github-bridge/intersect/queue-2024-01-22+18:51:31.542954805.store: removeLink: does not exist (No such file or directory)
Jan 22 18:51:31 ci1 systemd[1]: hydra-github-bridge-intersect.service: Main process exited, code=exited, status=1/FAILURE
Jan 22 18:51:31 ci1 systemd[1]: hydra-github-bridge-intersect.service: Failed with result 'exit-code'.
Jan 22 18:51:31 ci1 systemd[1]: hydra-github-bridge-intersect.service: Consumed 13.856s CPU time, received 2.0M IP traffic, sent 648.2K IP traffic.
Jan 22 18:51:41 ci1 systemd[1]: hydra-github-bridge-intersect.service: Scheduled restart job, restart counter is at 17.
Jan 22 18:51:41 ci1 systemd[1]: Started hydra-github-bridge-intersect.service.