The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1274356	maestro-auth-test/helix-queue-insights-test	Run Tests (Windows)	Log
1274336	dotnet/aspnetcore	Run build.cmd helix target	Log
1274333	dotnet/aspnetcore	Run build.cmd helix target	Log
1274334	dotnet/aspnetcore	Run build.cmd helix target	Log
1274082	dotnet/runtime	Send to Helix	Log
1266961	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#65226
1266936	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#65224
1274269	dotnet/roslyn	Test	Log
1266963	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#65226
1266937	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#65224
1274264	dotnet/machinelearning	Run Helix Tests	Log	dotnet/machinelearning#7575
1274111	dotnet/roslyn	Test	Log	dotnet/roslyn#82230
2891701	dotnet-dotnet	🛡 Guardian: ARMory	Log
2891580	dotnet-dotnet	🛡 Generate SBoM Manifest	Log
1273952	dotnet/runtime	Build product	Log	dotnet/runtime#123810
1273949	dotnet/machinelearning	Run Helix Tests	Log	dotnet/machinelearning#7573
1273806	dotnet/runtime	Send to Helix	Log	dotnet/runtime#123779
1273832	dotnet/maui	DeviceTestsMacCatalyst_CoreCLR (Windows)	Log
1273860	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#123142
2891579	dotnet-sdk	SDL Binary Analysis Tools Pre-Checks (1ES PT)	Log	#57370
2891582	dotnet-dotnet	Secure Supply Chain Analysis (auto-injected by policy)	Log	#57371
2891578	dotnet-sdk	Build	Log	#57370
1273781	dotnet/runtime	Send to Helix	Log	dotnet/runtime#123775
1273830	dotnet/maui	Run Integration Tests - Build windows	Log
1273825	dotnet/maui	Run Integration Tests - Build windows	Log	dotnet/maui#33618
1273802	dotnet/maui	Run Integration Tests - Build windows	Log	dotnet/maui#33799
1272799	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#123774
2891523	dotnet-dotnet	🛡 Generate SBoM Manifest	Log
1273773	dotnet/runtime	Send to Helix	Log	dotnet/runtime#123342
2891511	dotnet-dotnet	Build (azurelinuxArm64CrossContainer)	Log
1273679	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#123044
1273661	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#122638
2891488	dotnet-dotnet	Checkout dotnet-dotnet@release/11.0.1xx-preview1 to s	Log
1273644	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#123726
1273708	dotnet/aspnetcore	Run Quarantined Tests	Log	dotnet/aspnetcore#65263
1273684	dotnet/maui	Run Integration Tests - Build windows	Log	dotnet/maui#33234
2891445	dotnet-dotnet	Run Scancode (azurelinuxSourceBuildTestContainer)	Log
1272781	dotnet/runtime	Send to Helix	Log	dotnet/runtime#123273
2891439	dotnet-dotnet	Run Scancode (azurelinuxSourceBuildTestContainer)	Log
1273598	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#123753
2891430	dotnet-dotnet	🛡 Guardian: AntiMalware Scanner (Source)	Log
1273611	dotnet/runtime	Send to Helix	Log	dotnet/runtime#123800
1273581	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#123033
1273497	dotnet/maui	Run Integration Tests - Build windows	Log	dotnet/maui#29101
1273574	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#123251
2891422	dotnet-dotnet	🛡 Guardian: AntiMalware Scanner (Source)	Log
1273557	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#123794
2891407	dotnet-dotnet	🛡 Guardian: AntiMalware Scanner (Source)	Log
2891399	dotnet-dotnet	🛡 Guardian: AntiMalware Scanner (Source)	Log
1270002	dotnet/runtime	Send to Helix	Log	dotnet/runtime#123386
2891396	dotnet-dotnet	🛡 Guardian: AntiMalware Scanner (Source)	Log
1273556	dotnet/maui	Run Integration Tests - Build windows	Log	dotnet/maui#33792
1273306	dotnet/runtime	Send to Helix	Log
1273485	dotnet/aspnetcore	Run build.cmd helix target	Log
2891332	dotnet-aspnetcore	Run build.cmd helix target	Log
2891331	dotnet-aspnetcore	Run build.cmd helix target	Log
1273379	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#123791
1273373	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#123500
2891313	dotnet-arcade-services	Publish Using Darc	Log
1273427	dotnet/runtime	Build	Log	dotnet/runtime#123142
1273339	dotnet/runtime	Run Native AOT Test App Tests	Log	dotnet/runtime#123753
1273355	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#122970
1273356	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#123742
1273399	dotnet/maui	Run Integration Tests - Build windows	Log	dotnet/maui#33724
1273386	dotnet/maui	Run Integration Tests - Build macOS	Log	dotnet/maui#33567
1273389	dotnet/runtime	Build	Log	dotnet/runtime#123792
1273400	dotnet/maui	Run Integration Tests - Build windows	Log	dotnet/maui#33789
1273340	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#123483
1273368	dotnet/maui	Run Integration Tests - Build macOS	Log	dotnet/maui#33531
1273308	dotnet/runtime	Send tests to Helix (Unix)	Log
1273358	dotnet/runtime	Build	Log	dotnet/runtime#122970
1273343	dotnet/maui	Run Integration Tests - Build macOS	Log	dotnet/maui#33469
1273307	dotnet/runtime	Send tests to Helix (Windows)	Log
1273334	dotnet/runtime	Send to Helix	Log	dotnet/runtime#123787
1273320	dotnet/runtime	Send to Helix	Log	dotnet/runtime#123719
1261175	dotnet/aspnetcore	Run E2E tests	Log	dotnet/aspnetcore#64964
1273275	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#123433
1273267	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#121007
1273299	dotnet/aspnetcore	Run build.cmd helix target	Log
1273305	dotnet/aspnetcore	Run build.cmd helix target	Log
2891194	dotnet-dotnet	Build (centOSStreamContainer)	Log
1273266	dotnet/runtime	Send tests to Helix (Windows)	Log
1273199	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122722
1273250	dotnet/maui	Controls Layout	Log	dotnet/maui#33733
2891160	dotnet-runtime	Send job to Helix (Unix)	Log
1273220	dotnet/aspnetcore	Run build.cmd helix target	Log
1273206	dotnet/runtime	Send to Helix	Log	dotnet/runtime#119891
1273204	dotnet/runtime	Send to Helix	Log	dotnet/runtime#123601
1273175	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#122722
1273180	dotnet/maui	Run Integration Tests - Build macOS	Log	dotnet/maui#33784
1273191	dotnet/runtime	Send to Helix	Log	dotnet/runtime#123787
1272660	dotnet/runtime	Send tests to Helix (Unix)	Log
2891082	dotnet-dotnet-monitor	Build	Log
1273117	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#123588
1273108	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#123781
1272710	dotnet/runtime	Send to Helix	Log	dotnet/runtime#123764
2891040	dotnet-dotnet-monitor	Build	Log
1272667	dotnet/maui	Controls Layout	Log
1273098	dotnet/runtime	Send to Helix	Log	dotnet/runtime#123783
1273021	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#123753
Displaying 100 of 2245 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
95	1140	2245

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)

[CarnaViire]

For the "known issues" folks: the timeout here is reported twice, making BA red despite the tracking issue. E.g.:

1️⃣ dotnet/runtime#117255

2️⃣ dotnet/runtime#117800

cc @AlitzelMendez