Parallelize tests #17872
Parallelize tests #17872
Conversation
|
majocha
force-pushed
the
parallel-tests
branch
6 times, most recently
from
decc8cb
to
278e2ec
Compare
|
Thanks for your endurance, Jakub 💪 |
|
@psfinaki I will need some help with this Source-Build error: At this moment this is very stable locally but will also probably need testing on other machines than mine :) What's left to do is to tune this for stability in CI. I've been trying different things and timing runs. The most glaring problem is the I also noticed Linux run is constantly low on memory, this is unrelated as it happens on main, too. For this reason I set |
|
@majocha the error is weird, nothing comes to my mind right away. Let's rebase and rerun and see if it's still happening... Sorry, I know this is somewhat lame, it's just that SourceBuild is a Linux thing and it's not trivial to debug its issues locally. As for cooling things down - I also noticed this today, thanks for addressing this. What else do you think we can split from this PR into some separate ones? |
There are some small further fixes, maybe also the whole console handling does not really depend on parallel execution. Somewhat related thing I have on my mind recently is to implement a |
majocha
force-pushed
the
parallel-tests
branch
from
1c17ac7
to
0ed0442
Compare
Thanks! Rebasing did help. |
psfinaki
added
the
NO_RELEASE_NOTES
Yeah console handling would be probably good to isolate if possible.
Just for my understanding, what would this add on top of the current results the PR achieves? |
This would be an experiment for another PR, but basically, I don't like all that copying to temp dirs that I added in recent PRs. |
|
Right, yeah, I see. No it's worth playing with, although given that we don't touch these tests too much, it's probably worth seriously investing into only if it starts yielding reasonable performance fruits. |
| @@ -252,6 +252,7 @@ let processGraphAsync<'Item, 'Result when 'Item: equality and 'Item: comparison> | |||
| let rec queueNode node = | |||
| Async.Start( | |||
| async { | |||
| use! _catch = Async.OnCancel(completionSignal.TrySetCanceled >> ignore) | |||
There was a problem hiding this comment.
Fix. This was starting with a cancellation token but without catching the OCE.
There was a problem hiding this comment.
This is more complicated and Async.OnCancel of course does not catch OCE. What was I thinking?
tests/FSharp.Compiler.Private.Scripting.UnitTests/FSharpScriptTests.fs
Outdated
Show resolved
Hide resolved
| // Build command line arguments & start FSI session | ||
| let argv = [| "C:\\fsi.exe" |] | ||
| let allArgs = Array.append argv [|"--noninteractive"; if useOneDynamicAssembly then "--multiemit-" else "--multiemit+" |] | ||
|
|
||
| let fsiConfig = FsiEvaluationSession.GetDefaultConfiguration() | ||
| FsiEvaluationSession.Create(fsiConfig, allArgs, inStream, new StreamWriter(outStream), new StreamWriter(errStream), collectible = true) | ||
| FsiEvaluationSession.Create(fsiConfig, allArgs, TextReader.Null, TextWriter.Null, TextWriter.Null, collectible = true) |
There was a problem hiding this comment.
There are actually no tests using these streams.
There was a problem hiding this comment.
But they are useful for capturing test errors. This should be stdout / stderr.
|
https://dev.azure.com/dnceng-public/public/_build/results?buildId=863457&view=logs&j=7bab896a-24f8-544f-51eb-43745367a332&t=999dbed9-85e3-59ab-57f0-3e22828b5bad&l=3823 I've seen this test hang also locally, but it takes like few hundreds of iterations to trigger the deadlock (?). |
|
TODO, two test cases have poorer stability now: |
| @@ -137,7 +137,7 @@ module internal PervasiveAutoOpens = | |||
| type Async with | |||
|
|
|||
| static member RunImmediate(computation: Async<'T>, ?cancellationToken) = | |||
| let cancellationToken = defaultArg cancellationToken Async.DefaultCancellationToken | |||
| let cancellationToken = defaultArg cancellationToken CancellationToken.None | |||
There was a problem hiding this comment.
Maybe it's worth having cancellation token changes separately as well, for a more attentive review.
| @@ -8,6 +8,8 @@ open FSharp.Test.Compiler | |||
| open System | |||
| open System.IO | |||
|
|
|||
| // reportTime uses global state. | |||
There was a problem hiding this comment.
I like these explanations, good stuff, thanks.
| @@ -14,5 +14,5 @@ let success = | |||
| asm.Version.Major = 1 && | |||
| asm.Version.Minor = 2 && | |||
| asm.Version.Build = 3 && | |||
| (abs (asm.Version.Revision - (int defaultRevision))) < 10 // default value is seconds in the current day / 2. Check if within 10 sec of that. | |||
| (abs (asm.Version.Revision - (int defaultRevision))) < 60 // default value is seconds in the current day / 2. Check if within 60 sec of that. | |||
There was a problem hiding this comment.
This is flaky otherwise, right? Wonder if we can come up with a timeout-independent fix.
| @@ -93,14 +93,17 @@ let ``comprehensions-FSC_OPTIMIZED`` () = singleTestBuildAndRun "core/comprehens | |||
| [<Fact>] | |||
| let ``comprehensions-FSI`` () = singleTestBuildAndRun "core/comprehensions" FSI | |||
|
|
|||
| [<Fact>] | |||
| let ``comprehensionshw-FSC_DEBUG`` () = singleTestBuildAndRun "core/comprehensions-hw" FSC_DEBUG | |||
| // Cancels default token. | |||
There was a problem hiding this comment.
For my understanding, do we have a reasonable explanation on why it's happening? (default token cancellation)
There was a problem hiding this comment.
Yes, just some test cases call it directly.
| @@ -479,6 +458,9 @@ let x = | |||
| script.Eval(code) |> ignoreValue | |||
| Assert.False(foundInner) | |||
|
|
|||
| // Fails in NETFRAMEWORK with exception | |||
There was a problem hiding this comment.
Does it mean it wasn't failing before?
There was a problem hiding this comment.
Ah, no, iirc this whole test project was excluded on net472 and I re-enabled it for some reason.
There was a problem hiding this comment.
Nope I rememebered wrong, I need to revisit this.
| let getTempPath dir = | ||
| Path.Combine(TestFramework.tempDirectoryOfThisTestRun, dir) | ||
| let getTempPath dir = | ||
| Path.Combine(tempDirectoryOfThisTestRun.Value.FullName, dir) |
There was a problem hiding this comment.
Not a fan of forcing options, but not such a big deal in tests.
There was a problem hiding this comment.
Ah, this is a lazy, not an option. I tried to solve the temp dir cleanup somehow in a centralized manner and this is part of the solution.
| @@ -14,21 +14,15 @@ type Sentinel () = | |||
| module MyModule = | |||
| let test(x: int) = () | |||
|
|
|||
| [<CollectionDefinition("FsiTests", DisableParallelization = true)>] | |||
There was a problem hiding this comment.
Do I get it right that these are working even without applying the new xUnit matters?
There was a problem hiding this comment.
Yes, collection definition is pure xUnit.
|
😞 this is new. https://dev.azure.com/dnceng-public/public/_build/results?buildId=867536&view=logs&j=2363a781-3528-55f4-1450-adabad4eb250&t=4ee64dfd-475b-5640-3e58-0c1d0f6688da&l=20 |
|
/azp run |
|
Azure Pipelines successfully started running 2 pipeline(s). |
|
Most of the Utilities refactoring and the parallel console support is extracted to #17993. This excludes the xUnit customizations and parallelization of the tests. I hope to make sure this still works and tests fine with the state of the things we have in main. |
|
Awesome, thanks for splitting that - we will take a look there shortly :) |
|
I don't know what's up with the _GetRestoreSettingsPerFramework error, though. |
Enable running xUnit tests in parallel.
To use xUnit means to customize it. Two optional features added:
Running collection and theory cases in parallel based on https://www.meziantou.net/parallelize-test-cases-execution-in-xunit.htm
By default xUnit's unit of parallelization is test collection. Test cases in a collection run in sequence. Also, by default each class/module constitutes a collection. We have a lot of test cases in large modules and large theories that were bottlenecked by this.
This customization enables parallelism in such cases. It can be reverted to default for a particular module with
[<RunInSequence>]attribute.Relevant issue is on the roadmap for xUnit v3, so this will probably become unnecessary in the future.
Console streams captured universally and redirected to xUnit's output mechanism, which means you can just do printfn in a test case and it goes to the respective output.
This can be inspected in the IDE and in case of failure is printed out when testing from the command line.
The default way in xUnit is to use ITestOutputHelper. This is very unwieldy, because it requires placing test cases in a class with a constructor, and then threading the injected output helper into any function that wants to output text. We have many tests in modules not classes, and many of the tests are using a lot of utility functions. Adjusting it all to use ITestOutputHelper is not feasible. OTOH just outputting with
printfnis unobtrusive, natural and works well with interactive prototyping of test cases.This customization will probably become unnecessary with xUnit v3.
The above customizations are not required for the test suite to work correctly. They are contained to the XunitHelpers.fs file and enabled with conditional compilation using
XUNIT_EXTRASconstant defined in FSharp.Test.Utilities.fsprojSome local run times:
Test summary: total: 4489, failed: 0, succeeded: 4258, skipped: 231, duration: 199.0sTest summary: total: 579, failed: 0, succeeded: 579, skipped: 0, duration: 41.9sTest summary: total: 12963, failed: 0, succeeded: 12694, skipped: 269, duration: 253.3sSome considerations to make this work and keep it working
To run tests in parallel we must deal with global resources and global state accessed by the test cases.
Out of proc:
Tests running as separate processes are sharing the file system. We must make sure they execute in their own temporary directories and don't overwrite any hardcoded paths. This is already done, mostly in separate PR.
Hosted:
Many tests use hosted compiler and
FsiEvaluationSession, sharing global resources and global state within the runner process:FileSystemglobal mutable of the file system shim - few tests that mutate it, must be excluded from parallelization.Environment.CurrentDirectory- many tests executing in hosted session were doing a variation ofFile.WriteAllText("test.ok", "ok")all in the current directory i.e.bin, leading to conflicts. This is replaced with a threadsafe mechanism.DependencyManager, excluded from parallelization for now.Asyncdefault cancellation token - few tests doingAsync.CancelDefaultToken()must be excluded from parallelization.--timesoption - tests excluded from parallelization.ConcurrentDictionary. This seems no longer a problem, contained using some exclusions from parallelization.I'll ad to the above list if I recall anything else.
Problems:
Tests depending on tight timing, orchestrating stuff by combinations of
Thread.Sleep,Async.Sleepand wait timeouts.These are mostly excluded from parallelization, some attempts at fixing things were made.
Obscure compiler bugs revealed in this PR:
Internal error: value cannot be nullthis mostly happens in coreClr, one time, sometimes a few times during the test run.Error creating evaluation sessionbecause of NRE somewhere inTcImports.BuildNonFrameworkTcImports. This is more rare but may be related to the above.These were related to some concurrency issues; modyfing
frameworkTcImportsCachewithout lock and a bug in custom lazy implementation in il.fs. Hopefully both fixed now.Running in parallel:
Xunit runners are configured with mostly default parallelization settings.
dotnet test .\FSharp.sln -c Release -f net9.0will run all discovered test assemblies in parallel as soon as they're built.This can be limited with the
-mswitch. For example,dotnet test -m:2 .\FSharp.Compiler.Service.slnwill limit the test run to at most 2 simultaneous processes. Still, each test host process runs its test collections in parallel.
Some test collections are excluded form parallelization with
[<Collection(nameof DoNotRunInParallel)>]attribute.Running in the IDE with "Run tests in parallel" enabled will respect
xunit.runner.jsonsettings and the above exclusions.TODO:
BUILDING_USING_DOTNETscenario (Attempt to make FCS solution build without arcade and with the SDK specified in global.json #14677)