-
Notifications
You must be signed in to change notification settings - Fork 815
Tests: add helper Stress
attribute for stress testing
#18452
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
✅ No release notes required |
tests/FSharp.Compiler.ComponentTests/CompilerService/AsyncMemoize.fs
Outdated
Show resolved
Hide resolved
Good stuff. I think VS allows for "run until failure" but not sure if it stressing the thread pool this much, and also it's not x-plat. Would you mind adding a line about that to the testing guide? :) |
…ize.fs Co-authored-by: Petr <psfinaki@users.noreply.github.com>
The difference is VS "run until failure" will execute sequentially if we select only one test case. For example it couldn't reproduce the failure I linked above, while with Hmm, now I think the naming could be better. |
I'm not sure if this should to TESTGUIDE or DEVGUIDE? There was a section already to which I added. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is good enough for now :) Thanks!
Speaking of naming, I think "Stress" would be even better than "Repeat". |
Repeat
attribute for stress testingStress
attribute for stress testing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like this a lot, it looks super useful.
Can you add a test case where this attribute is used and that we verify it passes when al iterations succeed, and fails when one pass doesn't succeed.
Makes sense! But now I'm in decision paralysis where to put such test 😄. This will be a test of test utilities, so it doesn't belong anywhere... |
Well, we can always start a testutilities test assembly. With this as the first test, hereafter, we will have no excuse for not testing the hacks we add to the test framework. |
I'm guessing many of the smoketests are only smoketests because they are testing that the test framework does something interesting. |
Would it be ok to just unlock existing FSharp.Test.Utilities project as a test container? |
I would do it like that as well, keep the tests close to those utilities. |
Tests added :) |
This is useful for locally stress testing cases that are flaky in parallel execution.
Example usage: replace
[<Fact>]
with[<Theory; Stress(Count = 1000)>]
Unlike VS "Run until failure" feature which runs sequentially, this will start the test simultaneously many times and run in parallel, stressing the threadpool.
It can be used to locally track down bugs in flaky tests.