Setting a predefined maximum time where in a test should finish

[Aliazzzz]

Hi,

I had this idea: it can be very beneficial to specify a predefined maximum time to where in a test should finish.
The normal working of the assert is adhered.
However, if an assert is not finished with the predefined time-out time, the assert fails, provides a time-out message in the log and resumes to the next test.
This way, indefinite loops are gracefully broken.

Hope this idea is considered.

[sagatowski]

I like the idea. JUnit has something similar:
https://howtodoinjava.com/junit/how-to-force-timeout-in-jnuit-testcase-execution/

[sagatowski]

@Aliazzzz I've assigned this task to you. You're most welcome to implement it.

[I-Campbell]

@Aliazzzz Do you see this as:

1. At the start of every task cycle check if the timeout has expired for this test case. Yes=> don't run. Set test case status to aborted. No=> Run next cycle of the test.
   OR
2. If one cycle of the test case runs for too long, kill it.
   OR
3. Continually Monitor the total runtime of the test case. If the time expires while the test case is executing, kill it, otherwise just don't call it again. Either way set test case status to aborted.

[Aliazzzz]

I regard it as 3)
Continually Monitor the total runtime of the test case. If the time expires while the test case is executing, kill it, otherwise just don't call it again. Either way set test case status to aborted.

[I-Campbell]

Well, that's the hardest one. But at least the issue is better defined now.

[sagatowski]

@Aliazzzz I don't see any test case status of "aborted" in the standard.
https://llg.cubic.org/docs/junit/

I only see failed, skipped, passed, error. We need to be careful and not implement our own standard.

Edit: Also, is it possible to see how others have done it? (JUnit/Google Test)?

[RogerChristopher]

I don't understand the need for a timeout. A unit test that can hang must have an external dependancy. Generally such a dependancy (FileOpen, for example) will have a timout associated with it. If it times out, the test fails. In any case, the timeout should be part of the test implementation, not the framework, as it is test-specific. Most tests will not need it.

…

On Sun, 31 May 2020, 19:35 Jakob Sagatowski, ***@***.***> wrote: @Aliazzzz <https://github.com/Aliazzzz> I don't see any test case status of "aborted" in the standard. https://llg.cubic.org/docs/junit/ I only see failed, skipped, passed, error. We need to be careful and not implement our own standard. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/tcunit/TcUnit/issues/54#issuecomment-636510315>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH3VKDV7W72KFZMV64GIQNTRUKPPNANCNFSM4IWXXXUA> .

[PTKu]

The idea may have come from some existing testing frameworks like: https://docs.nunit.org/articles/nunit/writing-tests/attributes/timeout.html

[I-Campbell]

Maybe the code you are testing accidentally introduced an infinite loop, and you want to know the rest of the unit testing results? So I could say "if any test runs for 10 seconds, kill it and report as error."

[Aliazzzz]

Agreed, we should stick to the standard.....
@RogerChristopher surely has made good remarks on why we should omit it. However, agreeing on his points my only last concern is an indication of general test progress to the user. This can be useful in many scenario's where user interaction is involved.

At this moment the progress feedback is sub-optimal and could be improved upon. Therefore, maybe we can log some (debug level?) messages on the progress of the tests. This point was/is my question behind the previous question (I should have been more clear on it from the beginning. ..)

The PyTest test framework has got a good visual progress counter as it counts up from 0 to 100% and gives very good feedback;

. for success (per succes a dot is printed)
E for an error,
W for a warning.
S for skipped,
etc, etc.
lastly it prints the current percentual progress which is increased accordingly.

An output example: ...............s............................. [ 2%]

See https://blog.pytest.org/

EDIT
Also, this type of feedback will allow a user to detect that a test "hangs", as @ I-Campbell remarked.

[sagatowski]

Question is how are we going to present the status of the test without polluting the event-logger? TwinCAT doesn't have any standard output in the classical sense. Only way I see it is to add a variable that presents the overall status, which I think would be of very little use.

[Aliazzzz]

Agreed, a console output is not the same as a log output, we should think of a middleground.
A progress counter is a good idea, just how do we implement it in a clear way.

[RogerChristopher]

A small visualisation defined in the library would work. It only needs to reference the test results and can be inserted into a frame in the client code. The results could be shown in a page-able table, or a string of ............F.........F....... characters. This can then also have buttons to re-run the tests, etc if desired.

…

On Mon, 1 Jun 2020, 21:15 Aliazzzz, ***@***.***> wrote: Agreed, a console output is not the same as a log output, we should think of a middleground. A progress counter is a good idea, just how do we implement it in a clear way. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/tcunit/TcUnit/issues/54#issuecomment-637078731>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH3VKDQJPTWEG4ER5WPTNRDRUQD6FANCNFSM4IWXXXUA> .

[sagatowski]

@Aliazzzz Do you have time to implement it and sketch a PR?

[sagatowski]

Implemented in #181