Added before/after methods and test duration + deployment tests

[sebjulliand]

Changes

This PR enhances the test framework.

Deployment tests (requires an opened workspace folder)

• Added Deploy all test
• Added Deploy compare test

Test suite level

• Optional before method: if present, runs code before the test suite is executed and cancels the whole suite if it fails. The failure is displayed as a tooltip in the Tests browser.
• Optional after method: if present, runs unconditionally after the test suite is done running (even if cancelled)
• New status field
• Displays a spinning gear when the test suite is running

Test case level

• Added a duration field, filled when the test case is done. The duration is displayed as a tooltip in the Tests browser.

Tests view

• Added a Collapse All button
• Fixed error handling when the error thrown is not an Error

Checklist

• have tested my change
• for feature PRs: PR only includes one feature enhancement.

[chrjorgensen]

@sebjulliand Looking good to me - but can you provide me with some before and after methods to use for testing the code?

[sebjulliand]

Sure! I was keeping them in another PR for the deployment tests; but I can put a few of these tests in these PR 😉

[sebjulliand]

There you go @chrjorgensen ; I added the Deployment test suite as well.
It was originally for it that I introduced the before/after functions. So I guess adding it to the PR is not bad 😊

[chrjorgensen]

@sebjulliand before and after methods tested and works perfectly - of course! 😉

However, the deployment tests fails (probably okay, since no workspace folder were configured), but the tests in the deployment suite disappears from the tests tree:

[sebjulliand]

@chrjorgensen that was the point: if the before method fails, it is supposed to compromise the whole test suite.
When that happens, the tooltip holds the failure message and no tests are shown. Do you think it would be best to show them anyway?

[chrjorgensen]

@sebjulliand The tests were shown before they were activated, and then they were gone - which is/was confusing to me. Now that you explain why, I see what you're trying... is it possible to show the tests in grey or have an icon to show they were not run?

[sebjulliand]

@sebjulliand The tests were shown before they were activated, and then they were gone - which is/was confusing to me. Now that you explain why, I see what you're trying... is it possible to show the tests in grey or have an icon to show they were not run?

Great idea, as always! I'll do that!

[chrjorgensen]

@sebjulliand Beautiful - just what I wanted:

👌

However, I tried adding before and after to the ContentSuite testsuite, and when I throw an error in before, the test tree does not look correct?

???

[sebjulliand]

Interesting case! I didn't know you could just throw a string like that. I always threw an Error so far.

throw new Error("Boop");

So just throwing a string would result in no failure message being retrieved, and so the test case would not be considered as having failed.

It should be OK now.

I also made the test suite icon a spinning gear when running (because I like it) and also turned on the Collapse All button on the view.

[chrjorgensen]

Beginners are the best testers! 😆

What about errors in the after method? Should they stop the test or just be ignored like those in before? When I throw an error (string! 😉) in after, it stops the test run:

[sebjulliand]

Good question!
So from now on, if after fails, the TestSuite will be colored in red and the failure reported on the tooltip. Tests that ran in the suite won't be affected if after fails. What do you think?

[chrjorgensen]

Perfect, Seb! 😍

Approved and merged...