Rewrite re2c tests runner in Python #360

sergeyklay · 2021-06-16T17:03:58Z

This pull request introduces a completely rewritten test runner in Python.

The main goals of this change:

Provide a OS-independent way to run tests (at least using CMake)
Getting rid of dependence on Unix utilities in tests (grep, cat, tr, etc.)
Flexibility
Ease of maintenance

sergeyklay · 2021-06-16T22:35:27Z

There is no /home/skvadrik/devel/50/re2c/include among them, and no suitable relative path (considering that the working directory is /home/skvadrik/devel/50/re2c/test_210616222213/, -I include means a copy of /home/skvadrik/devel/50/re2c/test/include, not /home/skvadrik/devel/50/re2c/include.

If I patch the command and add -I ../include then
{c,go}/unicode_identifier.re do not fail anymore and I'm down to two
failures:

Yes you are right. This is exactly what I decided at the very beginning. However, this doesn't work for some reason:

Right now I tried:

diff --git a/run_tests.py b/run_tests.py
index 9c93e4a9..2551e9ee 100644
--- a/run_tests.py
+++ b/run_tests.py
@@ -539,7 +539,8 @@ def init_process(base_path, skeleton=False, keep_temp_files=False,
     # abshere('@top_srcdir@', 'include')
     #   vs
     # abshere('..', '@top_srcdir@', 'include')
-    _ctx.incpaths.append(f"-I {abshere(base_path, 'include')}")
+    _ctx.incpaths.append("-I /Users/egrep/work/re2c/include")
+    _ctx.incpaths.append("-I ../include")
 
     # Set tools paths
     _ctx.diff = which('diff')[0]

And output is still the same:

$ python run_tests.py --keep-temp-files

-----------------
All:         1615
Ran:         1615
Passed:      1611
Soft errors: 0
Hard errors: 4
-----------------
FAILED

sergeyklay · 2021-06-16T22:37:36Z

@skvadrik Can you please commit your patch with the fix of the two tests? I may be missing something

skvadrik · 2021-06-16T22:38:24Z

Thanks a lot for working on this!

One thing off the top of my head is that skeleton test failures can be deceptive: if re2c fails with an error, it is counted as a "soft error" because we have some tests that deliberately fail. Currently I see 222 soft failures in run_tests.py and 221 for run_tests.sh. The one extra error will likely go away after fixing non-skeleton tests, but the whole "soft error" concept is very weak and I added it as a temporary hack to monitor how many tests actually run and how many fail before that.

skvadrik · 2021-06-16T22:45:05Z

@skvadrik Can you please commit your patch with the fix of the two tests? I may be missing something

I think I know what it is, it should be two separate arguments -I and ./include, not -I ../include. At least I tried that change and the errors dropped from 4 to 2.

sergeyklay · 2021-06-16T22:46:05Z

Currently I see 222 soft failures in run_tests.py and 221 for run_tests.sh.

Yeah, I saw this yesterday, but for now I decided to focus on non-skeleton tests.

sergeyklay · 2021-06-16T22:48:24Z

I think I know what it is, it should be two separate arguments -I and ./include, not -I ../include. At least I tried that change and the errors dropped from 4 to 2.

Yay! I spent about 8 hours on this! You're Chuck Norris :)

skvadrik · 2021-06-16T22:51:08Z

Yay! I spent about 8 hours on this! You're Chuck Norris :)

I'm used to spending 8h a day on things like this (most of my working day really), so no wonder I could spot it. :D

sergeyklay · 2021-06-16T23:09:49Z

I think I know what it is, it should be two separate arguments -I and ./include, not -I ../include. At least I tried that change and the errors dropped from 4 to 2.

Yay!

diff --git a/run_tests.py b/run_tests.py
index 8b84fa3d..ecd8ae57 100644
--- a/run_tests.py
+++ b/run_tests.py
@@ -532,7 +532,8 @@ def init_process(base_path, skeleton=False, keep_temp_files=False,
     # Set include paths, relative to build directory
     cwd = os.getcwd()
     os.chdir(_ctx.base_path)
-    _ctx.incpaths = [f'-I {d}' for d in Path('.').rglob('*') if os.path.isdir(d)]
+    paths = Path('.').rglob('*')
+    _ctx.incpaths = sum([['-I', str(d)] for d in paths if os.path.isdir(d)], [])
     os.chdir(cwd)
 
     # TODO: Fix this to sort out with absolute and relative paths + @top_srcdir@

-----------------
All:         1615
Ran:         1615
Passed:      1615
Soft errors: 0
Hard errors: 0
-----------------

Explanation

list1 = ['111', '222', '333']
sum([['-I', item] for item in list1], [])

# Out: ['-I', '111', '-I', '222', '-I', '333']

sergeyklay · 2021-06-16T23:40:09Z

@skvadrik I updated the description of the PR. All tests are passed now.

skvadrik · 2021-06-17T06:14:09Z

@skvadrik I updated the description of the PR. All tests are passed now.

\o/

sergeyklay · 2021-06-17T09:18:32Z

It is ready to review

skvadrik · 2021-06-17T10:38:58Z

It is ready to review

Awesome, thanks! I may be unable to review it today, but I'll do it at the first opportunity.

sergeyklay · 2021-06-17T11:01:27Z

There are some issues with external tools on Windows. I'll try to sort out in next PRs to not mix changes in this one

sergeyklay · 2021-06-17T23:13:16Z

Some good (or bad) news: I managed to run tests runner under Windows. And I see half of the tests are failed. On the one hand, this is bad news, but on the other hand, we did not have the opportunity to run native compilation (not Mingw) and tests under WIndows until this moment. So now we see the real picture

skvadrik · 2021-06-17T23:21:45Z

Some good (or bad) news: I managed to run tests runner under Windows. And I see half of the tests are failed.

Can you start a new bug and post detail how the tests failed? E.g. upload the temporary test directory as an archive.

sergeyklay · 2021-06-17T23:31:59Z

Can you start a new bug and post detail how the tests failed? E.g. upload the temporary test directory as an archive.

Sure. Let me some time to sort out with unstable test runner under the Windows (some regex compilation issues under the loops)

sergeyklay · 2021-06-18T04:30:33Z

Some good (or bad) news: I managed to run tests runner under Windows. And I see half of the tests are failed. On the one hand, this is bad news, but on the other hand, we did not have the opportunity to run native compilation (not Mingw) and tests under WIndows until this moment. So now we see the real picture

I fixed tests runner to run under the Windows. There were some funny bugs like Windows directory separator \ interpreted as a regex escape command, some Cmake issues related to build under the Windows and so on. And at the moment we have the following at GitHub actions:

Running in 2 job(s) 
----------------- 
All: 1615 
Ran: 1615 
Passed: 264 
Soft errors: 0
Hard errors: 1351 
----------------- 
FAILED

I'll open a new PR with all necessary changes and show you all the remaining issues. But lets do it after this one will be merged so that I can rebase onto the proper branch and open a PR.

skvadrik

Thanks again for doing this work! I don't know python very well, and to me the code looks nice and clean (much better than bash!). There is a couple of things to fix, but largely it just works. \o/

CMakeLists.txt

run_tests.py.in

skvadrik · 2021-06-18T22:54:21Z

By the way, the new script is already ~2x faster than the old one (maybe more). :)

run_tests.py.in

This pull request introduces a completely rewritten test runner in Python. The main goals of this change: - Provide a OS-independent way to run tests (at least using CMake) - Getting rid of dependence on Unix utilities in tests (grep, cat, tr, etc.) - Flexibility - Ease of maintenance

sergeyklay · 2021-06-19T11:01:57Z

@skvadrik It is ready to merge now.

skvadrik · 2021-06-19T13:21:04Z

Great, I have just merged it! \o/

sergeyklay · 2021-06-19T13:56:23Z

Great, I have just merged it! \o/

Yay! I'm glad this finally happened :)

sergeyklay added the wip label Jun 16, 2021

sergeyklay self-assigned this Jun 16, 2021

sergeyklay linked an issue Jun 16, 2021 that may be closed by this pull request

Some skeleton tests are extremely slow on Windows #331

Open

sergeyklay force-pushed the feature/py-test-runner branch from 981ed52 to 57f7faf Compare June 17, 2021 08:25

sergeyklay changed the title ~~WIP: re2c test runner in Python~~ Rewrite re2c test runner in Python Jun 17, 2021

sergeyklay marked this pull request as ready for review June 17, 2021 08:26

sergeyklay removed the wip label Jun 17, 2021

sergeyklay requested a review from skvadrik June 17, 2021 08:27

sergeyklay removed a link to an issue Jun 17, 2021

Some skeleton tests are extremely slow on Windows #331

Open

sergeyklay force-pushed the feature/py-test-runner branch from ca802e9 to 3421d53 Compare June 17, 2021 09:15

sergeyklay force-pushed the feature/py-test-runner branch from 34949a4 to 3421d53 Compare June 17, 2021 11:01

sergeyklay force-pushed the feature/py-test-runner branch 2 times, most recently from 801f248 to d045ef2 Compare June 17, 2021 22:38

sergeyklay changed the title ~~Rewrite re2c test runner in Python~~ Rewrite re2c tests runner in Python Jun 18, 2021

sergeyklay force-pushed the feature/py-test-runner branch from 9962628 to cd433e3 Compare June 18, 2021 19:53

skvadrik requested changes Jun 18, 2021

View reviewed changes

CMakeLists.txt Show resolved Hide resolved

run_tests.py.in Outdated Show resolved Hide resolved

skvadrik reviewed Jun 18, 2021

View reviewed changes

run_tests.py.in Outdated Show resolved Hide resolved

trofi reviewed Jun 18, 2021

View reviewed changes

run_tests.py.in Show resolved Hide resolved

skvadrik mentioned this pull request Jun 19, 2021

run_tests.py: fix cleanup of temporary directories after the test. sergeyklay/re2c#1

Closed

sergeyklay force-pushed the feature/py-test-runner branch from fcf8e13 to 62733a6 Compare June 19, 2021 10:52

sergeyklay force-pushed the feature/py-test-runner branch from 62733a6 to 7600c33 Compare June 19, 2021 11:00

skvadrik approved these changes Jun 19, 2021

View reviewed changes

skvadrik merged commit 9c99811 into skvadrik:master Jun 19, 2021

sergeyklay deleted the feature/py-test-runner branch June 19, 2021 17:44

skvadrik mentioned this pull request Jun 19, 2021

run_tests.py: don't shard tests, use chunk size 1. #362

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rewrite re2c tests runner in Python #360

Rewrite re2c tests runner in Python #360

sergeyklay commented Jun 16, 2021 •

edited

Loading

sergeyklay commented Jun 16, 2021

sergeyklay commented Jun 16, 2021

skvadrik commented Jun 16, 2021

skvadrik commented Jun 16, 2021

sergeyklay commented Jun 16, 2021

sergeyklay commented Jun 16, 2021

skvadrik commented Jun 16, 2021

sergeyklay commented Jun 16, 2021 •

edited

Loading

sergeyklay commented Jun 16, 2021

skvadrik commented Jun 17, 2021

sergeyklay commented Jun 17, 2021

skvadrik commented Jun 17, 2021

sergeyklay commented Jun 17, 2021

sergeyklay commented Jun 17, 2021

skvadrik commented Jun 17, 2021

sergeyklay commented Jun 17, 2021

sergeyklay commented Jun 18, 2021 •

edited

Loading

skvadrik left a comment

skvadrik commented Jun 18, 2021 •

edited

Loading

sergeyklay commented Jun 19, 2021

skvadrik commented Jun 19, 2021

sergeyklay commented Jun 19, 2021 •

edited

Loading

Rewrite re2c tests runner in Python #360

Rewrite re2c tests runner in Python #360

Conversation

sergeyklay commented Jun 16, 2021 • edited Loading

sergeyklay commented Jun 16, 2021

sergeyklay commented Jun 16, 2021

skvadrik commented Jun 16, 2021

skvadrik commented Jun 16, 2021

sergeyklay commented Jun 16, 2021

sergeyklay commented Jun 16, 2021

skvadrik commented Jun 16, 2021

sergeyklay commented Jun 16, 2021 • edited Loading

sergeyklay commented Jun 16, 2021

skvadrik commented Jun 17, 2021

sergeyklay commented Jun 17, 2021

skvadrik commented Jun 17, 2021

sergeyklay commented Jun 17, 2021

sergeyklay commented Jun 17, 2021

skvadrik commented Jun 17, 2021

sergeyklay commented Jun 17, 2021

sergeyklay commented Jun 18, 2021 • edited Loading

skvadrik left a comment

Choose a reason for hiding this comment

skvadrik commented Jun 18, 2021 • edited Loading

sergeyklay commented Jun 19, 2021

skvadrik commented Jun 19, 2021

sergeyklay commented Jun 19, 2021 • edited Loading

sergeyklay commented Jun 16, 2021 •

edited

Loading

sergeyklay commented Jun 16, 2021 •

edited

Loading

sergeyklay commented Jun 18, 2021 •

edited

Loading

skvadrik commented Jun 18, 2021 •

edited

Loading

sergeyklay commented Jun 19, 2021 •

edited

Loading