race condition in error log detection in test_stdout.py

[benclifford]

Describe the bug

occasionally tests fail with an error like this:

=================================== FAILURES ===================================
______________________________ test_stdout_append ______________________________

tmpd_cwd = PosixPath('/home/runner/work/parsl/parsl/pytest-parsl/parsltest-current/test_stdout_append-vgi3zea4')
caplog = <_pytest.logging.LogCaptureFixture object at 0x7f3b7a3c7210>

    @pytest.mark.shared_fs
    def test_stdout_append(tmpd_cwd, caplog):
        """Testing appending to prior content of stdout (default open() mode)"""
    
        out = str(tmpd_cwd / 't1.out')
        err = str(tmpd_cwd / 't1.err')
    
        echo_to_streams('hi', stdout=out, stderr=err).result()
        len1 = len(open(out).readlines())
    
        echo_to_streams('hi', stdout=out, stderr=err).result()
        len2 = len(open(out).readlines())
    
        assert len1 == 1 and len2 == 2
    
        for record in caplog.records:
>           assert record.levelno < logging.ERROR
E           assert 40 < 40
E            +  where 40 = <LogRecord: parsl.dataflow.dflow, 40, /home/runner/work/parsl/parsl/parsl/dataflow/dflow.py, 1346, "Standard out for task 105 has unknown specification: ['t3.out', 'w']">.levelno
E            +  and   40 = logging.ERROR

parsl/tests/test_bash_apps/test_stdout.py:130: AssertionError

This test has an assertion that no error is logged. But I think it is detected an error logged from a previous test that does log an error - the error text suggests its from a log message from a different test in the same test module (test_bad_stdout_specs) and that there is a race condition around completion of that test and log messages being recorded.

I've seen this repeatedly over the years, but I couldn't find an open issue reporting it(!)