Windows test failures not caught by CI

[chapulina]

Environment

• OS Version: Windows
• Source or binary build? source

Description

• Expected behavior: Tests pass. And if they don't pass, CI is red.
• Actual behavior: Tests fail, and CI is green.

Steps to reproduce

1. Run Windows CI on Jenkins
2. See how it says on the front page that there are no test failures
3. See on the logs that there were indeed failures

Output

      Start 85: ign_log_record_no_overwrite

85: Test command: C:\Ruby30-x64\bin\ruby.exe "C:/Jenkins/workspace/ign_transport-ci-win/ws/ign-transport/log/test/integration/ign_log_record_no_overwrite.rb"
85: Environment variables: 
85:  IGN_CONFIG_PATH=C:/Jenkins/workspace/ign_transport-ci-win/ws/build/ignition-transport12/log/test/lib/ruby/ignition
85:  IGN_TRANSPORT_LOG_SQL_PATH=C:/Jenkins/workspace/ign_transport-ci-win/ws/ign-transport/log/sql
85: Test timeout computed to be: 10
85: C:/Jenkins/workspace/ign_transport-ci-win/ws/ign-transport/log/test/integration/ign_log_record_no_overwrite.rb:33:in `<main>': undefined method `zero?' for nil:NilClass (NoMethodError)
85: -----stdout-----
85: I cannot find any available 'ign' command:
85: 	* Did you install any ignition library?
85: 	* Did you set the IGN_CONFIG_PATH environment variable?
85: 	    E.g.: export IGN_CONFIG_PATH=$HOME/local/share/ignition
85: -----stderr-----
85: 
85: -----status-----
85: pid 10024 SIGSEGV (signal 11)
85/86 Test #85: ign_log_record_no_overwrite ............................***Failed    0.38 sec
test 86
      Start 86: ign_log_record_force

86: Test command: C:\Ruby30-x64\bin\ruby.exe "C:/Jenkins/workspace/ign_transport-ci-win/ws/ign-transport/log/test/integration/ign_log_record_force.rb"
86: Environment variables: 
86:  IGN_CONFIG_PATH=C:/Jenkins/workspace/ign_transport-ci-win/ws/build/ignition-transport12/log/test/lib/ruby/ignition
86:  IGN_TRANSPORT_LOG_SQL_PATH=C:/Jenkins/workspace/ign_transport-ci-win/ws/ign-transport/log/sql
86: Test timeout computed to be: 10
86: -----stdout-----
86: I cannot find any available 'ign' command:
86: 	* Did you install any ignition library?
86: 	* Did you set the IGN_CONFIG_PATH environment variable?
86: 	    E.g.: export IGN_CONFIG_PATH=$HOME/local/share/ignition
86: -----stderr-----
86: 
86: -----status-----
86: pid 1112 SIGSEGV (signal 11)
86/86 Test #86: ign_log_record_force ...................................***Failed    2.14 sec

https://build.osrfoundation.org/job/ign_transport-ci-win/19/consoleFull

[iche033]

in gz-transport13 and 14, there are test timeouts on Windows CI but the builds are still green and not marked as failed:

The following tests FAILED:
	 47 - INTEGRATION_twoProcsSrvCallStress (Timeout)
	 53 - INTEGRATION_twoProcsSrvCallWithoutInputStress (Timeout)
	 59 - INTEGRATION_twoProcsSrvCallWithoutOutputStress (Timeout)
	 87 - gz_log_record_no_overwrite (Failed)

The INTEGRATION_twoProcsSrvCall* tests actually pass but for some reason it times out at the end:

47: Test command: C:\J\workspace\gz_transport-pr-win\ws\build\gz-transport13\bin\Release\INTEGRATION_twoProcsSrvCallStress.exe "--gtest_output=xml:C:/J/workspace/gz_transport-pr-win/ws/build/gz-transport13/test_results/INTEGRATION_twoProcsSrvCallStress.xml"
47: Test timeout computed to be: 240
47: DynamicFactory(): Unable to parse descriptor set from [C:/J/workspace/gz_transport-pr-win/ws/install/gz-msgs10\share\gz\protos\gz-msgs10.gz_desc]
47: [==========] Running 1 test from 1 test suite.
47: [----------] Global test environment set-up.
47: [----------] 1 test from twoProcSrvCall
47: [ RUN      ] twoProcSrvCall.ThousandCalls
47: [       OK ] twoProcSrvCall.ThousandCalls (6160 ms)
47: [----------] 1 test from twoProcSrvCall (6160 ms total)
47: 
47: [----------] Global test environment tear-down
47: [==========] 1 test from 1 test suite ran. (6160 ms total)
47: [  PASSED  ] 1 test.
47/97 Test #47: INTEGRATION_twoProcsSrvCallStress ......................***Timeout 240.01 sec

[azeey]

@Crola1702 Is this something you can look into?

[Crola1702]

@Crola1702 Is this something you can look into?

Yes, will take a look

[Blast545]

@Crola1702 friendly ping