BROKEN: disabling tests

[cdsontag]

Looks like it. In pandeia:

eutychus:tests % pwd
/Users/sontag/dev/pandeia_test/tests

eutychus:tests % find . -name '*disab*'
./server/performance/calc_bench.py.disabled  # <-- cds - not sure what this one is doing
./server/performance/load_test.disable
./server/performance/db_bench.disable
./client/performance/niriss_ami.disable
./client/performance/miri_lrsslit.disable

and yet in last night's nightly run, there are 0 tests showing as disabled.

To be VERY clear, we should disable a single engine test and verify that IT also does not show up in the Disable column of pandokia.

[ariedel]

It looks like the three missing NIRISS AMI tests in the 2019-01-28 nightly appear because the process running that folder crashed upon hitting the disable file @cdsontag put in there to test:

Disabled : /installations/test_1/src/pandeia_test/tests/engine/jwst/instrument/niriss/defaults/ami/nrm_f277w_testing_pdk.jeng
Traceback (most recent call last):
  File "/installations/test_1/third_party_py2/envs/pandeia_9.1/bin/pdkrun", line 4, in <module>
    __import__('pkg_resources').run_script('pandokia==2.3.0.dev122+g5060a78d', 'pdkrun')
  File "/installations/test_1/third_party_py2/envs/pandeia_9.1/lib/python2.7/site-packages/pkg_resources/__init__.py", line 658, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/installations/test_1/third_party_py2/envs/pandeia_9.1/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1438, in run_script
    exec(code, namespace, namespace)
  File "/installations/test_1/third_party_py2/envs/pandeia_9.1/lib/python2.7/site-packages/pandokia-2.3.0.dev122+g5060a78d-py2.7.egg/EGG-INFO/scripts/pdkrun", line 19, in <module>
    sys.exit(pandokia.entry.run())
  File "/installations/test_1/third_party_py2/envs/pandeia_9.1/lib/python2.7/site-packages/pandokia-2.3.0.dev122+g5060a78d-py2.7.egg/pandokia/entry.py", line 182, in run
    (err, lstat) = x.run(args)
  File "/installations/test_1/third_party_py2/envs/pandeia_9.1/lib/python2.7/site-packages/pandokia-2.3.0.dev122+g5060a78d-py2.7.egg/pandokia/run.py", line 186, in run
    (err, lstat) = pandokia.run_dir.run(x, envgetter)
  File "/installations/test_1/third_party_py2/envs/pandeia_9.1/lib/python2.7/site-packages/pandokia-2.3.0.dev122+g5060a78d-py2.7.egg/pandokia/run_dir.py", line 102, in run
    l = m.lst(env)
AttributeError: 'module' object has no attribute 'lst'
End of output from process 61477 in slot 0, status=1

The missing tests sort lexicographically after the disabled test, and therefore did not run.

They also crashed on the 2019-01-28 Python 3 run, which doesn't check for missing tests.

[cdsontag]

CC'ing @ibusko since this is affecting HST tests as well.

[ariedel]

Still crashes on disable files as of 20190423. Same exact error and traceback.

[vglaidler]

Investigation so far:

It's dying in run_dir L102 because m has no .lst() method. There is a previous comment about m.list(), so this looks likely to have been the result of a list->lst sweep gone wrong somehow.

At L84, m is set to pandokia.run_file.get_runner_mod( runner )

The .jeng files are using the ttd runner

which probably maps to https://github.com/spacetelescope/pandeia_test/blob/master/test_utilities/pandokia_runner_ttd.py

which has a method named list (line 31), but not a method named lst

which suggests that the list->lst sweep to remove the use of reserved keywords broke an API that external runners were depending on.

Next steps:

• confirm that runner:ttd mapping to pandokia_runner_ttd is in fact correct
• modify pandokia_runner_ttd to change its method to .lst()
• find pandokia documentation on defining your own test runner & update it

[vglaidler]

The documentation on adding runners was indeed updated to change the API.