Tutorial Examples as Unit Tests

[martintb]

All of the examples in the tutorial should also be written to be unit tests. Since we already have the data extracted and pyPRISM scripts written (in the tutorial), this should not be difficult.

In general, adding full pyPRISM calculations for various systems as unit tests would be great for checking for regression when adding new features.

[HaroldHatchNIST]

I tried this in feasst but it is suboptimal and would be interested to see how you implement it. Currently I use a shell script to aggregate them and strip out some parts of the tutorial (in feasst they are called "testcases" and the nasty shell script is lines 11-20 of https://github.com/usnistgov/feasst/blob/master/tools/build/genGTestCases.sh). One problem I have is I either forget to set up the "tutorial" as a unittest and then when it fails I have to work to hunt it down; or if I set them up at unittests they look ugly to new users of the tutorial (e.g. extra unittest boiler plate..)

[martintb]

Hmmmm. I was more or less thinking of doing the same (manually copy tutorial examples from the Jupyter notebooks). From a maintainability point of view, this is obviously a not a idea.

It would be nice if the tutorial examples were automatically transferable as unit-tests. I believe that FiPy achieves this via doctest. Unfortunately, as my tutorial is in multiple Jupyter notebooks, I don't believe I can make use of doctest.

Let me know if you have other ideas and I'll do the same! I am primarily motivated to include tutorial examples as tests because one of my examples fails on certain machines and we have yet to figure out why. It would be much easier to detect this if the example was in the test-suite.

[HaroldHatchNIST]

I hadn't seen doctest before. That could work well for me. Thanks!