aNNoTest subjects

This repository collects various versions of open-source neural-network Python programs with 62 bugs surveyed in Islam et al. (2019).

Each buggy version includes annotations for the aNNoTest test generator, such that running aNNoTest on the annotated program automatically generates bug-triggering tests (as well as other tests).

The repository also includes scripts and data about the bugs, and thus it provides:

• Concrete examples of using aNNoTest

• Reproducibility data and artifacts for aNNoTest's experimental evaluation

• A reproducible dataset of bugs in neural-network Python programs, to support further research in this domain

Bug artifacts

File bugs_info.csv contains information about each of the 62 bugs in this repository: bug number #, unique identifiers internal_id and id, used deep-learning framework (e.g., Keras, TensorFlow), project name project and acronym, and link to the bug-introducing commit in the original repository.

For each bug, we include the complete source code of the project commit with the bug, as well as the following artifacts:

• Source-code annotations used by aNNoTest to generate tests (e.g., annotations).

• A comment # repo_bug indicating the location of the bug (e.g., bug location).

• File requirements.txt listing the project dependencies (e.g., requirements.txt).

• Script make_env.sh, which creates an Anaconda environment with a working installation of the project (e.g., make_env.sh).

• Script make_data.sh, which downloads any dataset required to run the project (e.g., make_data.sh). If there is no file make_data.sh, the project doesn't have any external data dependencies.

• The failing test generated by aNNoTest that reveals the bug (e.g., aNNoTest test) and file annotest_test_name.txt with the name of this test (e.g., annotest_test_name.txt). aNNoTest generates tests in Hypothesis format.

• A manually-written Pytest test equivalent to the failing test generated by aNNoTest (e.g., Pytest manual test). This is provided for easier manual analysis, as Pytest tests are usually easier to read.

• Script run_annotest_failing_test.sh, which activates Anaconda environment and runs the bug-triggering test generated by aNNoTest (e.g., run_annotest_failing_test.sh).

• Script run_manual_failing_test.sh, which activates Anaconda environment and runs the bug-triggering Pytest test (e.g., run_manual_failing_test.sh).

• File expected_error_message.txt with the error messages printed when running the bug-triggering test generated by aNNoTest, and the bug-triggering Pytest test (e.g., expected_error_message.txt).

Reproducing the bugs

To reproduce any of the bugs in this repository:

1. Install Anaconda 3 on your machine.

2. Set enviroment variable ANACONDA3_DIRECTORY to point to the installation directory of Anaconda on your machine (usually, ~/anaconda3).

export ANACONDA3_DIRECTORY=~/anaconda3

3. Clone this repository on your machine into a directory $ANNOTEST_SUBJECTS.

4. To reproduce a bug, go to the bug's directory in the repository. For example, bug keras_adversarial_b1.

cd "$ANNOTEST_SUBJECTS/keras_adversarial_b1"
# make sure scripts are executable
chmod u+x *.sh

5. Run make_env.sh to create a project-specific Anaconda environment and install the project.

6. Run make_data.sh to download any project-specific data dependencies (skip this step if there is no file make_data.sh).

7. Activate the Anaconda environment generated in the previous steps, and install aNNoTest inside it. For example, bug keras_adversarial_b1.

conda activate keras_adversarial_b1
pip install annotest

8. Run run_annotest_failing_test.sh to run the bug-triggering test generated by aNNoTest. Check that the test execution terminates with the same error message as in expected_error_message.txt.

9. Run run_manual_failing_test.sh to run the manually-written Pytest test (equivalent to the aNNoTest-generated test). Check that the test execution terminates with the same error message as in expected_error_message.txt.

10. If you want to generate the aNNoTest tests from scratch, remove directory test_annotest and then run annotest on the root directory project. For example, bug keras_adversarial_b1.

cd "$ANNOTEST_SUBJECTS/keras_adversarial_b1" 
rm -rf test_annotest
annotest .

Citing aNNoTest

You can cite the work on aNNoTest as follows:

Mohammad Rezaalipour, Carlo A. Furia: An annotation-based approach for finding bugs in neural network programs. J. Syst. Softw. 201: 111669 (2023)

@article{aNNoTest-JSS,
   title = {An annotation-based approach for finding bugs in neural network programs},
   journal = {Journal of Systems and Software},
   volume = {201},
   pages = {111669},
   year = {2023},
   issn = {0164-1212},
   doi = {https://doi.org/10.1016/j.jss.2023.111669},
   author = {Mohammad Rezaalipour and Carlo A. Furia}
}

Mirrors

This repository is a public mirror of (part of) aNNoTest's private repository of experiments. There are two public mirrors, whose content is identical:

• https://github.com/atom-sw/annotest-subjects
• https://github.com/mohrez86/annotest-subjects