- EMSE-BBC-experiment
- Table of contents
- Subject selection
- Test Generation
- Captured failures
- Data analysis
- License & Copyright
! Requirements: Defects4j, Rscript, and Docker
Note: The outputs of bug and subject selection is already available in this replication package. So, for continuing with the next steps, you can skip this step.
bash subjects/collect-bugs.sh
This script generates bugs.csv that contains project_id, bug_id, date fixed, and modified classes of selected bugs.
bash subjects/collect-target-classes.sh
This script generates subjects.csv in which all of the class under tests are indicated. This csv file will be used for the test generation process.
bash subjects/collect-and-compile-buggy-versions.sh
This script store and complie all of the buggy versions required for this experimen in subjects/buggy-versions.
For running the test generation tools:
First, you need to build the docker image:
. scripts/docker/build-test-generation-image.sh dockerThen, you need to run the docker container:
. scripts/docker/run-test-generation-container.sh dockerNote: This replication package contains the tests generated during our experiment.
Directory results/ contains the tests generated by each configuration; file results/results.csv contains all of the reports generated by each instance of EvoSuite; and csv files in results/results-with-intervals contains all of the EvoSuite reports that we executed to measure the EvoSuite's coverage in different time intervals. So, you can skip this step, if you want to run the next steps.
For running the test generation you need to use the following command:
docker exec -it test-generation-container bash -c ". scripts/main/test-generation.sh <number-of-parallel-processes> <first-round> <last-round> <time-budget> <memory-used-by-each-evosuite-instance> <list-of-configurations> <list-of-subjects> [<seeds-dicrectory>]"
These to input parameters indicate the number of rounds each test generation process will repeat. For instance, to run the first round of test generation (on all of the sybjects), these two parameters should be both 1. To limit the test generaiton configurations and class under tests, CSV files <list-of-configurations> and <list-of-subjects> should be modified, respectively.
All of the seeds used for our experiment is already saved in results/SEED. For replicating our experiment, you need to pass this address to the main test generation script:
docker exec -it test-generation-container bash -c ". scripts/main/test-generation.sh 30 1 30 600 3000 configurations/configurations.csv subjects/subjects.csv results/SEED/"
Note: In this case, <number-of-parallel-processes> (the first parameter) is set to 30, which means by running this command, the script will try to run 30 instances of EvoSuite on your machine. This value should be modified according the number of cores that your computing machine has.
For running the experiment with new seeds, just run the script without passing the seeds directory.
docker exec -it test-generation-container bash -c ". scripts/main/test-generation.sh 30 1 30 600 3000 configurations/configurations.csv subjects/subjects.csv"
After running the experiment, the results of the test generations are all saved in results directory.
Currently, this directory contains the results achieved during our experiment.
For collecting the whole results in the csv file, run the following script:
. scripts/csv/generate_final_csv.sh 1 30 configurations/configurations.csv subjects/subjects.csv
The csv will be saved at results/results.csv.
! All of the bug exposing stack traces are already saved in the root directory of buggy versions in subjects/buggy-versions/<bug-name>. However, to generate them again, you can run the following script:
. scripts/faults/generate_stack_traces.sh subjects/subjects.csv
This script can be executed in the test generation docker container as well.
! The generated tests without try/catch is already avaible in this replication package (directory tests-without-trycatch). So, you can skip this step and continue with Data analysis scripts.
For making sure that the infrustracture can collect all of the captured exceptions, first, we need to run the following script to remove all of the try catches from the generated test suites:
python scripts/faults/remove-try-catches.py
This script can be executed in the test generation docker container as well:
docker exec -it test-generation-container bash -c "python scripts/faults/remove-try-catches.py"
Finally, to collect the fault coverages, simply run the following script:
python scripts/faults/analyze-stacktraces.py
This script can be executed in the test generation docker container as well:
docker exec -it test-generation-container bash -c "python scripts/faults/analyze-stacktraces.py"
The csv file reporting the fault coverages will be saved as data/captured_exceptions.csv.
! Currently, this file contains the results that we have collected from our main experiment.
The R scripts used for the data analysis are available in the data-analysis/ folder.
To run the data analysis for RQ1, run the following command from the data-analysis/ folder:
Rscript BBC-called.R
To run the data analysis for RQ1.1-1.3, run the following command from the data-analysis/ folder:
Rscript analysis.r
To run the data analysis for RQ1.1.4, run the following command from the data-analysis/ folder:
Rscript time-intervals.R
The graphs and tables will be generated in the data-analysis/output/ folder.
The materials herein are all copyright of Pouria Derakhshanfar and Xavier Devroey. The material was produced while working at Delft University of Technology and the University of Namur.
All the source code in this repository is released under MIT License.
The documentation and data are licensed under a Creative Commons Attribution 4.0 License.