Skip to content

Latest commit

 

History

History

second_round_selection

README.md

Second round selection

Purpose

The purpose of this phase is that we go through the experimental data generated by running the bash scripts and find those that did not finish successfully. We analyze these cases manually and fix them if possible. For instance, if the problem is caused by a bug in FauxPy, we fix it, or if it is due to server crash, we remove the data and repeat that experiment. However, sometimes the problems cannot be fixed for the following reasons:

  1. Due to missing requirements, some buggy versions cannot run all their test cases while they have enough of their requirements to run their target failing tests. The reason is that some Python projects have two sets of requirements: one for development and one for usage. The are some cases of bugs in BugsInPy that do not include the first one.

  2. Due to timeout, some buggy versions do not finish. The cluster server we used for the experiments had a timeout limit of 48 hours, and buggy versions that required more than 48 hours, could not be used in our experiments.

We also go through the fishy results. A run is fishy if it does not have any record in its csv files, or all of the records in a csv file are of the same score. If a fishy run is correct, we add it to the correct_fishy.csv file so that we do not analyze them in other iterations of the second round selection.

After finding those that have problems, we check them manually to see if the problems can be fixed. If they cannot be fixed, we add them to the manually_removed_bugs.csv file, which is then transformed into file manually_removed_bugs.json. Then, we copy manually_removed_bugs.json to the first round selection directory and run the simulation again to find replacements for the ones that were removed at this phase.

To perform this phase, we followed the rules in script check_experimental_results.py (Line 195-201).

Before running this phase, set the variables in path_items.json. Variable RESULTS_PATH points to the path of results directory explained in the main README.md of the repository, and variable SCRIPTS_PATH points to the scripts directory.