diff --git a/README.md b/README.md index 2932d44..06c4ced 100644 --- a/README.md +++ b/README.md @@ -306,13 +306,38 @@ We recommend to create the following folder structure for MultiCorrupt using . . ``` -A script that generates all corruptions for all severity levels will be provided soon. + +### MultiCorrupt Evaluation +If you have created MultiCorrupt in the structure above, we recommend you to use +our simple [evaluation script](helper/multicorrupt_evaluation.sh) that iterates over +the whole dataset, executes the evaluation and extracts the NDS and mAP metrics. + +In the script you would need to replace the pathes for `multicorrupt_root`, +`nuscenes_data_dir` and `logfile` according to your setup. +```bash +#!/bin/bash + +# List of corruptions and severity levels +corruptions=("beamsreducing" "brightness" "dark" "fog" "missingcamera" "motionblur" "pointsreducing" "snow" "spatialmisalignment" "temporalmisalignment") +severity_levels=("1" "2" "3") + +# Directory paths +multicorrupt_root="/workspace/multicorrupt/" +nuscenes_data_dir="/workspace/data/nuscenes" +logfile="/workspace/evaluation_log.txt" + +. +. +. +``` + ## TODOs - [ ] Add more visualization - [ ] Add contribution guidelines + ## Contribution - **Coming Soon** - How to contribute diff --git a/helper/multicorrupt_evaluation.sh b/helper/multicorrupt_evaluation.sh new file mode 100644 index 0000000..61646b8 --- /dev/null +++ b/helper/multicorrupt_evaluation.sh @@ -0,0 +1,39 @@ +#!/bin/bash + +# List of corruptions and severity levels +corruptions=("beamsreducing" "brightness" "dark" "fog" "missingcamera" "motionblur" "pointsreducing" "snow" "spatialmisalignment" "temporalmisalignment") +severity_levels=("1" "2" "3") + +# Directory paths +multicorrupt_root="/workspace/multicorrupt/" +nuscenes_data_dir="/workspace/data/nuscenes" +logfile="/workspace/evaluation_log.txt" + +# Model evaluation command (replace with your actual command) + +# Loop over corruptions and severity levels +for corruption in "${corruptions[@]}"; do + for severity in "${severity_levels[@]}"; do + # Log the current configuration in the terminal + echo "Current Configuration: Corruption=$corruption, Severity=$severity" + + # Create soft link in /workspace/data/nuscenes + ln -s "$multicorrupt_root/$corruption/$severity" "$nuscenes_data_dir" + + # Perform model evaluation + output=$(bash tools/test.py /workspace/projects/configs/path_to_config.py /workspace/ckpts/path_to_checkpoint.pth) + + # Save the entire output to a separate text file + echo "$output" > "/workspace/${corruption}_${severity}_output.txt" + + # Extract NDS and mAP scores from the output + nds=$(echo "$output" | grep "NDS:" | awk '{print $2}') + map=$(echo "$output" | grep "mAP:" | awk '{print $2}') + + # Save results to the logfile + echo "Corruption: $corruption, Severity: $severity, NDS: $nds, mAP: $map" >> "$logfile" + + # Remove soft link + rm "$nuscenes_data_dir" + done +done