Accuracy Aware Quantization- Object Segmentation fails for OpenVINO 2024.3 and ultralytics_8.2.74 with yolov8-seg model and coco128-seg dataset

[fmishra747]

🐛 Describe the bug

Accuracy Aware segmentation script fails for openvino 2024.3 at

validator = model.ValidatorClass(args)
File "/data/venv/openvino_2024.1.0/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1729, in getattr
raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")
AttributeError: 'YOLO' object has no attribute 'ValidatorClass'

looks like need to find the right version of ultralytics, nncf which works with ov2024.3

Attached log with error for openvino 2024.3
accuracy_aware_segmentation_openvino2024.3.log

The same script works with
openvino 2024.1,
ultralytics 8.0.43
nncf 2.11.0.dev0+24f0ee70

Attached working log for openvino 2024.1
accuracy_aware_object_segmenttaion_ov2024.1_ultralytics8.0.43-working.log

Environment

The accuracy aware segmentation fails in the environment with openvino 2024.3 and the following packages:

OpenVINO environment
openvino 2024.3.0
openvino-dev 2024.3.0
openvino-genai 2024.3.0.0
openvino-telemetry 2024.1.0
openvino-tokenizers 2024.3.0.0
ultralytics 8.2.74
ultralytics-thop 2.0.0
nncf 2.13.0.dev0+eb91af2d
torch 2.4.0+cpu
torchvision 0.19.0+cpu
numpy 1.23.5

Minimal Reproducible Example

quantize_accuracy_aware_object_segmentation .py.md
quantize_accuracy_aware_object_segmentation .sh.md

The attached accuracy aware segmentation python and shell script were run with the following command :
./quantize_accuracy_aware_object_segmentation.sh /home/u01/dlworkbench/6598aac6c9d92e5f3fdc6c48/yolov8/seg/yolov8n-seg.xml /home/u01/dlworkbench/6598aac6c9d92e5f3fdc6c48/yolov8/seg/yolov8n-seg.pt /home/u01/dlworkbench/6598aac6c9d92e5f3fdc6c48/yolov8/seg/coco128-seg.yaml /home/u01/dlworkbench/6598aac6c9d92e5f3fdc6c48/yolov8/seg/datasets/coco128-seg FP32 CPU PERFORMANCE None None 640 640 /home/u01/dlworkbench/66a011d3b17d8e407a74799a yolov8m INT8 0.01 128 300

yolov8n-seg IR and coc128-seg dataset
coco128-seg.zip

Are you going to submit a PR?

• Yes I'd like to help by submitting a PR!

[daniil-lyakhov]

@fmishra747, hello,

I still could not help you because I could not run the script you have filed to me. I don't see following required parameters to reproduce:

MODEL_XML="$1"
MODEL_PT="$2"
YAML_FILE_PATH="$3"
ACCURACY_AWARE_DATASET_PATH="$4"
PRECISION="$5"
TARGET_DEVICE="$6"
PRESET="$7"
IGNORED_SCOPE_NAMES="$8"
IGNORED_SCOPE_TYPE="$9"
TARGET_W="${10}"
TARGET_H="${11}"
OUTPUT_PATH="${12}"
QUANTIZED_MODEL_NAME="${13}"
QUANTIZED_SCHEME="${14}"
MAX_ACCURACY_DROP="${15}"
SUBSET_SIZE="${16}"
RANKING_SUBSET_SIZE="${17}"

Could you please make an isolated reproducer without any parameters so I can run it locally?
Please confirm that it is a NNCF issue as it looks like it is a bug in your internal script

[alexsu52]

@andrey-churkin, I tried to update ultralytics to 8.2.74 for NNCF yolo examples and encountered the same problem. The branch is https://github.com/alexsu52/nncf/tree/as/ultralytics. Please, take a look at it.