Panoptic Quality Metrics (PQ): Unexpected PQ value during model evaluation.

[Skywalker83]

I'm using a custom dataset (fine-tuning) for the panoptic task using R101-FPN as COCO Panoptic Segmentation baseline (Panoptic FPN).

Both training and inferencing are able to obtain good result. However, when I perform model evaluation I got the following result:

Panoptic Evaluation Results:

	PQ	SQ	RQ	#categories
All	669.439	785.069	68.472	71
Things	51.593	75.233	66.680	2
Stuff	687.348	805.644	68.524	69

Evaluation metrics:
{
'PQ': np.float64(669.4392318591345),
'SQ': np.float64(785.0691309166493),
'RQ': 68.47162219667933,
'PQ_th': np.float64(51.592946379865104),
'SQ_th': np.float64(75.23326285684394),
'RQ_th': 66.68010678527925,
'PQ_st': np.float64(687.3478198440408),
'SQ_st': np.float64(805.6440836140348),
'RQ_st': 68.52355017961847
}

I'm not sure why SQ_st is in the hundreds range which I believe it causes PQ to be in hundreds range too. Would appreciate help if anyone could point out any possible issues. Thanks.

Instructions To Reproduce the Issue:

Unable to provide data to reproduce the issue.

Source code:

self._cfg.merge_from_file(self._trained_config_path)
self._predictor = DefaultPredictor(self._cfg)
self._initialize_datasets()

evaluator = COCOPanopticEvaluator(
dataset_name=eval_dataset_name,
output_dir=self._output_dir,
)
val_loader = build_detection_test_loader(self._cfg, eval_dataset_name)

results = inference_on_dataset(self._predictor.model, val_loader, evaluator)

Environment:

---

sys.platform linux
Python 3.12.7 (main, Oct 16 2024, 04:37:19) [Clang 18.1.8 ]
numpy 1.26.4
detectron2 0.6 @/pvc-data/workspaces/xxx/xxx/.venv/lib/python3.12/site-packages/detectron2
Compiler clang 18.1.3
CUDA compiler not available
DETECTRON2_ENV_MODULE
PyTorch 2.5.1+cu124 @/pvc-data/workspaces/xxx/xxx/.venv/lib/python3.12/site-packages/torch
PyTorch debug build False
torch._C._GLIBCXX_USE_CXX11_ABI False
GPU available No: torch.cuda.is_available() == False
Pillow 11.0.0
torchvision 0.20.1+cu124 @/pvc-data/workspaces/xxx/xxx/.venv/lib/python3.12/site-packages/torchvision
fvcore 0.1.5.post20221221
iopath 0.1.9
cv2 4.10.0

---

PyTorch built with:

• GCC 9.3
• C++ Version: 201703
• Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• LAPACK is enabled (usually provided by MKL)
• NNPACK is enabled
• CPU capability usage: AVX2
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.4, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.5.1, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,

[github-actions]

You've chosen to report an unexpected problem or bug. Unless you already know the root cause of it, please include details about it by filling the issue template.
The following information is missing: "Instructions To Reproduce the Issue and Full Logs";