Add accuracy_sample_count

[v-shobhit]

In the future, benchmarks (like gpt-oss) may have separate perf and accuracy datasets

This PR adds a separate config field, accuracy_sample_count, to set the number of samples in the acc eval dataset - separate from the existing performance_sample_count which will be used for the size of the perf eval dataset.

This new field defaults to performance_sample_count for backwards compatibility.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[nvzhihanj]

@pgmpablo157321 @tanvi-mlcommons @mrmhodak please help review this PR - the accuracy sample count is something new we add to separate the accuracy and performance test dataset. Can you help review and suggest what else is needed for this feature?

[mrmhodak]

@pgmpablo157321: Please take a look to see if you agree with this.

[arjunsuresh]

@nvzhihanj Can you please confirm if this PR has been tested for a full performance/accuracy run of retinanet where the dataset size is different from the performance_sample_count?

[v-shobhit]

@arjunsuresh the test failures above seem to not be related to the PR: https://github.com/mlcommons/inference/actions/runs/20966840528/job/60259481963?pr=2414

Can you please check?

[v-shobhit]

@arjunsuresh
Checked with retinanet:

Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=133.42s).
Accumulating evaluation results...
DONE (t=28.34s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.37582
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.52478
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.40635
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.02461
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.12698
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.41543
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.41975
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.59758
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.62703
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.08161
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.34103
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.67732
TestScenario.Server qps=134.19, mean=0.6929, time=184.674, acc=41.030%, mAP=37.582%, queries=24781, tiles=50.0:0.7219,80.0:0.8067,90.0:0.8279,95.0:0.8393,99.0:0.8690,99.9:1.4932

[pgmpablo157321]

@v-shobhit LGTM, but can we add this changes to the modularized submission checker as well. I have them in this branch, but I can't add them to shobbit's repository
https://github.com/mlcommons/inference/tree/acc_sample_count

[v-shobhit]

@v-shobhit LGTM, but can we add this changes to the modularized submission checker as well. I have them in this branch, but I can't add them to shobbit's repository https://github.com/mlcommons/inference/tree/acc_sample_count

@pgmpablo157321 is it the commit f81d32a

I will cherry-pick this