Clarification on AppWorld metrics computation in ReMe paper: early stopping, avg@4, score definition (after_score vs uplift_score), and binary vs fractional scoring

[nzj1120]

I’m reproducing the results of the ReMe paper on the AppWorld benchmark and have questions about how the reported metrics are computed. In particular, I noticed the evaluation traces include fields like before_score, after_score, uplift_score, and that the runner seems to have early stopping logic (stop when task is completed). These details affect how to compute metrics such as avg@4.

The evaluation code appears to support early stopping . For computing the paper metric avg@4, do we need to run exactly 4 rollouts/trajectories per task even if the task succeeds earlier?

When computing the final metric reported in the paper, should we use after_score or uplift_score?
after_score seems to reflect the post-run outcome (0–1 float).
uplift_score seems to be after_score - before_score .
Which one corresponds to the paper’s main score? If both are used in different places, could you clarify which metric maps to which field?

The score values in traces are floats in [0, 1]. For computing success/accuracy-style metrics, should we treat scores as:
binary success: count only score == 1.0 as success, else failure, or
fractional credit: directly average the float scores ?

[zouyingcao]

1. Yes, when calculating avg@4, you need to collect 4 trajs for each task by setting num_runs=4.
   You mean early stopping which may be related to failure-aware reflection mechanism when num_trials>1.

2. after_score corresponds to the paper’s main score, since it is the actual outcome reward.

3. We directly use the original float reward to calculate the average accuracy. You can refer to run_exp_statistic.py for our paper evaluation method.

[nzj1120]

When reproducing the no-memory setting, I found that many tasks have after_score as a fractional value between 0 and 1. If I directly use these values to compute the metrics, I end up with cases where avg@4 is much larger than pass@4. Have you observed this in your experiments as well? Specifically,we use qwen3-8b without thinking：
avg@4 pass@4
0.5048 0.3452

[zouyingcao]

In our AppWorld experiments, we report Task Goal Completion (TGC) metric (claimed in Appendix A of our paper), which measures percentage of tasks for which the agent passes all evaluation tests. after_score is the percentage of tests passed for per task. To calculate TGC, only after_score=1 means task completion. Therefore, we use threshold=1 in run_exp_statistic.py and the output Pass@1 is just Avg@n (here n=4).

[nzj1120]

so the meaning of Avg@4 is the average of four pass@1?

[zouyingcao]

Yes, Avg@4 is the Pass@1 performance averaged over 4 runs. For simplicity, we organize the total 4 trajs in a single file to calculate Pass@1 and Pass@4 together. Here, the results of Pass@1 and Avg@4 are equivalent.