Clarification: SEA “ensemble labelling” difference in the paper and the code

[Shrey-Singhal]

Hi OAT/SEA authors. I wanted a quick clarification about SEA mixed preference learning / model_rollout vs the paper.

From the SEA (Sample Efficient Alignment) paper, my understanding is:

After burn-in, you still select the duel pair using SEA’s selection rule (E&E or BAI), then you mix oracle-labeled vs ensemble-member pseudo-labeled preferences via mixture ratio, where the ERM label is produced by a randomly sampled ensemble member without querying the oracle for those samples.

In the released code, I’m a bit confused on these points:

• Label source used for training when is_model_data=True:
  In the model_rollout path, I see oracle.compare(...) still being called and binary_feedback computed. For is_model_data=True samples, are the preference labels used for training taken from:

  • the pseudo-label (sampled RM/ensemble member), or
  • the oracle.compare result?

• Pair selection for model-labeled samples:
  ModelBasedExplorer._random_select() samples an ensemble head and selects the pair with max |reward gap| under that head. This seems different from the Algorithm 2 in the SEA paper (which suggests the same SEA duel selection rule, with only the label source being mixed). Is this max-margin pair selection intended, and is it what was used for the reported SEA results?

• Budget accounting
  In preference.py the oracle is called for every prompt (even is_model_data=True), but in dap.py the counter only adds not is_model_data (query_step += sum(not is_model_data)). Does the ‘oracle query budget’ in the plots count only labels actually used for training, or all oracle calls? I’m trying to understand whether model‑data comparisons are excluded from the budget by design.

Thanks!

[lkevinzc]

Hey @Shrey-Singhal , thank you for your interest in our work!

1/ the preference labels will be taken from pseudo-labels, and the oracle calling is just to get that information to measure the pseudo-labels' accuracy, etc,

2/ out of all pseudo-labels data, we want to take those with largest reward margin (so that they are most confident ones). yes it's different from Algo 2 Line 8 and it's intended and used for the experiment.

3/ model-data is excluded so that we can draw conclusion about the sample efficiency. the oracle calling is mainly for metrics as mentioned in 1/.

Thanks again for your questions!

[Shrey-Singhal]

@lkevinzc Thanks for your response - I really appreciate it. I had one more question - In the paper, the BAI objective (line 7) uses variance of σ(rφ(x, y_t) − rφ(x, b)), but in code EnnBAITS uses uncertainty.logits_variance (pre‑sigmoid). Is that an intentional simplification, or should it use probabilities_variance/bernoulli_variance instead?

[lkevinzc]

Hi @Shrey-Singhal , I think you are right that the implementation did not closely follow the equation. I think both pre-sigmoid and post-sigmoid variances can serve the purpose of BAI (finding the one with largest relative uncertainty), but post-sigmoid is generally more sensitive to logits that are near the decision boundary (zero) and less sensitive to differences between logits that the model is already very certain about. Based on such intuition I chose do use the simpler one. But I feel the other option is also valid and perhaps it's interesting to compare them!