Confusion in choice of best model

[hnes94]

Description

The current implementation of the Trainer class in Neuromancer evaluates and selects the best model based on the "dev_loss" metric by default. However, this evaluation is only done on the last batch of the validation set, which seems counterintuitive and potentially incorrect. It appears more logical to use "mean_dev_loss" to evaluate the model performance across the entire validation set, rather than just the last batch.

Problem

The selection of the best model based on "dev_loss" instead of "mean_dev_loss" could lead to misleading results. Evaluating on a single batch does not provide a comprehensive view of the model's performance and might select suboptimal models.

Suggested Improvement

Modify the default behavior to use "mean_dev_loss" for evaluating the model. This would ensure that the evaluation metric considers the performance across the entire validation set, providing a more accurate representation of the model's effectiveness.

Additional Note

While it is possible to specify "mean_dev_loss" as the evaluation metric when instantiating the Trainer class, this behaviour is not immediately clear from the example notebooks or documentation. Making "mean_dev_loss" the default metric would improve intuitiveness and reduce confusion for users.

Note: I reformulated my original discussion with ChatGPT to make it more linguistically sound. Please correct me if I'm mistaken or misunderstanding something.

Cheers!

[RBirmiwal]

Hello @hnes94

Thank you for your interest in our library and for this point! We will definitely take it into consideration!

[hnes94]

So after all, is it as currently implemented desired behavior or not?

And if so why, am I missing something here?

I am just wondering if I'm doing something wrong/understanding something wrong here entirely and would be very grateful for clarification.

Thank you very much!