Adds Baseline workflow + fixes #363
Conversation
| """ | ||
| Compute baselines for given tasks. | ||
|
|
||
| It has been tested with generative and accuracy tasks, but may not work correctly for other task types. | ||
|
|
||
| The baseline is computed as follows: | ||
| - For multiple-choice tasks: It assumes random guessing, so the score is n_correct/number_of_choices. | ||
| - For other metrics: It assigns a score of 0, which may not be appropriate for all task types. | ||
|
|
||
| Note: | ||
| This baseline computation may not be suitable for all task types and should be used with caution. |
There was a problem hiding this comment.
It would be way better to update the dummy than add a new main script which will be confusing for users
There was a problem hiding this comment.
But I said it's not possible, I tried.
Unfortunately it's not possible to just use model as there is no info about type of tasks neither about gold_index in requests.
So there is no way one can leverage dummy for proper computation. If you have idea about how do it let me know.
There was a problem hiding this comment.
Let me go through the code again tmr morning, something seemed obvious an hour ago
There was a problem hiding this comment.
Went through things again, I agree, my ideas are way more costly than this.
We should remove the dummy model then as it will always be more costly too.
Wdyt @NathanHB ?
There was a problem hiding this comment.
I'm fine with removing the dummy model, it was only used for fast debugging on task releated issues. Does this baseline implem allow for debugging the logging of tasks ?
There was a problem hiding this comment.
I would keep the dummy model, it's useful when someone just need to passthrough the pipeline workflow. @NathanHB What do you mean by logging of tasks ?
There was a problem hiding this comment.
I meant loading, and I agree it's needed for quick debugging, we should keep it then.
What does it do
Adds a baseline model which allows reliably computing baseline scores for multichoice/generative tasks.
The DummyModel is kind of unreliable as it random based so the baseline can be +-1 from true baseline.
Implementation comments
Unfortunately it's not possible to just use model as there is no info about type of tasks neither about gold_index in requests.
Nits:
Fixes tasks listing/insepct