You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
target_for_fewshot_sorting: Optional[str] =None# will probably have to be removed in the future
I suggest keeping target_for_fewshot_sorting, but for another purpose. To let the user determine the criteria by which fewshot examples are sorted and picked in fewshot manager.
My concrete example is in fewshot structured generation in which model is to generate function call json representations. I want to pick the examples, in a balanced setting, according to the name of the functions but currently they're picked according to the whole representation of the function call which doesn't make sense as all of them are different.
The text was updated successfully, but these errors were encountered:
sadra-barikbin
changed the title
[FT] To keep or to remove target_for_fewshot_sorting
[FT] To keep target_for_fewshot_sorting for another purpose
Jul 28, 2024
Hi there!
In the code it's stated that
target_for_fewshot_sorting
might be removed in the future, maybe becauseline["__few_shots"]
is introduced.lighteval/src/lighteval/tasks/requests.py
Line 172 in 506abda
I suggest keeping
target_for_fewshot_sorting
, but for another purpose. To let the user determine the criteria by which fewshot examples are sorted and picked in fewshot manager.My concrete example is in fewshot structured generation in which model is to generate function call json representations. I want to pick the examples, in a
balanced
setting, according to the name of the functions but currently they're picked according to the whole representation of the function call which doesn't make sense as all of them are different.The text was updated successfully, but these errors were encountered: