You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello, thank you for maintaining this project! Super valuable. Wanted to make one small issue.
The "regex" filter removes whitespace from the response, which is unnecessary. It should be up to the user to provide a regex that includes or removes whitespace. Alternatively, if the user wants to remove whitespace, they can just use the "remove_whitespace" filter on top of the "regex" filter, but as it stands, it's impossible to use the regex filter and maintain the leading whitespace, which is important for code generation tasks like humaneval, which, because it's Python, needs the leading whitespace.
Hello, thank you for maintaining this project! Super valuable. Wanted to make one small issue.
The "regex" filter removes whitespace from the response, which is unnecessary. It should be up to the user to provide a regex that includes or removes whitespace. Alternatively, if the user wants to remove whitespace, they can just use the "remove_whitespace" filter on top of the "regex" filter, but as it stands, it's impossible to use the regex filter and maintain the leading whitespace, which is important for code generation tasks like humaneval, which, because it's Python, needs the leading whitespace.
lm-evaluation-harness/lm_eval/filters/extraction.py
Line 41 in 15ffb0d
@thnkinbtfly @haileyschoelkopf -- tagging you both because you're on the blame for this line
The text was updated successfully, but these errors were encountered: