Skip to content

Issues: EleutherAI/lm-evaluation-harness

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

296 Open 817 Closed
296 Open 817 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

bbh_zeroshot fails during to a custom filter issue. bug Something isn't working.
#2422 opened Oct 23, 2024 by shamanez
vllm mode shows much slow inference speed than normal (hf) mode.
#2418 opened Oct 21, 2024 by 95jinchul
How to evaluation openai model?
#2416 opened Oct 21, 2024 by 9mean2
Clarification Needed on Interface Implementation asking questions For asking for clarification / support on library usage.
#2415 opened Oct 21, 2024 by sorobedio
1
vllm mode has return "[A" asking questions For asking for clarification / support on library usage.
#2414 opened Oct 21, 2024 by 95jinchul
1
AnthropicChat fails when "until" is not provided explicitly in "generation_kwargs" or contains whitespace-only options bug Something isn't working.
#2412 opened Oct 20, 2024 by maaxap
1
Can I use lm-eval for training? asking questions For asking for clarification / support on library usage.
#2411 opened Oct 20, 2024 by yaolu-zjut
3
mgsm tasks not found when using Accelerate bug Something isn't working.
#2405 opened Oct 15, 2024 by Mugariya
1
How to evaluate local model with local-completions? asking questions For asking for clarification / support on library usage.
#2402 opened Oct 14, 2024 by liuzhuotao-teresa
3
Hang when evaluate nemo model with pipeline model parallel
#2399 opened Oct 12, 2024 by zhuango
How to fix the token length of the model input? asking questions For asking for clarification / support on library usage.
#2398 opened Oct 12, 2024 by lonleyodd
7
Stopping Criteria for Openai Models
#2395 opened Oct 11, 2024 by IsraelAbebe
How can I run the mmlu task in offline mode?
#2394 opened Oct 11, 2024 by 95jinchul
How to run MMLU with CoT asking questions For asking for clarification / support on library usage.
#2392 opened Oct 9, 2024 by brando90
1
Tasks not found when using vllm asking questions For asking for clarification / support on library usage.
#2386 opened Oct 8, 2024 by Mugariya
2
Gemma2 BOS token for likelihood_rolling
#2382 opened Oct 4, 2024 by MFajcik
how to reduce verbosity text in versin 0.4.3
#2381 opened Oct 4, 2024 by sorobedio
lm_eval --model vllm did not work when data_parallel_size > 1 bug Something isn't working.
#2379 opened Oct 3, 2024 by wukaixingxp
4
Is LLaMA3.2-Vision-90B/11B result on mmmu_val reproducible? validation For validation of task implementations.
#2377 opened Oct 2, 2024 by jybbjybb
5
regex filter strips whitespace
#2371 opened Oct 1, 2024 by jkamalu
Extracting vLLM metrics
#2365 opened Sep 29, 2024 by vsmolyakov
Which filter value should be used among the accuracy test results?
#2362 opened Sep 27, 2024 by KKwanhee
[multimodal] llava-1.5-7b-hf doesn't work on mmmu_val bug Something isn't working.
#2360 opened Sep 26, 2024 by BabyChouSr
5
Improve docs/model_guide.md with skeleton template code + description of utils like Collator and Reorderer documentation Improvements or additions to documentation. feature request A feature that isn't implemented yet.
#2358 opened Sep 26, 2024 by haileyschoelkopf
Add a test for scripts/write_out.py and other scripts/ utils
#2356 opened Sep 26, 2024 by haileyschoelkopf
ProTip! no:milestone will show everything without a milestone.