-
Notifications
You must be signed in to change notification settings - Fork 291
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test: Add model load test below n param threshold #1698
base: main
Are you sure you want to change the base?
Conversation
I think we could create a file to record whether a model load was successful, to avoid repeated loading attempts. |
Right now the main issue is disk running out of space. Looking into how not to save the weights into cache folder when loading a model. |
Also, test should skip API based models |
scripts/failures.json
Outdated
"colbert-ir/colbertv2.0": "To use the ColBERT models `pylate` is required. Please install it with `pip install mteb[pylate]`.", | ||
"google/flan-t5-large": null, | ||
"intfloat/e5-mistral-7b-instruct": "Over threshold. Not tested.", | ||
"intfloat/multilingual-e5-large-instruct": "XLMRobertaModel.__init__() got an unexpected keyword argument 'cache_folder'", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems that cache_folder=CACHE_FOLDER
breaks loading. Maybe use default HF cache folder with scan_cache_dir().delete_revisions
? https://huggingface.co/docs/huggingface_hub/en/guides/manage-cache#clean-cache-from-python
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will give it a try.
Fixes #1690
mteb.get_model
and output the exception (None or raw text)Checklist
make test
.make lint
.