BeastyZ/e5-R-mistral-7b fails

[Muennighoff]

2025-01-01 04:53:12.929644: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-01-01 04:53:12.942516: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-01-01 04:53:12.946128: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
INFO:mteb.cli:Running with parameters: Namespace(model='BeastyZ/e5-R-mistral-7b', task_types=None, categories=None, tasks=['STSBenchmark'], languages=None, benchmarks=None, device=None, output_folder='/data/niklas/results/results', verbosity=2, co2_tracker=True, eval_splits=None, model_revision=None, batch_size=32, overwrite=False, save_predictions=False, func=<function run at 0x7fdf8686cf70>)
WARNING:mteb.model_meta:Loader not specified for model BeastyZ/e5-R-mistral-7b, loading using sentence transformers.
WARNING:sentence_transformers.SentenceTransformer:No sentence-transformers model found with name BeastyZ/e5-R-mistral-7b. Creating a new one with mean pooling.

Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards:  17%|█▋        | 1/6 [00:00<00:01,  3.47it/s]
Loading checkpoint shards:  33%|███▎      | 2/6 [00:00<00:01,  3.59it/s]
Loading checkpoint shards:  50%|█████     | 3/6 [00:00<00:00,  3.65it/s]
Loading checkpoint shards:  67%|██████▋   | 4/6 [00:01<00:00,  3.71it/s]
Loading checkpoint shards:  83%|████████▎ | 5/6 [00:01<00:00,  3.75it/s]
Loading checkpoint shards: 100%|██████████| 6/6 [00:01<00:00,  3.78it/s]
Loading checkpoint shards: 100%|██████████| 6/6 [00:01<00:00,  3.72it/s]
INFO:mteb.evaluation.MTEB:

## Evaluating 1 tasks:
─────────────────────────────── Selected tasks  ────────────────────────────────
STS
    - STSBenchmark, s2s


INFO:mteb.evaluation.MTEB:

********************** Evaluating STSBenchmark **********************
No config specified, defaulting to the single config: stsbenchmark-sts/default
INFO:datasets.builder:No config specified, defaulting to the single config: stsbenchmark-sts/default
Loading Dataset Infos from /env/lib/conda/gritkto/lib/python3.10/site-packages/datasets/packaged_modules/json
INFO:datasets.info:Loading Dataset Infos from /env/lib/conda/gritkto/lib/python3.10/site-packages/datasets/packaged_modules/json
Overwrite dataset info from restored data version if exists.
INFO:datasets.builder:Overwrite dataset info from restored data version if exists.
Loading Dataset info from /data/huggingface/datasets/mteb___stsbenchmark-sts/default/0.0.0/b0fddb56ed78048fa8b90373c8a3cfc37b684831
INFO:datasets.info:Loading Dataset info from /data/huggingface/datasets/mteb___stsbenchmark-sts/default/0.0.0/b0fddb56ed78048fa8b90373c8a3cfc37b684831
Found cached dataset stsbenchmark-sts (/data/huggingface/datasets/mteb___stsbenchmark-sts/default/0.0.0/b0fddb56ed78048fa8b90373c8a3cfc37b684831)
INFO:datasets.builder:Found cached dataset stsbenchmark-sts (/data/huggingface/datasets/mteb___stsbenchmark-sts/default/0.0.0/b0fddb56ed78048fa8b90373c8a3cfc37b684831)
Loading Dataset info from /data/huggingface/datasets/mteb___stsbenchmark-sts/default/0.0.0/b0fddb56ed78048fa8b90373c8a3cfc37b684831
INFO:datasets.info:Loading Dataset info from /data/huggingface/datasets/mteb___stsbenchmark-sts/default/0.0.0/b0fddb56ed78048fa8b90373c8a3cfc37b684831
INFO:mteb.abstasks.AbsTask:
Task: STSBenchmark, split: test, subset: default. Running...
INFO:mteb.models.sentence_transformer_wrapper:No model prompts found for task=STSBenchmark prompt_type=None
INFO:mteb.models.sentence_transformer_wrapper:Encoding 1379 sentences.
ERROR:mteb.evaluation.MTEB:Error while evaluating STSBenchmark: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.
Traceback (most recent call last):
  File "/env/lib/conda/gritkto/bin/mteb", line 8, in <module>
    sys.exit(main())
  File "/data/niklas/mteb/mteb/cli.py", line 387, in main
    args.func(args)
  File "/data/niklas/mteb/mteb/cli.py", line 145, in run
    eval.run(
  File "/data/niklas/mteb/mteb/evaluation/MTEB.py", line 623, in run
    raise e
  File "/data/niklas/mteb/mteb/evaluation/MTEB.py", line 562, in run
    results, tick, tock = self._run_eval(
  File "/data/niklas/mteb/mteb/evaluation/MTEB.py", line 304, in _run_eval
    results = task.evaluate(
  File "/data/niklas/mteb/mteb/abstasks/AbsTask.py", line 126, in evaluate
    scores[hf_subset] = self._evaluate_subset(
  File "/data/niklas/mteb/mteb/abstasks/AbsTaskSTS.py", line 88, in _evaluate_subset
    scores = evaluator(model, encode_kwargs=encode_kwargs)
  File "/data/niklas/mteb/mteb/evaluation/evaluators/STSEvaluator.py", line 47, in __call__
    embeddings1 = model.encode(
  File "/data/niklas/mteb/mteb/models/sentence_transformer_wrapper.py", line 105, in encode
    embeddings = self.model.encode(
  File "/env/lib/conda/gritkto/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 589, in encode
    features = self.tokenize(sentences_batch)
  File "/env/lib/conda/gritkto/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 1044, in tokenize
    return self._first_module().tokenize(texts)
  File "/env/lib/conda/gritkto/lib/python3.10/site-packages/sentence_transformers/models/Transformer.py", line 398, in tokenize
    self.tokenizer(
  File "/env/lib/conda/gritkto/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 3021, in __call__
    encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
  File "/env/lib/conda/gritkto/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 3109, in _call_one
    return self.batch_encode_plus(
  File "/env/lib/conda/gritkto/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 3302, in batch_encode_plus
    padding_strategy, truncation_strategy, max_length, kwargs = self._get_padding_truncation_strategies(
  File "/env/lib/conda/gritkto/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2923, in _get_padding_truncation_strategies
    raise ValueError(
ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.

[Samoed]

I think it would be better to add model metadata and use the instruct_wrapper for Mistral-based models, similar to e5-mistral https://github.com/embeddings-benchmark/mteb/blob/main/mteb/models/e5_instruct.py. This would also allow for additional fixes to be applied.