Allow AdapterModels to have custom tokens

[mapmeld]

PEFT has a feature for adapters to add tokens to a model: https://github.com/huggingface/peft/blob/main/examples/causal_language_modeling/peft_lora_clm_with_additional_tokens.ipynb

When using an AdapterModel with new tokens in LightEval, the script fails because:

• AdapterModel._create_auto_tokenizer always uses the base model path, without checking config.tokenizer
• the new tokenizer and base model will have a mismatched number of embeddings

Notebook with error: https://colab.research.google.com/drive/1AMJ6_MiZGFTBf8KdRn-zj7soKyZrzpbf?usp=sharing

This PR would create the tokenizer from config.tokenizer or config.base_model
and run base.resize_token_embeddings(...) before PeftModel.from_pretrained(base, adapter_weights)
This is based on my fix for llm-evaluation-harness: EleutherAI/lm-evaluation-harness#1828

Notes:

• +1 please merge Fixes bug: You can't create a model without either a list of model_args or a model_config_path when model_config_path was submited. #298 (cherry-picked here), their one-line fix passed model_config_path to AdapterModel and this is necessary for most advanced models
• opinion: I think that adapter_weights and delta_weights should be optional so I do not need delta_weights: false in my config

[clefourrier]

Hi!
That works for me, but I would add a more detailed hlog message at the resize step: if you're adding tokens, we need to know how many, since they are likely to behave weirdly, and if you're removing tokens from the vocab, we would want to know which ones.
So a log like "You're using the adapter model's tokenizer, which has less/more tokens than its base: adding/removing X tokens" + in case of removing specifying which, would be useful.

You'll also need to run ruff to fix the style.

[clefourrier]

Re your comment, adapter_weights and delta_weights should be skippable by default, what's the error message you're getting when not specifying them?

[mapmeld]

I rebased, reformatted, and created a smaller demo to trigger the adding/removing token messages https://colab.research.google.com/drive/1iZhdL-HTuStKGZ3ezspd53eDt_lhXbvm

Unfortunately I can't test due to a Torch/TorchVision compatibility issue. If I install and run lighteval on CoLab I get this error:

RuntimeError: Failed to import transformers.models.bert.modeling_bert because of the following error (look up to see its traceback):
operator torchvision::nms does not exist

After upgrading torch and torchvision, I get:

  File "/content/lighteval/src/lighteval/data.py", line 28, in <module>
    from torch.utils.data.distributed import DistributedSampler, T_co
ImportError: cannot import name 'T_co' from 'torch.utils.data.distributed' (/usr/local/lib/python3.10/dist-packages/torch/utils/data/distributed.py)

[clefourrier]

Hi! Yes, you can't use the latest of pytorch, it's got a breaking change.

[mapmeld]

OK ✔️ I changed how the dependencies are installed and was able to run lighteval on the larger and smaller tokenizers

Updated CoLab, same URL: https://colab.research.google.com/drive/1iZhdL-HTuStKGZ3ezspd53eDt_lhXbvm

At the end of the notebook, I have added the error if I leave out delta_weights: false from the config

File "/usr/local/lib/python3.10/dist-packages/lighteval/models/model_config.py", line 439, in create_model_config
    if config["merged_weights"]["delta_weights"]:
KeyError: 'delta_weights'

I added a commit to this PR to change the line to if config["merged_weights"].get("delta_weights", False): and the same for where it checks adapter_weights