TypeError: LoraConfig.__init__() got an unexpected keyword argument 'exclude_modules'

[imrankh46]

System Info

Name: peft
Version: 0.13.2

Who can help?

when i try to load the adapter for inference its showing the following error.

TypeError: LoraConfig.__init__() got an unexpected keyword argument 'exclude_modules'

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder
• My own task or dataset (give details below)

Reproduction

!pip install -q transformers
!pip install -q bitsandbytes peft
!pip install -q hf_transfer

from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer
import torch

model = AutoPeftModelForCausalLM.from_pretrained('FINGU-AI/Qwen2.5-7b-lora-e-6').to("cuda")
tokenizer = AutoTokenizer.from_pretrained('FINGU-AI/Qwen2.5-7b-lora-e-8')

Expected behavior

TypeError: LoraConfig.__init__() got an unexpected keyword argument 'exclude_modules'

[JINO-ROHIT]

this is a little weird since exclude_modules shouldnt exist in v13.2

[JINO-ROHIT]

I tried to load another model and it seemed to work fine

model = AutoPeftModelForCausalLM.from_pretrained('ybelkada/opt-350m-lora').to("cuda")

did you happen to train the lora by installing peft from source? the 'exclude_modules' is only available when you install peft from the main branch, its not there is 13.2 yet, can you double check once?

[imrankh46]

@JINO-ROHIT what do you mean?
i just include this in the lora config

  peft_config = LoraConfig(
            lora_alpha=32,
            lora_dropout=0,
            r=32,
            bias="none",
            task_type="CAUSAL_LM",
            target_modules= ['k_proj', 'q_proj', 'v_proj', 'o_proj', "gate_proj", "down_proj", "up_proj"],
            modules_to_save = ['lm_head', 'embed_tokens']
    )

the issues i think here. it 'embed_token' not 'embed_tokens' right?

[JINO-ROHIT]

have you tried to pass exclude_modules anywhere during finetuning lora with source peft installation?

[imrankh46]

@JINO-ROHIT not install from the source

just remove this part # modules_to_save = ['lm_head', 'embed_token'] . its not working with qwen model

[JINO-ROHIT]

removing that part works?

[imrankh46]

@JINO-ROHIT yes

[JINO-ROHIT]

ahh okay

[imrankh46]

but qwen model showing 'embed_tokens' not 'embed_token'

[JINO-ROHIT]

not quite sure what you mean here, mind sharing the whole code?

[imrankh46]

@JINO-ROHIT is there any way to handle and save this part for each model?

# modules_to_save = ['lm_head', 'embed_token']

[JINO-ROHIT]

im not quite following , are you having trouble using the finetuned lora adapter during inference? or are you hacing trouble during finetuning?

[imrankh46]

talking about this

from transformers import AutoModelForCausalLM
model_name = "Qwen/Qwen2.5-7B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
              model_name, torch_dtype=torch.bfloat16, attn_implementation='flash_attention_2'
    )
model
Qwen2ForCausalLM(
  (model): Qwen2Model(
    (embed_tokens): Embedding(152064, 3584)
    (layers): ModuleList(
      (0-27): 28 x Qwen2DecoderLayer(
        (self_attn): Qwen2FlashAttention2(
          (q_proj): Linear(in_features=3584, out_features=3584, bias=True)
          (k_proj): Linear(in_features=3584, out_features=512, bias=True)
          (v_proj): Linear(in_features=3584, out_features=512, bias=True)
          (o_proj): Linear(in_features=3584, out_features=3584, bias=False)
          (rotary_emb): Qwen2RotaryEmbedding()
        )
        (mlp): Qwen2MLP(
          (gate_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (up_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (down_proj): Linear(in_features=18944, out_features=3584, bias=False)
          (act_fn): SiLU()
        )
        (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
        (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
      )
    )
    (norm): Qwen2RMSNorm((3584,), eps=1e-06)
    (rotary_emb): Qwen2RotaryEmbedding()
  )
  (lm_head): Linear(in_features=3584, out_features=152064, bias=False)
)

and check this

  peft_config = LoraConfig(
            lora_alpha=32,
            lora_dropout=0,
            r=32,
            bias="none",
            task_type="CAUSAL_LM",
            target_modules= ['k_proj', 'q_proj', 'v_proj', 'o_proj', "gate_proj", "down_proj", "up_proj"],
            modules_to_save = ['lm_head', 'embed_tokens']
    )

so these two are different

'embed_tokens' and 'embed_token'`

[imrankh46]

facing issues with inference. getting this error.
TypeError: LoraConfig.__init__() got an unexpected keyword argument 'exclude_modules'

[JINO-ROHIT]

can you try installing from source and see if that fixes the issue?

pip install git+https://github.com/huggingface/peft

[imrankh46]

let me try

[imrankh46]

@JINO-ROHIT now facing this issues

from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer
import torch

model = AutoPeftModelForCausalLM.from_pretrained("FINGU-AI/Qwen2.5-7b-lora-e-6").to("cuda")
tokenizer = AutoTokenizer.from_pretrained("FINGU-AI/Qwen2.5-7b-lora-e-6")

RuntimeError: Error(s) in loading state_dict for PeftModelForCausalLM:
	size mismatch for base_model.model.model.embed_tokens.modules_to_save.default.weight: copying a param with shape torch.Size([152064, 3584]) from checkpoint, the shape in current model is torch.Size([151665, 3584]).
	size mismatch for base_model.model.lm_head.modules_to_save.default.weight: copying a param with shape torch.Size([152064, 3584]) from checkpoint, the shape in current model is torch.Size([151665, 3584]).

[JINO-ROHIT]

can you make sure you did training and inference using the same version?

[imrankh46]

yeah. but the issues is related to this
modules_to_save = ['lm_head', 'embed_tokens']

am i right?

[imrankh46]

if i remove this line the fine tune lora is working with peft for inference

[JINO-ROHIT]

im really unsure at this point, sorry give me sometime ill try and debug this

[imrankh46]

ok sure. but try qwen2.5 small model with 10 step.

[BenjaminBossan]

TypeError: LoraConfig.init() got an unexpected keyword argument 'exclude_modules'

This error occurs because the LoRA adapter that you're trying to load was trained with a from source install of PEFT, which already includes the feature:

https://huggingface.co/FINGU-AI/Qwen2.5-7b-lora-e-6/blob/main/adapter_config.json#L6

You have 2 options: 1. You upgrade your PEFT version to be from source too, as @JINO-ROHIT suggested, or you just edit the adapter_config.json and remove the line I quoted. In the future, PEFT will allow configs to be forward compatible, which will avoid this error.

RuntimeError: Error(s) in loading state_dict for PeftModelForCausalLM:
size mismatch for base_model.model.model.embed_tokens.modules_to_save.default.weight: copying a param with shape torch.Size([152064, 3584]) from checkpoint, the shape in current model is torch.Size([151665, 3584]).
size mismatch for base_model.model.lm_head.modules_to_save.default.weight: copying a param with shape torch.Size([152064, 3584]) from checkpoint, the shape in current model is torch.Size([151665, 3584]).

This error most likely occurs because the checkpoint was trained with a resized vocabulary, probably to add more special tokens. This is why we need modules_to_save, as they save the new embedding tokens/LM head, you can't omit this argument.

[imrankh46]

@BenjaminBossan that module to save not working for Qwen2.5 model

[BenjaminBossan]

that module to save not working for Qwen2.5 model

I don't understand why you think so. Do you get an error specifically related to modules_to_save? If yes, what is it exactly?

the issues i think here. it 'embed_token' not 'embed_tokens' right?

No, embed_tokens is right, as can be seen here. The checkpoint also names it correctly. You may have to resize the embedding weights after loading the base model, as it seems that the LoRA authors did that too. The sizes must match, otherwise you'll get an error when loading.

i just include this in the lora config

I'm not really sure why you define a LoRA config. When loading the adapter, that step is not necessary.