merge_and_unload docs do not clarify behaviour for quantized base models

[RonanKMcGovern]

System Info

NA

Who can help?

@BenjaminBossan could you add a note to the docs to explain the default behaviour, and also any work arounds (e.g. loading a base model and dequantizing and loading the adapter and then merging) for best performance? Thanks

https://huggingface.co/docs/peft/main/en/package_reference/lora#peft.LoraModel.merge_and_unload

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder
• My own task or dataset (give details below)

Reproduction

NA

Expected behavior

NA

[BenjaminBossan]

I agree that the information is a bit sparse. Could you expand on what exactly you would like to see? What is the workaround for that you mentioned, do you mean quantization methods that don't support merging? What type of performance do you have in mind?

[RonanKMcGovern]

Hi Benjamin. Specifically I'd like to see a recommendation on how to merge adapters with the quantized base model. If you naively merge, then there will be 16 bit adapters merged with an original 16-bit base model. This will introduce error. The correct way to merge is to merge the 16 bit adapters onto a model that has been quantized and then dequantized back to 16 bits. Unsloth allows this kind of merge in one step, and - if it's not directly possible with transformers - it would be worth at least explaining. The workaround, albeit painful, is to save a lora adapter, load the base model freshly in quantized form, dequantize that base model, load the trained adapter and then merge. Does that make sense? Thanks

…

On Fri, Sep 27, 2024 at 10:00 AM Benjamin Bossan ***@***.***> wrote: I agree that the information is a bit sparse. Could you expand on what exactly you would like to see? What is the workaround for that you mentioned, do you mean quantization methods that don't support merging? What type of performance do you have in mind? — Reply to this email directly, view it on GitHub <#2105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASVG6CX2CQJWJOWXPXSLOPLZYUNCXAVCNFSM6AAAAABO5R5FXOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZYG44TONJSG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[BenjaminBossan]

Thanks for the suggestion. To clarify, is this what you're suggesting:

Approach 1 (wrong):

1. I start with a 16 bit model
2. I quantize the model (using bnb) to 4 or 8 bit.
3. I add a LoRA adapter (16 bit)
4. I train the model with LoRA
5. I merge the 16 bit LoRA weights into the 4 bit base model weights

Approach 2 (correct):

1. I start with a 16 bit model
2. I quantize the model (using bnb) to 4 or 8 bit.
3. I add a LoRA adapter (16 bit)
4. I train the model with LoRA
5. I dequantize the base model to 16 bit
6. I merge the 16 bit LoRA weights into the 16 bit base model weights
7. I quantize again to 4/8 bit??

Not sure about step 7, as it would surprise me if approach 2 would actually work better than approach 1 if step 7 is included. If you mean that we omit that step and leave the model in 16 bit, it would be a different story. Of course, this assumes that for inference, we have enough memory to run the model at 16 bit. If we don't, approach 1 still looks reasonable to me.

If you have further resources that show that approach 2 works better, please share them.

Also, from what you wrote, it feels almost more like a feature request to have a method to dequantize + merge, rather than just a docs update, is that right?

[RonanKMcGovern]

Howdy, thanks for laying that out clearly. Yes, the idea is to do Approach 2, without step 7 (agreed that if you want step 7, you already directly have that via the ability to merge and push in 4bit). The broad motivation for all of this is that you lose quite a bit of accuracy when doing Approach 1. Also, vLLM and similar libraries are quite slow working with those nf4 quants (if at all). So, it's a useful feature to be able to merge to a dequantized base and then you have a pretty good quality fp16 or bf16 model. And yes, it could be a feature request, although there's already a dequantize method, so I think it could be sufficient to just show the steps using that in the docs

…

On Mon, Oct 7, 2024 at 12:10 PM Benjamin Bossan ***@***.***> wrote: Thanks for the suggestion. To clarify, is this what you're suggesting: Approach 1 (wrong): 1. I start with a 16 bit model 2. I quantize the model (using bnb) to 4 or 8 bit. 3. I add a LoRA adapter (16 bit) 4. I train the model with LoRA 5. I merge the 16 bit LoRA weights into the 4 bit base model weights Approach 2 (correct): 1. I start with a 16 bit model 2. I quantize the model (using bnb) to 4 or 8 bit. 3. I add a LoRA adapter (16 bit) 4. I train the model with LoRA 5. I dequantize the base model to 16 bit 6. I merge the 16 bit LoRA weights into the 16 bit base model weights 7. I quantize again to 4/8 bit?? Not sure about step 7, as it would surprise me if approach 2 would actually work better than approach 1 if step 7 is included. If you mean that we omit that step and leave the model in 16 bit, it would be a different story. Of course, this assumes that for inference, we have enough memory to run the model at 16 bit. If we don't, approach 1 still looks reasonable to me. If you have further resources that show that approach 2 works better, please share them. Also, from what you wrote, it feels almost more like a feature request to have a method to dequantize + merge, rather than just a docs update, is that right? — Reply to this email directly, view it on GitHub <#2105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASVG6CW4HWZUCW4B2SLXDPLZ2JT3RAVCNFSM6AAAAABO5R5FXOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJWGYZDAOJQGI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[BenjaminBossan]

Okay, got it thanks.

although there's already a dequantize method, so I think it could be sufficient to just show the steps using that in the docs

Are you referring to what you mentioned earlier?

The workaround, albeit painful, is to save a lora adapter, load the base model freshly in quantized form, dequantize that base model, load the trained adapter and then merge.

If you already have some code for this, could you share it? Also, if you have any references that this works better than merging into the quantized weights, that would be greatly appreciated. Of course, even if it's just for speed reasons, it would be good to include in the docs.