google/gemma-2-27b-it QLORA

[CyberNativeAI]

Hi everyone,

I have tried QLORA fine-tuning gemma-2-27b with chatml and flash-attention yesterday and the result model seems confused, even tho the loss seemed to go down and overall everything looked smooth. May someone please share tips&tricks for fine-tuning this model?

[NanoCode012]

Hey, sorry for late response, since it's a qlora, the impact on the weights are not as large as lora / full fine-tuning. Have you tried those other options?

[NanoCode012]

Hey, I'll close this discussion for now but if you're still having this issue, please re-open / create a new one.

[CyberNativeAI]

It worked out eventually, had to play with parameters. Please close.

[NanoCode012]

Thanks for updating us on this!