LORA on T5 model

[vivektreddy]

I am trying to use LORA on a loaded Checkpoint of a CodeT5 model. However when I do, the run time is about the same, and my result is not as good as when I finetune the whole thing. Am I intializing the model properly?

rank=16
lora_alpha=4
lora_dropout=0.05
model = T5ForConditionalGeneration.from_pretrained("Salesforce/codet5-small")
#freeze parameters
for name, param in model.named_parameters():
param.requires_grad = False
lora_config = LoraConfig(inference_mode=False, r=rank, target_modules=['q', 'v'], lora_alpha=lora_alpha, lora_dropout=lora_dropout)
lora_model = LoraModel(model, lora_config, "default")

Thank you