Can unlimiformer work with common fine-tuning methods？

[mrlzh]

I set max_source_length with 10240 and training t5 models，but it ran out of CUDA memory.
I would like to know if unlimiformer can run together with fine-tuning methods such as LoRA.

[abertsch72]

Hi @mrlzh , thanks for your interest!

We haven't tried using Unlimiformer with LoRA, but there isn't a theoretical reason that they wouldn't work together. If you try it, please let us know how it goes!