is it possible to fine-tune a LLAMA2 70B with 4 A6000s?

[akaikite]

I was wondering if it is possible to train on 4 A6000s with the 4096 context length option.
I think it's probably not possible, but I ask just in case.

[KaiLv69]

I think it worth a try. You can simply use the script here https://github.com/OpenLMLab/collie/blob/dev/examples/profile/memory_optim.py, with just modify the 2048 to 4096 :)
BTW, AdaLomo should outperform LOMO generally. So, if possible, AdaLomo is recommended.

[akaikite]

Thanks for the answer. I'm going to get 4 a6000's and give it a go.