qLoRA+fsdp llama 400b

[xgal]

👋
Tried to follow @winglian guide https://medium.com/@winglian/qlora-finetuning-llama-3-1-405b-with-axolotl-with-256gb-system-ram-c2474a3d3fa5

I used the specific commit on HF to load weights to cpu only on rank0 and tried to build bnb from source from the fork version and from the official gh repo (which I saw the commit was already merged to)
also tried to load with the latest docker image of axolotl but I a problem with dtypes (ValueError: Cannot flatten integer dtype tensors)
I suspected its something related to bnb or peft but can't make it work :\

would really appreciate the help !
thanks !
Gal

File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/distributed/fsdp/_flat_param.py", line 768, in _validate_tensors_to_flatten
[rank0]:     raise ValueError("Cannot flatten integer dtype tensors")
[rank0]: ValueError: Cannot flatten integer dtype tensors

[NanoCode012]

Could you see if retrying now fixes the issue? If not, I can try to followup this with wing.

[xgal]

Works fine now
Thanks