Loaded weights with error during training

[t626644510]

Self Checks

• This template is only for bug reports. For questions, please visit Discussions.
• I have thoroughly reviewed the project documentation (installation, training, inference) but couldn't find information to solve my problem. English 中文 日本語 Portuguese (Brazil)
• I have searched for existing issues, including closed ones. Search issues
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template and fill in all required fields.

Cloud or Self Hosted

Self Hosted (Source)

Environment Details

WSL2, python3.10, all envs build just follow the introduction from document.

Steps to Reproduce

The strange warning displayed every time when running the train.py(or webui from manage.py) either 1.5 or 1.4.3 here.
I'm nor sure is it common situation during train process and some friends said that's definitely abnormal.
Fig1 from 1.5 and Fig2 from 1.4.3.

✔️ Expected Behavior

everthing goes well

❌ Actual Behavior

The model file can be created (normally?) and merged.
But the train speed is quite slow and the warning will be displayed again when merging.
Furthermore, the accuracy from Top5 hardly increase slightly and the inferenced voice by this strange model is also weird.

[t626644510]

some issue said that's the problem from GPU but is there anyway to avoid or bypass it? How did this problem arise?QAQ

[ZiyangSong0230]

I encountered the same issue. May I ask if you have resolved it? You mentioned above that it might be a GPU-related problem. Could you please share what GPU you are using? I am currently using an A10 and facing this issue.

[yw0nam]

I got same issue while fine tuning..
Here is the training log.

train.log

[Whale-Dolphin]

It's normal, our 1.5 model does not have lora weight, but it exists in model_dict, so if you use our model directly to train, it will show this warning, it's fine and will not cause any crash.

[Whale-Dolphin]

And once you merge lora weight the warning will disappear.