[Bug]: calculate_loss.py error

[tuhaybey]

What happened?

Running calculate_loss.py generates an error.

It appears that modules/module/GenerateLossesModel.py needs to replace "StableDiffusionFineTuneDataLoader" with "StableDiffusionFineTuneVaeDataLoader" in lines 3 and 25. At least making that change- adding the "Vae"- made it work for me.

What did you expect would happen?

It should run :)

Relevant log output

((venv) ) desk@LocalAI:/data/OneTrainer$ python scripts/calculate_loss.py -h
Traceback (most recent call last):
  File "/data/OneTrainer/scripts/calculate_loss.py", line 7, in <module>
    from modules.module.GenerateLossesModel import GenerateLossesModel
  File "/data/OneTrainer/modules/module/GenerateLossesModel.py", line 3, in <module>
    from modules.dataLoader import StableDiffusionFineTuneDataLoader
ImportError: cannot import name 'StableDiffusionFineTuneDataLoader' from 'modules.dataLoader' (unknown location)

Generate and upload debug_report.log

No response

[dxqb]

• Please let me know what you're trying to do, and if you know, what the intention of calculate_loss.py was.
• at first glance, this looks like code that has not been maintained, and it might have been an early version of validation loss? We have validation loss in the UI now
• calling scripts directly is not supported. use run-cmd.sh

[tuhaybey]

It's actually pretty awesome. It's a script you can run to identify the loss with the current model for each image. It creates a json with the loss for each image. You can use it to identify which images the model is struggling with, which was pretty helpful. When I just did it, with the tweak to the module, I found that some of the ones with high loss, it was just struggling with because they're inherently difficult, but a lot of them turned out to be ones where I had bad captions, so fixing those lead to better training results. The script is discussed in the readme in the CLI section, so I think the intent is that it is still in use. I am not aware of another way to know which images it is struggling with.

I think, unless I'm missing something, that the only thing that needs to be changed to get it running again is replacing "StableDiffusionFineTuneDataLoader" with "StableDiffusionFineTuneVaeDataLoader" in lines 3 and 25 of modules/module/GenerateLossesModel.py.

[tuhaybey]

Good tip on using run-cmd.sh, but if you run it using that, you get the same error:

desk@LocalAI:/data/OneTrainer$ ./run-cmd.sh calculate_loss
[OneTrainer] Using Python Venv environment in "venv"...
[OneTrainer] + python scripts/util/version_check.py 3.10 3.13
[OneTrainer] + python scripts/calculate_loss.py
Traceback (most recent call last):
File "/data/OneTrainer/scripts/calculate_loss.py", line 7, in
from modules.module.GenerateLossesModel import GenerateLossesModel
File "/data/OneTrainer/modules/module/GenerateLossesModel.py", line 3, in
from modules.dataLoader import StableDiffusionFineTuneDataLoader
ImportError: cannot import name 'StableDiffusionFineTuneDataLoader' from 'modules.dataLoader' (unknown location)

And with run-cmd.sh, it seems to work correctly if you make the change I noted to GenerateLossesModel.py. At least, it runs without an error- I don't know if the numbers it is generating are correct, or if StableDiffusionFineTuneVaeDataLoader is actually testing something other than was intended.