Inconsistent GPU usage

[alpergel]

Hey @yzslab Im still having the issue of gpu usage spiking and causing non-optimal performance do you have any hypotheses that I can explore to try to fix this?

[yzslab]

Have you met the same issue on graphdeco-inria/gaussian-splatting?

[alpergel]

nope!

[alpergel]

If it helps, I think its wherever the TQDM is looking over the epoch iteration, since whenever it freezes tqdm's curr time and est time are 00:00<00:00

[yzslab]

I really have no idea about it currently, since I can not reproduce it.

Or you can try upgrading the PyTorch and Lightning:

conda create -yn gspl-torch25 python=3.12 pip
conda activate gspl-torch25

pip install -r requirements/pyt251_cu124.txt
# make sure you have installed CUDA 12.4, and the `nvcc -V` prints the right version
pip install -r requirements/lightning25.txt
pip install -r requirements/gsplat.txt
pip install -r requirements/fused-ssim.txt

[yzslab]

Have you tried the simplest command, i.e. without specificing any extra options except the experiment name: python main.py fit --data.path ... -n ...?

[alpergel]

Have you tried the simplest command, i.e. without specificing any extra options except the experiment name: python main.py fit --data.path ... -n ...?

still happening unfortnately

[yzslab]

Try turning off the logger to see if it works:

1. Comment out the line 367-370 here:

   gaussian-splatting-lightning/internal/gaussian_splatting.py

   Lines 367 to 370 in 1aa5e01

   	self.logger.log_metrics(
   	metrics_to_log,
   	step=self.trainer.global_step,
   	)

2. Start training with --logger none

[alpergel]

Yes! That was it!!!

[alpergel]

Its nearly 10 minutes faster! I highly suggest that comment is an option by default, if thats possible

[yzslab]

Normally the logger should not have such a huge overhead.
The bottleneck may be your storage. Check if the wa shown in the top command is high.