Training speed is very slow

[1179021477]

@RogerChern I train tridentNet_1x with resnet50 on 4 GPU (a machine with 8 GPU), and I need 2 days. Especially, when others use other left GPUs in my machines, the speed of training my models is slower. Is there any way to make training faster? Like how to construct multi-thread, etc. My machine is TITAN X (Pascal).

[xchani]

Our dataloader does use multi-threading to load images.
According to your description, you are sharing gpu server with others, then jobs from others may occupy cpu resource in that server, which slow down your training. Also the speed of disk(IOPS) is another major factor should be considered.