nnUNetv2_train additional flags

[jamesgoldfarb]

Fabian, Great job!!!!!

I would find these flag very useful.

#1 --no_val flag. The final validation with (nnUNetv2_train) is causing a crash of all gpu processes. It would be great to have a --no_val in addition to --val.
#2 A flag to change the checkpoint saving from 50 epochs to a smaller number.
#3 --num_epochs to change the number of epochs from 1000

[FabianIsensee]

agree with the no val and checkpointing interval. will do these now. Num epochs will not happen because this breaks the nnunet system. It owuld not longer be clear how many epochs were used to train what model when looking at the results. You need to create separate classes for that

[FabianIsensee]

Why is validation causing a crash with multiple GPUs? Can you elaborate?

[jamesgoldfarb]

I can't be 100% sure. Prob memory, or gpu memory or disk space as you discuss....
It's weird since the terminal also closes.
I come back the next morning and the terminal is gone, the log shows that only a few validation inferences were performed.

[FabianIsensee]

could be an out of memory issue then

[jamesgoldfarb]

Could be is an understatement..... It's a system memory problem... 256Gb mem. Still running out and having a process die.
Sorry, I realize that you have discussed this before.

My solution is to hard code/reduce num_process

line 1067 nnUNetTrainer.py

    with multiprocessing.get_context("spawn").Pool(6) as segmentation_export_pool:

Is it possible to set this via command line?
Seems to be set from nnunetv2.utilities.default_n_proc_DA import get_allowed_n_proc_DA