Hi,
I’m using DDP to train on multiple GPUs and I’m using a shared server. I sometimes get errors where one GPU process will run out of memory for the dataloader and error out, but the other GPUs will keep going. Eventually, this results in the entire job freezing, but not erroring out (and resulting in large server costs without any progress being made). Is there any way to specify with DDP to kill all GPUs if one fails?
Thanks!