I use DDP. I enabled anomaly detection. Below is the message I get
/pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:57: UserWarning: Traceback of forward call that caused the error:
File “”, line 1, in 
File “/usr/lib/python3.6/multiprocessing/spawn.py”, line 105, in spawn_main
exitcode = _main(fd)
File “/usr/lib/python3.6/multiprocessing/spawn.py”, line 118, in _main
return self._bootstrap()
File “/usr/lib/python3.6/multiprocessing/process.py”, line 258, in _bootstrap
self.run()
File “/usr/lib/python3.6/multiprocessing/process.py”, line 93, in run
self._target(*self._args, **self._kwargs)
File “/home/beinan/.local/lib/python3.6/site-packages/torch/multiprocessing/spawn.py”, line 19, in _wrap
fn(i, *args)
File “/home/beinan/Desktop/pytorch-bcn/jupyter/train.py”, line 153, in main_worker
train(train_loader, model, criterion, optimizer, epoch, args)
File “/home/beinan/Desktop/pytorch-bcn/jupyter/train.py”, line 199, in train
output = model(images)
File “/home/beinan/.local/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 541, in call
result = self.forward(*input, **kwargs)
File “/home/beinan/.local/lib/python3.6/site-packages/apex/parallel/distributed.py”, line 560, in forward
result = self.module(*inputs, **kwargs)
File “/home/beinan/.local/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 541, in call
result = self.forward(*input, **kwargs)
File “…/bcn/models/semantic/resnet34.py”, line 295, in forward
x, encoder_features, encoder_feature = self.encoder(x)
File “/home/beinan/.local/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 541, in call
result = self.forward(*input, **kwargs)
File “…/bcn/models/semantic/resnet34.py”, line 224, in forward
x = self.bn(self.conv(x))
File “/home/beinan/.local/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 541, in call
result = self.forward(*input, **kwargs)
File “…/bcn/layers/conv.py”, line 38, in forward
groups=self.groups
File “/home/beinan/.local/lib/python3.6/site-packages/apex/amp/wrap.py”, line 28, in wrapper
return orig_fn(*new_args, **kwargs)
Traceback (most recent call last):
File “train.py”, line 314, in 
main()
File “train.py”, line 66, in main
mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, args))
File “/home/beinan/.local/lib/python3.6/site-packages/torch/multiprocessing/spawn.py”, line 171, in spawn
while not spawn_context.join():
File “/home/beinan/.local/lib/python3.6/site-packages/torch/multiprocessing/spawn.py”, line 118, in join
raise Exception(msg)
Exception:
– Process 1 terminated with the following error:
Traceback (most recent call last):
File “/home/beinan/.local/lib/python3.6/site-packages/torch/multiprocessing/spawn.py”, line 19, in _wrap
fn(i, *args)
File “/home/beinan/Desktop/pytorch-bcn/jupyter/train.py”, line 153, in main_worker
train(train_loader, model, criterion, optimizer, epoch, args)
File “/home/beinan/Desktop/pytorch-bcn/jupyter/train.py”, line 208, in train
scaled_loss.backward()
File “/home/beinan/.local/lib/python3.6/site-packages/torch/tensor.py”, line 166, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File “/home/beinan/.local/lib/python3.6/site-packages/torch/autograd/init.py”, line 99, in backward
allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Function ‘CudnnConvolutionBackward’ returned nan values in its 1th output.
This consistently happens after 90+ epochs but only if I use DDP. Single GPU single node does not have this problem. BTW, I train with fp16 precision. Is it possible that fp16 + DDP + SyncBatchNorm somehow leads to this?