bPangolin:

my predictions became nan even with gradient clipping, for some reason that I can’t figure out either. Any insight to why that would be?


No idea at the moment and you could use torch.autograd.set_detect_anomaly(True) at the beginning of the script to get a stack trace, which should hopefully point to the first NaN creation.



 bPangolin:

How do you change the momentum in all batchnorm layers?


momentum is an argument in all batchnorm layers, which you can directly pass to them:
nn.BatchNorm2d(num_features=3, momentum=0.1)