Thank you for answering @ptrblck
That would explain a lot of why the model was not improving much… or if at all.
I have a batch size of 16 and am accumulating over 4 batches before passing the gradients to the parameter server for an optimizer step. I can’t increase the batch size anymore due to memory constraints. When I tried InstanceNorm2d, my predictions became nan even with gradient clipping, for some reason that I can’t figure out either. Any insight to why that would be?
How do you change the momentum in all batchnorm layers?