When I am doing gradient accumulation, the BatchNorm2d layers are not properly accumulated, right? Though, I don’t entirely understand exactly what is going on. The running mean and std deviation are reset on each consecutive batch, correct?
If the BatchNorm2d layers are not properly accumulated, does that also disrupt the gradient accumulation in the accompanying Conv2d layers?
From what I’ve found online the “solution” is to use InstanceNorm2d. I used InstanceNorm2d layers, but my model would not improve and drives predictions to nan even with clamping gradients between (-1,1).
I’m using these layers in a custom UNet model creating segmentation masks.
Does anyone have any insight to this?