I have narrowed down the problem to the BatchNorm layers. Additionally I plotted the running_mean and running_var properties, and they seem fine (no NaNs for example.).
My guess is that it is related to the way I am sampling while training. I am using triplet loss, which means the same model is used 3 times on three batches, each containing samples of a single label.
anchor, positive, negative = model(input1), model(input2), model(input3)
loss = F.triplet_margin_loss(anchor, positive, negative) 

Taking MNIST as an example, this means the mini-batch input1 at each iteration contains randomly picked samples from a single label (for example only 9’s). input2 contains the other samples from the same label as input1. Finally, input3 contains samples from another label, for example only contains 4's.
Will mini-batches like these cause problems with BatchNorm properties? Should I switch to another sampling strategy?