Since you can only use a single sample in your forward pass, the running estimates might be off.
You could try to adjust the momentum, but I think nn.InstanceNorm or nn.GroupNorm would be a better alternative.