OK, I see.
Sure, I try to explain it a bit further.
nn.BatchNorm layers are tracking the mean and std of your data by calculating these values from batches and updating their running_mean and running_var, respectively.
The running estimates are updated using the momentum argument as explained in the docs.
The mean and std of a single sample might differ a lot compared to the overall data, so that the running estimates are working like a “smoothing filter” following the sample’s mean and std instead of tracking the dataset stats.
You could try to play around with the momentum to give a little bit less weight to the sample stats and more to the running stats, but generally it has been shown, that BatchNorm performs bad using a small batch size. The GroupNorm paper 1 introduced a new normalization layer, which should perform better using small batch sizes.
However, if you use single instances, I would assume that InstanceNorm would might be a good choice.