Hi @ptrblck ,
I am not very familiar on what will hold the pytorch codebase back from just literally having batch normalisation do nothing (no trainable parameter) when batch_size=1 and feature_map_size=1 for training.
I think users will be unaware of the problem as the training will work most of the time and then suddenly fail (ex: DataLoader loads one final image for training).  This is also painful for users trying to train on a big dataset as they have to go through a lot of iterations just to get hit by the batch normalisation error caused by the final data batch being 1 (assuming in the networks, there is a module that pools to a feature size=1).