tlim:

It seems PyTorch has already allow batch size of 1 as long as we increase the feature size, example:


Because a larger feature size would allow the batchnorm layer to calculate the stddev from these values, while a batch size and feature dimension of 1 would have an undefined (or invalid) stddev.
How would you calculate the stddev in the case of a single value? Would you rather return NaNs in the forward pass?