Hi all,
I found that PyTorch currently does not have a solution to batch normalisation under batch size of 1 and feature size of 1. Example that will cause error:
    batch = 1
    channels = 3
    feature_size = 1
    in_ = torch.ones((batch,channels,feature_size,feature_size))
    norm  = nn.BatchNorm2d(channels)
    y = norm(in_)

Error will be raised (which I know makes sense as there is batch size of 1 has nothing to normalised with). However, there are many potential situation in training where you will load a single image as the last batch. It seems PyTorch has already allow batch size of 1 as long as we increase the feature size, example:
    batch = 1
    channels = 3
    feature_size = 2
    in_ = torch.ones((batch,channels,feature_size,feature_size))
    norm  = nn.BatchNorm2d(channels)
    y = norm(in_)

No error will be raised. Why don’t they do the same for feature_size=1?
If you put norm.eval(), there will be no issue. However, I am asking mainly for training.
Another way around it is to set training DataLoader drop_last=True to prevent this but some user with different GPU memory size will use different batch size and not realise what error is hitting them especially when they are just the user.
Edit: *user of another open-source repository, this is evident in a few pytorch open-source projects open issues.