@ptrblck I have another system in which I can use a batch size of max 8. But I read on a blog that BatchNorm works best with batch_size 32. Would you recommend using InstanceNorm2D or GroupNorm instead of BatchNorm2D?
Edit:
After reading the GroupNorm paper it is clear that GorupNorm combined with Weight Standardization should be able to theoretically outperform BatchNorm for smaller batch sizes. What do you think?