You could try to use torch.utils.checkpointing to trade compute for memory.
However, you even if it’s working with a batch size of 2, the running estimates might be quite off due to the small batch size. You might need to tune the momentum or rather switch to another normalization layer, e.g. InstanceNorm.