You might try to change the momentum of your batch norm layers, so that the running estimates will be smoothed more.
Besides that you could try to trade compute for memory using torch.utils.checkpoint 2.