You could try to adapt the momentum so that the stats from each batch have less influence on the running estimates.
Alternatively, you could also use torch.utils.checkpoint to trade compute for memory and try to increase the batch size.