Try to decrease the batch size and make sure no tensor which are attached to the computation graph are stored in each iteration.
Also, you could try to use torch.utils.checkpoint to trade compute for memory.