Yes I think that I can help you 

Make sure to do the backpropagation. loss.backward()

Don’t hold on to the image, labels, loss or any other GPU tensor in another list. If you want to save e.g. the loss, save the loss.item() so the GPU can free the memory.

If this won’t solve your problem, please post the code of your training loop