There might be some small issues in your code:

Why are you using retain_graph=True in loss.backward()? As your code is quite long, I couldn’t check it properly, but this might be a source of growing memory. Do you get an error if you don’t specify it?
Currently you are re-initializing the optimizer in each epoch. All internal estimates of Adam will be lost, which result in bad training.

Variables are deprecated since PyTorch 0.4.0. If you are using a newer version, you can just use tensors instead.
The loss tensor will be accumulated in total_loss, which will increase the memory usage, as the computation graph is attached to each loss tensor. If you just need total_loss for printing/debugging purposes, you should store it as total_loss += loss.item()


I think repackage_hidden should work fine.
Let me know, if these points help somehow.