Hi, sir, torch.cuda.empty_cache() is really help. Recently, I also came across this problem. Normally, the tasks need 1G GPU memory and then steadily went up to 5G. If torch.cuda.empty_cache() was not called, the GPU memory usage would keep 5G. However, after calling this function, the GPU usage decrease to 1-2 G.
I am training an RL project with PyTorch 0.4.1. So, here I am still confused and cannot find reason. I used TF before and there is no such issue.