I would like to do a hyper-parameter search so I trained and evaluated with all of the combinations of parameters.
But watching nvidia-smi memory-usage, I found that GPU-memory usage value slightly increased each after a hyper-parameter trial and after several times of trials, finally I got out of memory error. I think it is due to cuda memory caching in no longer use Tensor. I know torch.cuda.empty_cache but it needs do del valuable beforehand. In my case, I couldn’t locate memory consuming variable.
What is the best way to release the GPU memory cache?