I had the same problem before. After every batch in the training and testing adding
torch.cuda.empty_cache()
worked for me. However it is always better to watch the gpu stats. You can use gpustat 15. For example if you run the code and dont restart the kernel for the next run, some data from the previous run stuck in GPU memory.