Hello there,
There are a few things that you can do to lower your memory footprint. First is to run your validation code with torch.no_grad() so to not save any gradients. Are you by any chance running your validation code inside your training loop? If so, there might be a few tensors that you could delete from the training loop, perhaps del training_input, del training_output, del .... Lastly, I found that putting this once before training lowers my memory footprint but I don’t know it’s inner workings.
torch.backends.cudnn.benchmark = True # Optimizes cudnn
I don’t know about the cache though, good luck