This error usually occurs when your are running the model without clearing the previous gradients using model.zero_grad(), as the gradients take up memory and get piled up to eat your GPU if not cleared.
While the usual practice is using model.zero_grad() while training, hence this should not occur while training until the very first iteration of the data is costly in terms of memory on your gpu - in that case, try to use torch.cuda.empty_cache() to clear the cache or try to detach any useless tensors using tensor.detach() to not evaluate and store their gradients in your GPU.
While on testing or validation side, we don’t want to create and store gradients for any forward pass so try to run your testing or validation code under the torch.no_grad() context manager.
Hope this helps,
Thanks.