Try to wrap your validation loop in with torch.no_grad() as this will avoid storing intermediate tensors, which are required to calculate the gradients during the backward pass.
Currently you are neither deleting loss, which holds a reference to the computation graph and thus all intermediates, nor call loss.backward(), which is correct for the validation loop, but would clear the intermediate tensors.