If you don’t use with torch.no_grad(), Autograd will create the computation graph, which is needed to calculate the gradients in the backward pass.
Now if you store a tensor, which is attached to this computation graph, such as the loss or model output, in e.g. a list, the whole computation graph will also be stored.
To avoid the increased memory usage, you could either wrap the validation code in torch.no_grad() or call tensor.detach() before storing it.