If you are not wrapping the validation forward pass into with torch.no_grad(), the computation graph will be created, which would be needed to calculate the gradients. Unless you are calling backward() on the output or loss, no gradients will be calculated and the model will thus not be changed. As long as you hold a valid reference to the model output the computation graph will be kept alive.