Addison:

From my understanding, every time you do a forward pass with a torch model (and you are not in a torch.no_grad() block), then gradients will accumulate for the weight tensors of the model, so that when you go to do the backward pass, the gradients are already there and just need to be multiplied.


That’s not quite right. The forward pass will store the intermediate activations, which would be needed to compute the gradients in the backward pass. The backward pass will then accumulate the gradients to each used parameter.
Using with torch.no_grad() will not store the intermediate activations and will thus save memory.
And yes, you are right that no_grad() can and is also used during the validation run (not only when deploying the model) as seen e.g. in the ImageNet example 16. Here you can see that the model is set to eval() first and the validation loop is executed in the no_grad context.