I’ve seen posts (such as this 3 one) where people discuss using no_grad() purely for the purpose of saving memory during inference.
However I was wondering if it makes sense to use it during training? In most setups, including mine, a researcher will train their model for 100 epochs say, and every 10 epochs they’ll do a forward pass on validation data to report loss and/or accuracy. From my understanding, every time you do a forward pass with a torch model (and you are not in a torch.no_grad() block), then gradients will accumulate for the weight tensors of the model, so that when you go to do the backward pass, the gradients are already there and just need to be multiplied.
If this is the case, then doesn’t this mean it is a good idea to use torch.no_grad() whenever we test our validation loss/accuracy? Otherwise, won’t the model’s gradients be impacted by validation data, meaning validation data impacts training?
I’m guessing I’m misunderstanding something about how torch.no_grad() works / how gradients are really computed / stored.
Any clarification would be great!