Hi,
I am trying to understand the way torch functions.
I have tried to search the docs, but though torch docs are very descriptive, I couldn’t find a concrete answer.
I will write this question differently, I will write what I have understood, and I will be glad if someone will correct me.
I understood that the way PyTorch and the autograd works is as follows:


The computational graph is being built from the ground up in every .forward() pass.


The .forward() pass is adding the .grad attribute to all the parameters with the .requires_grad attribute set to True (but not attaching any value to it).


Then, when we use .backward(), we are backpropagating through the graph and adding the gradients to the .grad attributes of the parameters with the .requires_grad=True.


If we won’t do .backward(), the ‘.grad’ will be None (even if the .requires_grad is equal to True).


If we do multiple .backward() one after the other (even with no .forward() in between), the .grad attribute for every parameter will be accumulated.


If we do .forward() pass with torch.no_grad(), we are building the graph with .grad of all the parameters, set to None. So .backward() will have no effect.


If we do optimizer.zero_grad() we are setting to 0, the .grad attributes of all the parameters inside the optimizer (or inside the model, if we set the optimizer on all the model parameters).


The optimizer.step() is changing the weights of the parameters based on their .grad attribute.


I’m sure I have a mistake in few of these, maybe in all of them, but I want to understand why the autograd is being called auto… maybe it is calculating the gradients automatically when we do the .forward() pass? if so, why do we need the .backward()? we can do just the optimizer.step() and update the weights based on the gradients that has been calculated by the autograd…


I hope someone will clarify some of this for me…
Thank you,