Hey,
Will you please clear where am I wrong in the below thought-process?
Assume, I only have one trainable weight parameter to update. So, on calling loss.backward() the gradient of loss wrt that weight has been saved in the node corresponding to the tensor comprising of that weight.
The way I perceive in-place operation is:- "The new value of weight oust the previous value in the ‘same’ node i.e. according to me( unsure if it’s correct ), there is no change in the computational graph other than the “value” update or no new node/edge has been added.
Going by my perception, since except values nothing else has changed, it shouldn’t be illegal.
Please correct the parts where I am wrong.