I am just thinking not to create tensor every time when forward is called OR not pass by value during forward call as i don’t want it to backpropagate.
One more question which i don’t understand for ReLU backward function is how loss is backpropagate using chain rule of dL/(d(w))=dL/(d(relu_out))     (d(relu_out))/(d(relu_in))    (d(relu_in))/(d(w))?
As per my understanding 1st term on RHS is “grad_output” which is 1, 2nd term is “grade_input” which is also 1 and 3rd term is “relu_input” lets say it is (w * x) then derivatives become x only.
So from chain rule answer is  1 * 1 * x = x which multiply with learning rate and weight “w”  will change accordingly.
Now the question is if loss value is 0.2 OR it change to 0.8, how it participate to change the weight value? As in both the case passed value of grad_input and grad_output is 1.
Please let me correct if i misunderstood something.