albanD:

That is not true in general, this is actually whatever the grad computed by the other layers is (lower layers).


Let’s say input is given to convolution layer and output of convolution layer is given to ReLU activation function. From output of ReLU, loss is calculated. Assume L1 loss and lets say loss value is 0.2.
So for ReLU, grad_output is coming from L1 loss (lower layer as per your reply i think) and grad_input is given to convolution layer (upper layer) . As per the autograd function of ReLU, “grad_output” (dL/(d(relu_out)) is always 1 (https://github.com/DingKe/pytorch_workplace/blob/master/basic/functions.py ) as ReLU function is f(x) =x and L1_loss = F(x) - target. So derivative with respect to relu_out which is F(x) becomes 1. That is why i am thinking “grad_output” pass to backward function of ReLU is 1.



 albanD:


“grade_input” which is also 1

Is again that depends on the other layers (the ones above you this time).


As per the backward function of ReLU, return value of function is “grade_input” and is always 1.
If output of convolution convolution layer is w*x which is input ot relu_in then (d(relu_in))/(d(w)) will be x.
So dL/(d(w))=dL/(d(relu_out)) (d(relu_out))/(d(relu_in)) (d(relu_in))/(d(w)) = 1 *1 *x=x.
New weight updation will be, w_new = w - learning_rate *  dL/(d(w))
So the question is, how loss has contributed to change in weight? I hope i clarified my understanding. Please let me know what i misunderstood if i am wrong.