Let me clarify. The ReLU function is  f(x)=max(0,x) . It means if  x<=0  then  f(x)=0 , else  f(x)=x . In the first case, when  x<0 , the derivative of  f(x)  with respect to  x  is  f'(x)=0 . So, we perform  grad_input[input < 0] = 0 . In other words, we "compute the gradient of the   output  with respect to the input” and not "compute the gradient of the loss with respect to the input”.