albanD:

So the only thing that will influence in which direction you should move is going to be x.


Just going one level deeper. Lets say i want to predict y = 2*x and initial weight = 1 which we want to be 2 as we want y_pred = w * x. Lets say every time we give the input value x=2 and learning_rate = 0.1 so the weight change at step (1) according to



 cbd:

w_new = w - learning_rate * dL/(d(w))


is 0.8 (1- 0.1 * 2). Again same input x=2 given so in step (2), w_new = 0.6 (0.8 - 0.12). For the same input x=2, in step (3), w_new = 0.4 (0.6-0.12). Even if you change x to any positive value it is going to decrease and w_new never converge to value 2 and so the network.
It means we should never take ReLU output as our final output ? OR should i missed something?