cbd:

Lets say i want to predict y = 2*x and initial weight = 1 which we want to be 2 as we want y_pred = w * x. Lets say every time we give the input value x=2 and learning_rate = 0.1


I’m sorry I can’t really follow this.
What would be the pair (x, expected_y) that you want?
What would be the initial weights w0?

It means we should never take ReLU output as our final output ?

It is not advised in general no. Mainly because it does not have any benefit (if you’re value is positive, it will just be an identity) and has the drawback of potentially getting your training stuck if the value becomes negative (as you will get 0 gradient and nothing will move anymore).

OR should i missed something?

Be careful here that for the l1 loss, the gradient can be 1 or -1 depending on the sign of the difference! That will make sure you don’t keep going in one direction forever.