Saikumar_Tadi:

it seems to initially fall down and then it stays constant at not so low value without any change


I think your 1/Y.size() term is incorrect, you’re overemphasizing L2 penalty.



 Saikumar_Tadi:

what do you mean by epoch 0 prediction is too far off


e.g. if true_y = x * 100 + b, but your w initialization range is like -3…3 (and you don’t model bias at all). Accelerating optimizers help here, but that may be not enough for harder problems and mini-batches.