Thank you a lot for your reply! I understand what you mean, however I am still a bit confused. Shouldn’t gradient descent take care of this (my batch size was 512), and make sure I reach the global minima after a few iterations? Mainly, my confusion is this: in a normal NN (say for image recognition) one is able to fit a very complex and highly non linear function with (say) 64x64 inputs and several outputs, millions of parameters, a lot of local minima and without much effort (and less training examples) and get very good parameters. In my case, the function is a lot simpler, already written in the desired mathematical form and having just 2 parameters and yet the parameters are sometimes even double than their desired value. I am not sure I understand why does it perform so badly here, compared to a significantly more complex NN.
I also tried using np.sqrt(0.583x1)+np.exp(0.112x2) instead, so no oscillations, and still not getting convergence, using the ranges:
x1 = np.random.rand(10000)*10
x2 = np.random.rand(10000)*2-1