Hello Silviu -



 smu226:

Shouldn’t gradient descent take care of this (my batch size was 512), and make sure I reach the global minima after a few iterations?


No.  In general gradient descent will drive you to the nearest local
minimum, after which you will stay there.  (Putting in big jumps by
hand, using large step sizes (large learning rate), or the randomness
from using a batch (instead of averaging the gradient over the whole
training set) can – but won’t necessarily – take you out of a local
minimum.)


Mainly, my confusion is this: in a normal NN (say for image recognition) one is able to fit a very complex and highly non linear function with (say) 64x64 inputs and several outputs, millions of parameters, a lot of local minima and without much effort (and less training examples) and get very good parameters. In my case, the function is a lot simpler, already written in the desired mathematical form and having just 2 parameters and yet the parameters are sometimes even double than their desired value. I am not sure I understand why does it perform so badly here, compared to a significantly more complex NN.


Let me give you two bits of intuition.  These things might not be true,
but it’s how I think about it.
First, when you train a neural network, you aren’t trying to find the
one, true (with respect to the training set) global minimum.  You’re
looking for one of potentially many similar good / good enough / very
good local minima.  (In fact, in many real-world cases, if you try to
drive your model parameters to “too good” a minimum (global or not)
you encounter over-fitting where your performance on an out-of-sample
test set starts getting worse, even as your performance on you training
set keeps getting better.)
Second, in higher-dimensional parameter spaces there is “more
room” (more directions) to move around, so you can escape from
what would have been a local minimum in a lower-dimensional
parameter space by moving “around” the barrier that would have
kept you in the local minimum by moving in one of the other directions.
(Having made such a bold assertion, I must confess that I find it very
hard to develop intuition about the structure of the loss-function surface
in these very-high-dimensional parameter spaces.)


I also tried using np.sqrt(0.583x1)+np.exp(0.112x2) instead, so no oscillations, and still not getting convergence, using the ranges:
x1 = np.random.rand(10000)*10
x2 = np.random.rand(10000)*2-1



That should work fine.  Take a look at the code that Katherine posted,
and see why hers works and yours doesn’t.
If you can’t get your version working, feel free to post a complete,
runnable script (preferably one-dimensional, just to make things
simpler) that illustrates the problems you are still having.
Good luck.
K. Frank