Hello Silviu



 smu226:

I have 10000 tuples of numbers (x1,x2,y) generated from the equation: y = np.cos(0.583*x1)+np.exp(0.112*x2). I want to use a NN like approach in pytorch to find the 2 parameters i.e. 0.583 and 0.112 using SGD.
…
In theory it should work easily, but the loss doesn’t go down. What am I doing wrong?


I think you are trying to solve a problem that is hard to solve with
gradient descent.
I don’t see any obvious errors in your code.  (I looked at it briefly,
but not in detail.)  So I don’t think that you’re doing anything “wrong.”
Because you add your x1 and x2 terms together, your problem
decouples into to solving for the two parameters independently.
So let us look at just the cos() piece.
The oscillatory nature of cos() means that your loss function will
likely have several local minima in which the gradient-descent
minimization algorithm can get “stuck.”  Whether this happens will
depend on the range and distribution of the x1 you use (which you
didn’t tell us).
To illustrate this I run two examples:  First I sample 10,000 values of
x from a gaussian distribution with a mean of zero and a standard
deviation of 5.0, and then again with a standard deviation of 500.0,
and calculate y = np.cos (0.583 * x).
I then try to fit 0.583 with trial values of alpha running from 0.0 to 2.0
by calculating y_alpha = np.cos (alpha * x), and calculating the
mean-squared-error of the y_alpha with respect to the “ground-truth”
y.
Here is a plot of the mean-squared-error vs alpha for both values
of the standard deviation:

As you can see in both cases the loss function takes on a global
minimum of 0.0 for alpha = 0.583.  So far, so good.  But for
stdev = 500, the global minimum is hiding in a tiny range of alpha
surrounded by lots of shallow local minima.  Unless you get lucky
and start very close to 0.583 (and for stdev = 500, your starting
value of 0.7 is not close), nothing about your gradient will tell you
that a deep global minimum exists, and your gradient-descent
algorithm won’t converge to the global minimum.
On the other hand (for the ensemble of values of x that I’ve generated
for this example) for stdev = 5, your starting value of 0.7 is well within
the basin of the global minimum, and gradient descent should converge
nicely to the global minimum (for adequately small values of the learning
rate).  However, if you were to start out above about 1.25 (or jump up
above 1.25 because of an overly-large learning rate), you could get
stuck in the local minimum around alpha = 1.5) (or one of the local
minima that occur for still larger values of alpha).
(The batch size you use in your training loop – which you didn’t tell
us –  will also affect how “noisy” your loss.backward() gradients
are and how much your optimizer jumps around, but this doesn’t
change the essence of what is going on.)
Note, exp() doesn’t have the oscillatory behavior that is the root
cause of the local minima.
So try doing a one-dimensional problem where you just have
exp (0.112 * x), and when you get that working, try a
two-dimensional problem where you replace cos() with exp(),
i.e., y = exp (0.583 * x1) + exp (0.112 * x2).  That should
be an “easy” minimization problem, so please follow up with further
questions if you can’t get that to work.
Good luck.
K. Frank