Hi Katherine!



 crowsonkb:

I implemented this from scratch based on your description and provided partial code samples and it works for me: the loss value decreases steadily over time. Hope it helps:


Thanks for this.  Could you tell us the values you obtained for a and
b after training?
One comment:  From your github code:
x1 = torch.rand(10000) * 10
...
y = torch.cos(0.583 * x1) + torch.exp(0.112 * x2)

x1 will range over [0, 10) and the argument of cos() in x1's
contribution to y will range over [0, 5.83).  That is less than one
full period of cos() ([0, 2 * pi = 6.28)), so you won’t see multiple
oscillations, you won’t see a forest of local minima, and your starting
point for a of 0.7 should be well within the basin of attraction of the
global minimum.
(The gaussian distribution that I used for the plot in my earlier post
and your uniform distribution are not fully comparable, but the standard
deviation of your uniform [0, 10) distribution is 2.89, not too different
from my standard deviation of 5, so the scale of the global minimum’s
basin will be broadly similar in the two cases.)
Best.
K. Frank