Hi Martijn!



 Martijn:

I’m trying to create a model that is able to learn any lower polynomial function. (Can this even be done?)


Yes*!
*)  For some definition of “lower polynomial.”


So far I’ve got code that is able to learn the y=6x^2 + 2x - 4 quadratic equation.
I noticed that by providing [x, x^2] instead of [x] as the input the model performs much better.


As Alex noted, by providing x**2 as input, you are giving your
model very valuable raw material with which to construct your
quadratic polynomial.
In fact, including x**2 lets you train an almost trivial model that
consists of a single Linear layer (and no activation functions):
class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = Linear(2, 1)

    def forward(self, x):
        x = self.fc1(x)
        return x

The Linear (2, 1) layer (with bias), is, in fact, exactly the general
quadratic polynomial when fed with [x, x**2] as its input, so this
model will train to essentially machine precision, and the weight
and bias of the Linear will train to match the coefficients of your
our_function() quadratic polynomial.


However, when I use [x, x^2, x^3] as the input the model doesn’t work anymore (huge test set loss). Why does providing x^3 to the model result in such a poor result?


First, adding x**3 just confuses the issue, making the training less
nimble.  But more importantly, as Alex noted, x**3 is, in your case,
very large.
Given the range of your sample data, x is of order one thousand,
and x**2 is of order one million, while x**3 is of order one billion.
As training starts, the network has no idea what the three inputs
“mean” and has no sense of their relative scales.  The x**3 term
has the largest effect on the output initially.  So the network has to
spend many training iterations learning to ignore x**3 (or to process
it into some approximation to a quadratic), which wastes valuable
time, and potentially sends the training off on some wild goose chase
from which it never recovers.


Below is the code that I’m using. I’m quite new to Pytorch so general tips/critique is also appriciated.
        self.criterion = MSELoss()
...
            print(f"Epoch: {epoch} Loss: {loss / data_size:,.10f}")



A minor comment:  MSELoss (using its defaults) takes the mean
of the loss over the batch, so loss / data_size is incorrectly
dividing by the batch size a second time.
In terms of understanding your loss, x is of order one thousand,
so your quadratic (that is, y_train) is of order one million.  If your
errors (that is, y_pred - y_train) were comparable to y_train
(that is, your relative errors were of order 1), then your errors and
squared errors would be of order one million and one trillion,
respectively.  So your loss (MSELoss means “mean-squared-error”)
will be of order one trillion (10**12) as training starts.  Just be aware
that even if your training works well and your loss falls a lot, it will
still be numerically large.
You should probably train on MSELoss, but maybe also print out
sqrt (MSELoss) – the so-call root-mean-squared (rms) error.
Even this will be large, starting out on the order of one million.  So
I would suggest also tracking the relative rms error, something like
sqrt (MSELoss / (y_train**2).mean()).  This gives you a sense of
how well your predictions are doing relative to the scale of the values
your are trying to predict.



 googlebot:

Generally, a neural net fits a piecewise function of x, and there are no x*x terms in a network.


I’d like to expand on this a little.  It is true that Linear layers introduce
no quadratic terms.  Also the ReLU activation function is piecewise
linear.  But many non-linear activation functions do have quadratic (and
higher-order) terms in their expansions, so they do introduce quadratic
terms into the overall function computed by the network.
(For example, pytorch’s ELU (“exponential-linear unit”) has a regime
where the quadratic term dominates.)


Thus, adding x^2 input is beneficial.


My quibble about activation functions introducing quadratic terms
notwithstanding, explicitly adding x**2 is, of course, beneficial in a
case like yours because the network doesn’t have to reproduce it by
digging it out of non-linearities in the activation functions.
Best.
K. Frank