Simple Gradient Descent Project plausibility I am currently in a numerical analysis class at my university and wanted to tackle a project applying gradient descent. Fair warning: I am new to machine learning, but my professor believed in me, so I had trouble saying no. Please bare with me and correct me wherever possible. I really would appreciate some guidance from you guys!
The idea they gave me was to estimate a family of functions, specifically cubics by applying gradient descent on the loss function which will be least squares. I can start with some cubic so I know the outputs/labels.
So from my understanding our hypothesis is $h(x) = \sum_{k=0}^{3}\alpha_kx^k$?
Our loss function is $L(h(x),f(x))=(h(x_i)-f(x_i))^2$. My professor said it would be best to use the emperical error $L(\theta) = \dfrac{1}{n}\sum_{i=0}^{n}L(h(x_i),f(x_i))$ where $\theta = [\alpha_k]_{k=0}^{3}$ and we apply the gradient descent algorithm to find $\theta^\star=\arg\min L(\theta)$.
So I believe the idea is,

*

*Have some known cubic with some gaussian noise.

*Initiate a cubic with random weights.

*Calculate the gradients of the least squares for each coefficient/weight.

*Apply gradient descent to $L(\theta)$ and find the optimal weights?

Question 1: In general cubics are not convex. I need to explain why the method works and most of assumptions rely on having a convex function. Or am I misunderstanding something?
Question 2: Why is the emperical error method a good idea here?
Question/comment 3: Least squares is tripping me up, from my perspective it is no different than MSE, as it seems very similar?

The idea you outlined in bullet points 1-4 is correct. Although a cubic is not convex in general, it wouldn't matter even if this was a 100-degree polynomial because this is actually a linear regression, and with mean squared error (MSE) as the loss function, it's a convex problem.
Empirical error is the error you calculate using the data you have, so when learning from data, I don't see another option here.
As wikipedia says:

The method of least squares is a standard approach in regression
analysis to approximate the solution of overdetermined systems (sets
of equations in which there are more equations than unknowns) by
minimizing the sum of the squares of the residuals made in the results
of every single equation.

Least squares is the name of the method, but MSE is the loss function used.
