Hello Brandon!



 ljj7975:

Based on my understanding of back prop and gradient descent,
Loss is multiplied to gradient when taking a step with gradient descent.
So when gradient becomes negative, gradient descent takes a step in the opposite direction.


This isn’t true.  All common optimization algorithms I’m aware
of – and in particular, gradient descent – only care about the
gradient of the loss, and not the loss itself.
Plain-vanilla gradient descent takes the following optimization
step:
new weights  =  old weights  -  learning rate * gradient
Could you have misread “learning rate” for “loss” at some point?


Such idea is well captured when implementing gradient ascent,
as it can simply be implemented by multiplying -1 to the loss.
Then, what happens if my loss starts from positive and goes below zero?
If what I said above is correct, training will go for minimization when the loss is positive and go for maximization when the loss is negative.
And this means that gradient descent is going for zero instead of minimum value…
Can anyone help me understand this concept?


Gradient descent (and, again, all common algorithms that I am
aware of) seek to minimize the loss, and don’t care whether that
minimum value is a large positive value, a value close to zero,
exactly zero, or a large negative value.  It simply seeks to drive
the loss to a smaller (that is, algebraically more negative) value.
You could replace your loss with
modified loss = conventional loss - 2 * Pi
and you should get the exact same training results and model
performance (except that all values of your loss will be shifted
down by 2 * Pi).
It is the case that we often use loss functions that become equal
to zero when the fit of the model to the training data is perfect,
but the optimization algorithms don’t care about this, and they
drive the loss function to algebraically more negative values,
and not towards zero.
Good luck!
K. Frank