Hi Brandon!



 ljj7975:

Thank you Frank.
Oh I thought, loss affects the gradient descent directly.
So it simply provides the search space and training a model purely depends on learning rate, not the loss
flipping the sign of loss turns the problem into maximization because within the search space everything is flipped (min point becoming max point).
Is this correct?


If I understand what you are saying, yes that is  correct.
But, just to be sure, you could try the following exercise:
Set up a simple neural network and train it using gradient
descent on some well-behaved data set.  Make a plot of
your training loss (and, while you’re at it, your training
accuracy) as a function of batch number or epoch number,
and make sure that it is training stably and that you’re getting
reasonable results.
Use a standard loss function when you do this.  Let’s call this
loss-original.
Let’s say that your loss runs from 1.0 down to 0.1 when you
train.
Now define both:
   loss-shifted = loss-original - 1.5
   loss-negative = -loss-original

and train your neural network again using these two modified
loss functions and make your loss and accuracy plot for each
of these two modified training runs.  See if you get the results
you expect, and, if not, post what you got and ask any questions
that you might have.
Best.
K. Frank