Based on my understanding of back prop and gradient descent,
Loss is multiplied to gradient when taking a step with gradient descent.
So when gradient becomes negative, gradient descent takes a step in the opposite direction.
Such idea is well captured when implementing gradient ascent,
as it can simply be implemented by multiplying -1 to the loss.
Then, what happens if my loss starts from positive and goes below zero?
If what I said above is correct, training will go for minimization when the loss is positive and go for maximization when the loss is negative.
And this means that gradient descent is going for zero instead of minimum value…
Can anyone help me understand this concept?