Solved by KFrank in post #3 
              
              
                Hello Brandon! 

This isn’t true.  All common optimization algorithms I’m aware 
of – and in particular, gradient descent – only care about the 
gradient of the loss, and not the loss itself. 
Plain-vanilla gradient descent takes the following optimization 
step: 
new weights  =  old weights  -  lea…