Thanks Frank, I did the exercise. It behaves as I expected.
Training is fine and generates exactly same accuracy for  loss-shifted  (even though loss is < 0).
For  loss-negative , training fails, the graph say that loss decreases but since the sign is flipped, conceptually it is increasing the loss by applying gradient ascent.
I actually have another question about loss.
From our previous discussion, it is clear that value of loss itself does not mean anything.
what actually matter is the gradient with respect to input and their direction (sign).
I am in the situation where I have to define a new loss function that is not differentiable.
Based on my understanding, (though it is not going to be easy) it is possible to train a model with non-differentiable loss function, if I define a custom Backward function that returns “fake” gradient with appropriate direction and magnitude.
Is my assumption correct?