Exactly what I did, it turns out, that, the overall loss is decreasing, while there is always one of them increasing, positive loss or negative loss, whichever. I’m doubting the gradient calculated in this way, e.g., sums positive and negative loss all together, will be opposed to the minimization direction, as one of the positive and negative loss will go higher.