Sorry, I have made some wrong expressions in the question. I mean the gradients back propagated from the two losses seems to be simply accumulated. Can the Scaler tells the two set of gradients apart, and unscales each of them properly for param. updating?