zhi_Li:

It seems that the two losses that scaled by different scaling parameters are just simply accumulated.


The losses are not accumulated, as the backward call is used on each of the losses separately.



 zhi_Li:

Can the scaler unscales the two set of gradients with their corresponding scaling parameters when it comes to parameter updating?


You can unscale the gradients implicitly using:
# You can choose which optimizers receive explicit unscaling, if you
# want to inspect or modify the gradients of the params they own.
scaler.unscale_(optimizer0)