Hi all,
I am trying to implement mixed precision training in pytorch, which would involve scaling the loss (computed in half precision) with a scaling factor and compute the gradient, then update the master model (in one precision) with the computed gradient divided by the same scale factor.
I know that I can extract the gradient by:
print(model.layer.weight.grad)
However, are there any ways that I could manually manipulate such gradient (say division/multiplication) and apply it to update another model for mixed precision training?
Many thanks for your help!