Hi, thanks for the reply, but unless I misunderstood something this isn’t what I’m looking for: in the link you provided it is suggested to unscale the optimizer so that we can unscales gradients held by optimizer’s assigned parameters, but those are not the gradient I’m interested in. I’ll try to explaing it better.
In my project I need to use the intermediate gradients of the neurons (NOT the gradient of the loss w.r.t. the weights) and to fetch those I employ some backward hooks.
When I run the backward() pass WITHOUT amp I obtain the following grad tensor tensor([[-0.6644,  0.6644]], device='cuda:0').
When I run the backward() pass WITH amp, instead, I get tensor([[-43552.,  43552.]], device='cuda:0', dtype=torch.float16). Now I’d like to unscale this float16 gradients to the values I get without amp, so that I can still use them in the application. Is this possible?