Hi all,
I am confused here FP32 parameters are stored indeed as the master copy, but why not the gradients appear as FP16 to store and transfer to reduce the load as Nvidia Apex Philosophy details? Is there any tradeoff between the implementation and design of torch autocast?
Any feedback will be appreciated!