Thank you for your explanation and advice. However I found this behavior unexpected. A model may be called several times in one forward pass (and possibly be called by non-user defined functions), and gradients may not always be needed in all calls, e.g., in semi-supervised learning. I think in complicated cases it will be difficult to use separate autocast to handle caches correctly.
Is it possible for the fp16 and fp32 tensors which belong to the “same” model variable to share requires_grad information? Or maybe the no_grad() context can be extended to take care of caches automatically. I think the latter maybe more practical, since there are many cases where caches may cause problems.