I got a response on my open issue on Github: Increased memory usage with AMP · Issue #61173 · pytorch/pytorch · GitHub 11
I guess basically, because the model parameters are still kept in fp32, we don’t see reduction in memory usage while using large models. The reduction comes from not keeping the activations in fp32, so maybe try a smaller model with larger batch size.