Thanks for the detailed analysis!
From the plots it seems that the allocated memory stays approx. at the same level.
Could you try the native amp implementation 9 in PyTorch 1.6.0 and check the memory usage?
Also, which GPU are you using?