Solved by ptrblck in post #7 
              
              
                Thanks for the detailed analysis! 
From the plots it seems that the allocated memory stays approx. at the same level. 
Could you try the native amp implementation in PyTorch 1.6.0 and check the memory usage? 
Also, which GPU are you using?