Solved by David_Harvey in post #4 
              
              
                may i know why Adam need to load the optimizer state while SGD not? 

This is the result of optimizer internal design. Adam has two tracking status m_t and v_t to store historical information, while SGD doesn’t. And that’s why Adam optimizer takes more memory than SGD. 

So, in my case, I used Adam…