Solved by ptrblck in post #4 
              
              
                Thanks for the detailed explanation and I think your “sanity check” makes sense. 
Note that I don’t have a background in numerical mathematics, so please take this post with a grain of salt. 

Yes, this is correct. The optimizer stores references to all parameters and you could inspect the id() of t…