Solved by tom in post #2 
              
              
                So I covered some of that along with many “little mysteries” in a course recently (slides 28-35). From the summary slide: 
There are two main situations where inplace operations cannot be used: 

When operating on leaf tensors, the leaf tensor would be moved into the graph, which would be bad.
When …