Solved by albanD in post #2 
              
              
                Hi, 
The short answer is: 
Pytorch is actually built in such a way that the overhead of the graph creation is low enough (compared to all the computations you do in the forward) that you can do it at every iteration. 
The long answer is: 
This is true for neural network and when each op is quite “la…