Actually, do you mind spelling out one detail that is particularly confusing to me? When we loop and do SGD, when exactly are new computation graphs created? I think thats the part that is confusing to me and even made me post the following question:

  
    
    
    Why is the gradient update not accidentally included as part of the computation graph? 
  
  
    I was trying to understand how pytorch decides what operation and which not to include in the computation graph that eventually does backprop. 
For the simplest example I could think, consider a (maybe ugly) manual implementation of SGD: 
for i in range(nb_iter):
    # Forward pass: compute predicted Y using operations on Variables
    batch_xs, batch_ys = get_batch(X_train,Y_train,M,dtype) # [M, D], [M, 1]
    ## FORWARD PASS
    y_pred = mdl.forward(batch_xs)
    ## LOSS
    batch_loss = (1/M)…
  


however it seems that the answer to is is just “acting on mdl.W.data does not add things to the computation graph”. Then when is the computational graph formed?