Solved by colesbury in post #2 
              
              
                In this specific case you do not need retain_graph=True, but in general you may need it. As you compute the forward pass, PyTorch saves variables that will be needed to compute the gradients in the backward pass. For example, z = y * y needs to save the value of y, because dz/dy = 2*y (or y + y). Ho…