Yes, indeed that’s what I thought too, that the construction of the tensor has to happen in the forward pass. However, what I don’t understand yet is where exactly the . grad_fn gets “lost”. You mentioned calling zero_grad() resets the logit_parameter.grad_fn but having a look at the documentation only mentions that zero_grad() sets the gradients to zero and does not touch the .grad_fn.  I guess it’s happening somewhere else?