I’m not sure 100% on the phasing of the statement but I assume you mean ‘forwarding’ as in calling a particular Module/Layer within your model? If this is true, then a particular layer within your network that doesn’t take any input data and just applies arithmetic to intermediate layers might not have a grad_fn (depending on the arithmetic at play) in the case of no grad_fn it’d be skipped by the optimizer step method (as you said)