@ptrblck That is confusing to me.
What is the difference between torch.no_grad() and to wrap the line with .requires_grad_(False) -> .requires_grad_(True)?
That seems equivalent to each other.
With torch.no_grad() I get a runntime exception since there is no gradient:
def forward(self, x):
    out1 = self.linear(x)  # backprop gradients and adjust weights here
    with torch.no_grad():
        out2 = self.linear(out1)  # only backprop gradients here
    return out2


RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

By wrapping it with .requires_grad_(False) -> .requires_grad_(True) it seems to run through but I don’t get why. For me that looks identical to the first solution.
def forward(self, x):
    out1 = self.linear(x)  # backprop gradients and adjust weights here
    self.linear.requires_grad_(False)
    out2 = self.linear(out1)  # only backprop gradients here
    self.linear.requires_grad_(True)
    return out2

I have the same confusion with .detach. If I would detach it, then how should the gradient be backpropagated?
Maybe the confusion is on the terminology here but I would suspect no_grad, requires_grad(False) and detach to block gradient propagation. Maybe a feature request for a no_weight_adjustment / without_weight_adjustment parameter or environment makes sense?
How should the detach solution work? Could you give an example or add a link to a source where something similar is described?