So do you suggest I use the method you gave above for all such operations? I don’t see why the operation being in place makes a huge difference. As a n example
x = torch.rand(1, in, requires_grad=False)
x = nn.Linear(in, out)(x)
loss = x[0,0]
loss.backward()

Here we would get no errors even though x is needed to compute gradients of the layer parameters. The only difference is presumably that nn.Linear doesn’t do inplace modifications, but that seems like an almost irrelevant point. Wouldnt nn.ReLU run into the whole exact problem above since its needs to update each element of a tensor-based on its current value.