jasam-sheja:

The problem PyTorch won’t allow any modification, period.


I don’t think that’s true, as this code snippet shows that inplace manipulations are allowed after the computation graph was freed and thus no gradients can be (wrongly) calculated anymore:
# setup
lin1 = nn.Linear(1, 1)
lin2 = nn.Linear(1, 1)
x = torch.randn(1, 1)

# disallowed since out1 is needed to calculate the gradients
out1 = lin1(x)
out2 = lin2(out1)
out1[0] += 1
out2.backward()

# manipulate afterwards
out1 = lin1(x)
out2 = lin2(out1)
out2.backward()
out1[0] += 1 # works