I know saved tensors should not be modified because they could be used multiple times.
However, when we save the output result of some layer, once the backward pass reaches that layer, there should not be any other layer that’d use it (right?) because the backward pass has already gone through them before. i.e., the output of A used in later B, C, etc., layers, so B, C, etc., should not modify it in place. Only A should be able to.
The problem PyTorch won’t allow any modification, period.
Why is this useful? We could reuse that Tensor for other calculations and save precious memory by doing it in place.