As I mentioned, new_tensor = nn.Parameter(old_tensor) breaks the computational graph of autograd.
In other words, new_tensor does not contain any information about the way old_tensor has been computed, which I want.