Thank you for your answers !
I just tried to do x.data += 1 and this modifies x.data in-place.
I am surprised to see that x.data += 1 is not equivalent to x.data = x.data + 1.
But I guess that is inherent to Python and not Pytorch as seen here 9
The default behavior in Python is always referencing, it’s just that x += 1 modifies in-place and  x = x + 1 doesn’t.
So I was wrong in my question, saying that

default behavior when we do prev_x = x.data is referencing and not copying, like in Numpy