I came across two different behaviors of tensor updating, and I don’t understand why they give different results, and in which case, one or the other is meant to be used.
Case 1 :
>>> a = torch.Tensor([1.0,2.5,-2.0])
>>> x = Variable(a, requires_grad=True)
>>> prev_x = x.data

>>> x
Variable containing:
 1.0000
 2.5000
-2.0000
[torch.FloatTensor of size 3]

>>> prev_x
Variable containing:
 1.0000
 2.5000
-2.0000
[torch.FloatTensor of size 3]

>>> x.data.sub_(1)
 0.0000
 1.5000
-3.0000
[torch.FloatTensor of size 3]

>>> prev_x
 0.0000
 1.5000
-3.0000
[torch.FloatTensor of size 3]

Case 2 :
>>> a = torch.Tensor([1.0,2.5,-2.0])
>>> x = Variable(a, requires_grad=True)
>>> prev_x = x.data

>>> x
Variable containing:
 1.0000
 2.5000
-2.0000
[torch.FloatTensor of size 3]

>>> prev_x
Variable containing:
 1.0000
 2.5000
-2.0000
[torch.FloatTensor of size 3]

>>> x.data = x.data - 1

>>> x
Variable containing:
 0.0000
 1.5000
-3.0000
[torch.FloatTensor of size 3]

>>> prev_x
 1.0000
 2.5000
-2.0000
[torch.FloatTensor of size 3]


First of all, I understand that in order to make a copy of a tensor, I should write prev_x = x.data.clone().
But I do not understand why the default behavior when we do prev_x = x.data is referencing and not copying, l̶i̶k̶e̶ ̶i̶n̶ ̶N̶u̶m̶p̶y̶ ̶(̶A̶F̶A̶I̶U̶,̶ prev_x  ̶a̶n̶d̶ x.data ̶a̶r̶e̶ ̶b̶o̶t̶h̶ ̶t̶e̶n̶s̶o̶r̶s̶)̶.̶ (EDIT See my answer below)
Second, in case 1, the .sub_() method indicates with the “_” suffix that it is done in-place.
It is natural then that the x is changed, and since prev_x = x.data is referencing, prev_x is changed too.
But in case 2, I update directly x.data, with something I would consider “in-place” : x.data = x.data - 1, but this time, prev_x doesn’t change.
Could someone explain that behavior ?
I know that we shouldn’t use in-place methods (“Supporting in-place operations in autograd is a hard matter, and we discourage their use in most cases”), but I don’t want to use them, I found it in the update step of the Adadelta optimizer 3, and that is what lead me to stumble on this.
Another thing that is unclear to me, is when to use the different notations for basic operations, meaning:

Actual symbols +,-,x,/ e.g. a = a + b

Not in-place operators add(),sub() e.g. a = a.add(b)

In-place operators add_(),sub_() e.g. a.add_(b)


Thank you !