Hi Rohit!



 Rohit_R:

batch = torch.zeros(3, 3, 256, 256, dtype=torch.uint8)



Batch is a single “holistic” tensor of shape [3, 3, 256, 256].  It is
not a “collection” (in the sense of, say, a python list or dictionary) of
three 3x256x256 tensors.


batch[0] = image



Pytorch is doing some moderately fancy stuff with python here.
This is not simply assigning image to the 0 element of the batch
“collection.”  I don’t understand the python details, but this line of
code, roughly speaking, calls something like:
batch.modify_tensor_slice (0, image)



          id(batch[0]) == id(image)
          out: False

Should this not be true as both hold references to the same Tensor object ‘image’?


No, batch[0] and image are two different tensors.
In this case (unlike in the assignment batch[0] = image) you should
understand batch[0] as calling something like:
batch.return_slice_as_new_tensor (0)

Pytorch tensors are fancy objects that can do a lot of things.  In
this case, batch[0] is indeed a new tensor object, but it is a
“view,” so to speak, into another tensor, batch.  But even though
the tensor batch[0] and the tensor batch share some of the
same underlying data, they are two distinct tensor objects.


          id(batch.storage()) == id(image.storage())
          out: True

Since both ‘batch’ and ‘image’ share the same underlying storage, why does the change in batch[0][0][0][0] not reflect when I print the corresponding element of ‘image’ ?


This is a confusing fake-out.  batch and image do not share the
same storage.
I don’t know the actual details of what is going on, but I deduce that
some_tensor.storage() returns a new “storage” object that wraps
(for whatever reason) the actual underlying storage.
Consider this:
>>> import torch
>>> torch.__version__
'1.7.1'
>>> batch = torch.tensor ([[1, 2], [3, 4]])
>>> image = torch.tensor ([10, 20])
>>> id (batch.storage())
139730615547904
>>> id (image.storage())
139730615547328
>>> id (batch.storage()) == id (image.storage())
True
>>> batch_storage = batch.storage()
>>> image_storage = image.storage()
>>> id (batch_storage) == id (image_storage)
False
>>> id (batch_storage)
139730615547904
>>> id (image_storage)
139730615546112
>>> id (batch.storage())
139730615547328
>>> id (image.storage())
139730615547584

You can see that calling .storage() multiple times on the same
tensor returns a new, different “storage” object each time.  (Why
pytorch does things this way I don’t know.)
But how then can we have:
>>> id (batch.storage()) == id (image.storage())
True

Python / pytorch creates a new storage object when image.storage()
is called, gets its id() (address in memory), and then discards the
storage object.  The same thing happens when batch.storage() is
called – a new storage object is created.  Due to the vagaries of the
python interpreter the new storage object created by batch.storage()
happens to land at the same location in memory previously being
used by the since-discarded storage object created by
image.storage(), so the two have the same id().
(Different python objects are only guaranteed to have different id()s
if they exist at the same time.  Memory – and hence id()s – can be
reused after an object is disposed of.)
Finally, consider this:
>>> batch = torch.tensor ([[1, 2], [3, 4]])
>>> image = torch.tensor ([10, 20])
>>> b0 = batch[0]
>>> batch
tensor([[1, 2],
        [3, 4]])
>>> image
tensor([10, 20])
>>> b0
tensor([1, 2])
>>> batch[0] = image
>>> batch
tensor([[10, 20],
        [ 3,  4]])
>>> image
tensor([10, 20])
>>> b0
tensor([10, 20])
>>> batch[0, 0] = 55
>>> batch
tensor([[55, 20],
        [ 3,  4]])
>>> image
tensor([10, 20])
>>> b0
tensor([55, 20])
>>> image[0] = 666
>>> batch
tensor([[55, 20],
        [ 3,  4]])
>>> image
tensor([666,  20])
>>> b0
tensor([55, 20])
>>> b0[0] = 9999
>>> batch
tensor([[9999,   20],
        [   3,    4]])
>>> image
tensor([666,  20])
>>> b0
tensor([9999,   20])

To reiterate, batch, image, and b0 are three distinct tensor objects.
But batch and b0 share the same underlying storage (in that b0
is a “view” into batch), while image has its own, unrelated storage.
Best.
K. Frank