Hi,
That is expected. The backward of sum, for more efficiency return a tensor that is expand.
That means that in practice, there is a single memory element allocated for it, and the stride is 0 for all dimensions as you can see.
This means that the first element in the first dimension is at position 0*0=0 but the second element is at position 1*0=0 as well. The full formula is element_index * stride = position_in_memory.
In pytorch, you have last dimensions that are contiguous, and so after cloning and getting a full tensor. The stride of the last dimension is 1 as expected: you need to move of 1 to get the next value of that dimension. For the previous to last dimension, the stride is 5 (which is the size of the last dimension) as there are 5 elements between each consecutive elements on that dimension.
Note that pytorch memory access also have a storage_offset element that means that the element [0,0,0,0] of your tensor might not be the first element in the pointer for data.
If you’re not familiar with these things, it might be simpler to keep the clone and then you can assume that you have a contiguous tensor in the cuda code and don’t need to pass storage_offset and stride as arguments.