http://pytorch.org/docs/0.2.0/tensors.html#torch.Tensor.expand 26 It essentially just change one stride to 0. So it is effectively a matrix with four tensor repeated, but the underlying storage is the same vector. And of course autograd handles it as well. See https://github.com/pytorch/pytorch/blob/master/torch/autograd/_functions/tensor.py#L141 9.