This tensor won’t be split, as dim0 has only a single sample and thus only a single GPU will execute this tensor.
If you want to make sure each GPU gets a chunk of the input tensor, make sure dim0 has at least a size of num_gpus.