Hi, thanks for your reply!
The tensor is stored in one of the nodes. More spefcifically, I have, say, two nodes and each of them have 8 gpus. I have a text dataset train.txt. I have written a function that convert the text data to  large tensor X.
If I used torch.nn.parallel.DistributedDataParallel in the following way:
model = torch.nn.parallel.DistributedDataParallel(mode.cuda())

Would model(X) do what I want, that is, split the X and distribute the pieces to 16 gpus?