There are a lot of ways of doing this. But if you mean how to split a batch of data to multiple GPUs you can use nn.DataParallel, this 66 might give you some insights on how to do it properly in PyTorch.