Alex_Ge:

As you can imagine, with large data-sets that’s very hard to do, the GPU memory is also used by the model and the batch samples. So I am looking into having parts of the data-set across multiple-GPUs.


The usual work flow would be to try to load the next batch into the RAM while the GPUs are busy training the model to hide the latency of the data loading step.
In each iteration you would push a batch to the device and start loading the next batch.



 Alex_Ge:

According to the error produced, and as you explained that’s a problem because the weights, batch samples and other ops (CE Loss tensors) need to be on the same device?


Yes, that’s correct. In order to execute the wanted operation (e.g. a matrix multiplication), the parameters (data and weight) need to be on the same device.



 Alex_Ge:

It is my understanding (please correct me if I’m wrong) that I need to have copies of the network/model across multiple GPUs (which is where nn.DataParallel comes into play) and I also need to ensure that the batch tensors are also on the same GPU?


Generally yes. However, you usually don’t need to care about the actual data scattering and gathering as DDP will do this all for you.
You would have to specify a default device (e.g. GPU0), push the data onto this default device, and execute your training. DDP should automatically scatter the provided data in dim0 to all devices, as well as take care of all synchronization.



 Alex_Ge:

Alternatively, I was wondering if I should have multiple dataloaders each associated with a GPU, and then just train a model across those different dataloaders/GPUs.


Multiple DataLoaders might make sense for specific use cases, but you would have to check your system resources and see, if this would create a bottleneck (loading data from SSD, preprocessing on CPU etc.).
I would recommend to use a single DataLoader and DDP first as given in the liked tutorial.



 Alex_Ge:

how can I ensure that they are on the correct gpu device?


nn.DataParallel and DDP will split the batch in dim0 and scatter each chunk to the corresponding device.
This tutorial 112 gives you a good example.
Let me know, if something is unclear.