nn.DataParallel will use the default device (GPU0 if you haven’t changed it) to store the data as well as the model and will then scatter the data and the model to each device (which is why DistributedDataParallel is preferred as it doesn’t create an imbalanced GPU memory usage and also avoids the model replication). The detailed workflow is explained in e.g. this blog post 2.