Hi, I’m trying to use DistributedDataParallel to enable training on multiple GPUs on a single node. According to what I read from online documents, the procedure is the following (please correct me if it’s not true):
dist.init_process_group(backend=backend, init_method=init_method, rank=rank, world_size=world_size)
model = DistributedDataParallel(model, device_ids=[local_rank])

And then spawn the process in the main function:
multiprocessing.spawn(fn, args=args, nprocs=world_size, join=True)

I have two questions regarding this:


What should I set these parameters to? e.g., backend, init_method, rank, world_size… I didn’t find an example showing details of setting these parameters.


I saw some examples online used DistributedSampler. Is this necessary for using DistributedDataParallel?


Thanks!!