Hello everybody,
I need help setting up DataDistributedParallel.
I would like to run 8 processes in parallel in 8 V100 Teslas but only one machine.
seed = args.seed + dist.get_rank()
torch.manual_seed(seed)
np.random.seed(seed)
random.seed(seed)
torch.cuda.set_device(dist.get_rank())

model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.local_rank])

torch.distributed.init_process_group(backend='gloo') # I tried many variations

args.device="cuda:{}".format(args.local_rank)
model.to(args.device)

To launch it I am using:
 python3 -m torch.distributed.launch --nproc_per_node=8  train.py (other args...)
When I run it with ‘nccl’ as backend it will freeze in torch.nn.parallel.DistributedDataParallel.
When I use ‘gloo’ instead it claims I dont have memory:
RuntimeError: CUDA out of memory. Tried to allocate 224.00 MiB (GPU 0; 15.78 GiB total capacity; 724.41 MiB already allocated; 191.25 MiB free; 794.00 MiB reserved in total by PyTorch)
    model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.local_rank])

Which doesn’t make any sense to me because I was supposed to have enough memory according to the own error message.
Thank you very much