Yes I’m using Embedding with arg sparse=True. But seems not ok to run DDP only if I using scatter_add later.
If using 2 processes to run DDP. Then I set CUDA_VISIBLE_DEVICE=0,1 for each prcoess.
code like below
rank = dist.get_rank()
device = torch.device(‘cuda’, rank)
model = model.to(device)
model = torch.nn.parallel.DistributedDataParallel(model,device_ids=[rank],output_device=rank)
I tried to launch each process with CUDA_VISIBLE_DEVICE=0 CUDA_VISIBLE_DEVICE=1 but seems not work.