742617000027:

Is that what you were asking?


Yes.
model = torch.distributed.nn.parallel.DistributedDataParallel(model)

The line above might be the reason. IIUC, torch.cuda.set_device(args.local_rank) only sets the default device, but DDP process can still see two GPUS. By default DDP would try to replicate models on all visible GPUs.
Could you please try use the following to construct DDP?
    model = torch.nn.parallel.DistributedDataParallel(model,
                                                      device_ids=[arg.local_rank],
                                                      output_device=arg.local_rank)