@space1panda Hi! I have 3 models and 3 optimizers in my framework as well. Actually, I implemented three torch.nn.parallel.DistributedDataParallel classes as follows, instead of ONE. And also the log ‘Reducer buckets have been rebuilt in this iteration.’  repeated three times. I only use one GPU to train. So it is ok to use multiple DistributedDataParallel models in Pytorch DDP?
model1 = ddp(model1, device_ids=[local_rank], output_device=local_rank)
model2 = ddp(model2, device_ids=[local_rank], output_device=local_rank)
model3 = ddp(model3, device_ids=[local_rank], output_device=local_rank)