Thanks for posting @divinho. From your description it seems that your model is dropping a particular layer and the loss calculation does not use that layer at all. If you want to train in such way with DDP, can you try use find_unused_parameters=True when initializing DDP?
Currently, find_unused_parameters=True must be passed into torch.nn.parallel.DistributedDataParallel() initialization if there are parameters that may be unused in the forward pass.