After several hours of debug I have found out the potential problem. In my setup I have initialized my model, moved it on the GPU inside the master process and then re-used it in all the processes composing the DDP. Moving the creation of the model inside each single process (instead of doing it in the master one) solved the problem. I think that the major problem was moving it on the GPU before the creation of multiple processes. My supposition is that PyTorch cannot move the parameters from a process to another if they are CUDA tensor.
So for all future readers, better to create your model after the mp.spawn command