The initialized model is broadcast from rank 0 to the other processes from the DDP constructor, so the initialized parameters will be identical when you first call forward.
Numerical reproducibility depends on determinism on the inputs/transforms, deterministic partitioning of the dataset, as well as deterministic execution of your model (e.g. there are modes where cuDNN does not execute deterministically). If at any of these you have some non-determinism, numerical reproducibility is impossible.