You could take a look at the Collective Communication docs 3 which explain how tensors can be scattered, gathered, and reduced in a distributed setup.
Once you’ve made sure the data is available on the desired devices, I assume you can calculate the loss and let DDP compute the gradients etc.