In a distributed setup using DistributedDataParallel, you would use a single process for each device.
Each process would then take care of its data loading.
Could you give an example of your use case and how it could interact in a bad way?