Thanks for posting @kaleemiqb If using the recommended DistributedDataParallel (DDP) mode, where there is a dedicated process for each GPU, DDP does not split input data. Each process will have its own data loader and its own DDP instance. DDP only help to automatically compute the global averaged gradient in the backward pass. So it really depend on the dataload next batch is loaded, which I think it’s random.
for the second question, you can record the input batch, and the output of the model in its own process in a map, and if you want to concat them together do a all_gather manually, but input_batch across process might not rank properly.