@pritamdamania87 thank you for your explanations. They are really helping out!

Thanks for providing a code example. IIUC, you want to just compute a particular math function in parallel and not really train a model in data parallel fashion where you aggregate gradients across ranks in the backward pass? If so, you don’t really need to use DistributedDataParallel.

Yes this is right. I’m wanting to do a calculation in parallel and collect the results. Is there a better way to do this in pytorch?