@soumyadipghosh Thanks for contributing this to the community and for the C++/MPI example PR!
Just as a general note for this thread, using the c10d APIs will enable distributed data parallel training that will produce the same results as DDP. However, calling allreduce after the backward pass to synchronize gradients will likely lag in performance as compared to DDP, which overlaps computation and communication by synchronizing smaller buckets of gradients during the backward pass.