I don’t have an idea of the process group and allreduce API. Could you please elaborate on this process or share relevant links to follow up?

Check out “Collective functions” in the tutorial 2. This is lower-level API for communication. Ideally, you probably don’t need to do any explicit communication, and you can just try DDP 2, which does not let you feel any difference compared with the training on a single process.

Do you mean to run the first distribution command on one VM and the second command on the other VM or run both the commands on both the VMs?

Run the first distribution command on one VM and the second command on the other VM