Sanjayvarma11:

Coming to your question we looked at various articles but we are so confused on using this,Like we have to find initially whether pytorch identifes our connected computers.So can you please provide a starter code to check that??


You can find a tutorial for this here: https://pytorch.org/tutorials/intermediate/ddp_tutorial.html. In particular the init_process_group part that is called on every node is how PyTorch identifies all the connected nodes.



 Sanjayvarma11:

So if our main computer runs training code then what our client computers should run in order to accept and process data.We dont have any idea.We connected our computers using LAN cable.Thank you sir for answering…)


To use DDP, all your nodes would actually run the same code and train the same model on all nodes. DDP would take care of gradient synchronization across all nodes to ensure all the models are in sync.