@ptrblck
@soumith chintala
Hello everyone.My friend and I are trying to implement multiple cpu distributed training in pytorch using python sockets.we looked at pytorch inbuilt functions but we didn’t get any starter code for detecting whether cpu’s are connected or not.so we decided to implement it in sockets
Main problem:
We distributed data perfectly but I have a theoretical doubt.So my plan is I will compute loss for distributed cpu’s for their batches and send all those losses to main computer and compute average.
Now I will send those average loss to all the distributed cpu’s and update model according to that loss so that model wts will be synchronized.but the avg loss which I am getting from server computer will not no relationship graph wrt to the model stored in the computer.so how can I tackle this??
Main doubt is:
Can we update model wrt to loss that has no relation wrt to model??