I am a deep learning beginner and running PyTorch’s Demo for study. I have several computers and laptops at home, the situation is different for each machine, some machines only have CPU, and some have powerful GPU, but I hope they all come together to speed up a training process.
My implementation idea is the C / S structure. When the server is doing model input, if there is a request from the client, any number of samples are sent to the client through the socket, and the calculation of these sent samples is skipped. After the client completed, it sends back the results of the model forward, and the server merges the results. Finally, if the server completed an epoch, it blocks and waits for the results of all clients to return.
Now, what I don’t know is how to merge the results of the model foward on the client into the server. I don’t know if this is correct …
PS: Or, when the client’s model calls forward (), it does not make a classifier. Before the classifier is called, the data is sent back and the server model completes the classifier. Can the main calculation amount in the forward process be shared Arrived