I was looking into training machine learning models in multiple cores. To be more clear, suppose I have “N” machine learning units (for eg. three layered neural network [in-hid-out] ). Each of the units are identical to each other. However, I want to train each network with different input of same nature (for eg. If I have 10 machine learning units with MNIST data as input, each of the 10 units will be trained on different sets of data). You can think of it as training the networks with MNIST in 10 geographically dispersed location where we are not sure which network will have what set of inputs. However, at somepoint I want to communicate between the machine learning models while updating the weights. I want to distribute same weights to all the models. For people who know federated learning, its like applying federated learning in multiple CPU/GPU cores.
Is it possible to do something like this in multiple cores of a CPU or GPU? Or is there any documentation that you can provide me?
@ptrblck @rasbt