Hi,
I’m trying to implement the following paper: Population based training 56 for a simple CIFAR classifier.
As a part of this I need to train multiple models, with different hyperparameters, in parallel (they will be fed the same data). Each of these models would then update a global dict with its validation accuracy as well as its parameters. These models would then periodically and asynchronously explore and exploit using this list.
Is there any way to simultaneously train different models (models with different hyperparameters), each on a separate GPU, in parallel? Additionally how can they be made to update a global list or dictionary? Communication doesn’t have to be synchronous.
I came across this post on the forum but the implementation seems to be sequential rather than parallel. https://discuss.pytorch.org/t/model-parallelism-in-pytorch-for-large-r-than-1-gpu-models/778/2
Any help would be greatly appreciated