Thanks a lot for your response!
These networks are not connected to each other. They are completely standalone models. However, every once a while, I need to manage these models somehow. Basically, I am trying to use different types of optimizers to train multiple neural networks with same architectures, say, ResNet-50. After every 5 epochs, I need to observe the training accuracies for these models and then modify the optimizers or even delete some optimizers accordingly if they behave too bad.
Hence, I need to train all of them and keep track of the performance as well as the weights for these neural networks to be able to continue the same process every 5 epochs. Based on what you are saying, should I specify the GPU for each of them in my script and then collect the performance after every 5 epochs? For example, if I have 50 such networks and 10 GPUs, I then need to loop through them by running 50 / 10 = 5 times to gather all the results?
However, this method seems like a little bit dirty and hard to manage. I am wondering if there is any better method. Thanks again!