Hello,
I am trying to train n-models.  Each model has the same structure, same inputs, but is learning a different output.  Since the models are distinct from one another, I would like to parallelize them.  My attempt so far, is putting models on different GPUs.  This is done through:
def try_gpu(i=0):  #@save
    """Return gpu(i) if exists, otherwise return cpu()."""
    if torch.cuda.device_count() >= i + 1:
        return torch.device(f'cuda:{i}')
    return torch.device('cpu')

model = [LinearNet(net_num).double().to(try_gpu(i%num_gpu) for i in range(net_num)]

So for example, if I have 6 models and num_gpu = 2, then model[0] is on cuda:0, model[1] is on cuda:1, etc.
I have also made sure to put the inputs onto the different GPUs as well.  However, I’m stuck at actually making this parallel - I’m still just looping over models and running them sequentially.  How would I actually run model[0] and model[1] in parallel?
Thanks for the help!