Hello @ptrblck ,
So what I’ve done so far, is that I send model1 to gpu 1 and model2 to gpu 2, and I define
models = [model1, model2]

In my training model then, I have
loss_fn = nn.MSELoss()
optimizer = [optim.Adam(_.parameters(), lr=lr for _ in models]

After sending the input tensors (X,y) to the appropriate GPUs as well, I then have
for _ in models:
    _.train()

pred = []
for i in range(len(models)):
    pred.append(model[i](X[i])

And it is at this point that despite the models being on different GPUs, that I’m worried I’m still doing things in a serial fashion.
Thanks for the help!