Let me know, what is w1 and w2?
Think that if you have a loss which is a nn.Module like
class MyLoss(nn.Module):
init 
w=nn.Parameter(torch.tensor(1))
w_gpu = w.cuda()

You only have to pass w to the optimizer, not w_gpu. w should also be a leaf tensor if defined that way.
Here you don’t have to pass w_gpu as it is a copy of w (and non-leaf tensor whose backward is just passed to w)
Same effect should be achieved if you allocate that module
class MyLoss(nn.Module):
   init 
      w=nn.Parameter(torch.tensor(1))

instance = MyLoss().cuda()

In fact, you should code it this way as the previous one is a way of allocating different parts of a submodel on different gpus, thus, it’s not intended to be used like that.
It would be useful if you paste the essential part of your code instead