Thanks for your reply. I’m now moving all losses to GPU 0 and then doing backpropagation. And it seems to work as all parameters in both Model1 and Model2 would update. But I notice that the parameters in Model2 only update slightly compared to Model1. I’m not sure whether the backpropagation is influenced in Model2 due to the location of model losses or it’s all because of the way I’m using the loss. Please let me know if this works for you.