Nazila-H:

model_global.to(device) is a typo error it would be model_net.to(device). I rewrote them.


You do not need model_net.to(device) inside training loop.
Push the model to GPU before creating the optimizer.



 Nazila-H:

with loss.backward(retain_graph=True) the model would update in terms of all leaf nodes (data and weights).


I am not sure what do you mean by this. Even without retain_graph=True, the model update will do what you mentioned. No?
It seems like you have a larger distributed training mechanism. Maybe if you try simple one-node training to overfit the model with small batches of data, you might be able to find the cause.