This is a follow up to my previous question asked here How can I insert a branch variable in a model graph in pytorch? 4,
The problem is , I noticed when training, the GPU utilization is very bad, and at first I thought maybe its because I didn’t make my parameters use cuda() it seems a lot is being off loaded to CPU instead of GPU. here is a screenshot during training.

pytorch_customparameter.gif467×577 19.1 KB

However, when I simply changed my parameter from
self.mybranch = nn.Parameter(torch.zeros(1))

to
self.mybranch = nn.Parameter(torch.zeros(1)).cuda()

Not only the GPU utilization is not fixed, even worse, the backpropagation now wont work for my parameter at all and it wont get tuned!
If I remove .cuda(), it gets trained fine!
So here I’m facing two issues,

Why using .cuda() prevents my parameter from participating in the optimization process?
Why is GPU not utilized like before?

by the way, my model uses .cuda() ( the CrossEntropyLoss also has .cuda() but it seems it works fine without it too! I mean it seems it doesn’t make any difference whether I use .cuda() on criterion)
What is wrong here?
What am I missing here?