Thank you for your response. I was able to make it work using the nn.Parameter and register_parameter but I noticed that if I just initialize the z tensor like z = torch.zeros_like(x) instead of torch.zeros(x.size()), it is automatically loaded to the same device as x.