I wonder the inner mechnism when calling mymodule.to(device) for a typical module;
class MyModule(nn.Module):

  def __init__()
    super...
    self.w = nn.Parameter(torch.ones(1))

When we call mymodule.cuda() the returned tensor w from nn.Parameters() will pointer to gpu tensor w. However, our optimizer can not optimize non-leaf tensor if  gpu w is created by gpu_w = cpu_w.cuda(), so What does pytorch solve this problem?
Thank you.