OK I figured it out by myself.
When I defined the parameter, I used
self.M = nn.Parameter(torch.randn(size0, size1)).cuda()
but actually it should be:
self.M = nn.Parameter(torch.randn(size0, size1).cuda())
I think the program should report an error/warning in this case, instead of just letting the code run.