Thank you, @albanD!
About moving to GPU, I accidentally left the device argument. After removing it and trying to move to(device) again, it didn’t work in Pytorch 1.1.0:
import torch
import torch.nn as nn

class Foo(nn.Module):
    def __init__(self):
        super(Foo, self).__init__()
        self.weights = nn.Parameter(torch.Tensor(4, 4))
        self.eye = torch.eye(4, requires_grad=False)
        torch.nn.init.uniform_(self.weights)

    def forward(self):
        return self.weights - self.eye

bar = Foo()
bar = bar.to('cuda')
print(bar.weights)  # device='cuda'
print(bar.eye.device)  # 'cpu'
bar()  # error

I also tried in Pytorch 1.3.0, and it also doesn’t move self.eye to GPU, only self.weights.
I’m currently keeping the device argument, just in case.