Move the to(self.device) operation into the Parameter call:
self.W = Parameter(torch.ones((3, 5)).to(self.device))

The .to is a no-op for CPU tensors in your code, while you are manipulating and thus losing the nn.Parameter for the GPU case.