Hi Ptrblck, I added nn.Parameter and it works! Thank you very much!! A further question: when I deleted .cuda() in self.filter1 and self.filter2, the parameters have size:
<class 'torch.Tensor'> torch.Size([20, 1])
<class 'torch.Tensor'> torch.Size([20, 1])
<class 'torch.Tensor'> torch.Size([20, 1])
<class 'torch.Tensor'> torch.Size([20])
<class 'torch.Tensor'> torch.Size([20, 1, 5, 5])
<class 'torch.Tensor'> torch.Size([50, 20])
<class 'torch.Tensor'> torch.Size([50, 20])
<class 'torch.Tensor'> torch.Size([50, 20])
<class 'torch.Tensor'> torch.Size([50])
<class 'torch.Tensor'> torch.Size([50, 20, 5, 5])
<class 'torch.Tensor'> torch.Size([500, 800])
<class 'torch.Tensor'> torch.Size([500])
<class 'torch.Tensor'> torch.Size([10, 500])
<class 'torch.Tensor'> torch.Size([10])

It’s wrong, I should not have the parameters with size ([20, 1, 5, 5]) and ([50, 20, 5, 5]), since I am not using the conventional convolution kernel here. Moreover, the loss is nan during training. But when I keep .cuda() there, it’s correct and training is ok:
<class 'torch.Tensor'> torch.Size([20, 1])
<class 'torch.Tensor'> torch.Size([20, 1])
<class 'torch.Tensor'> torch.Size([20, 1])
<class 'torch.Tensor'> torch.Size([20])
<class 'torch.Tensor'> torch.Size([50, 20])
<class 'torch.Tensor'> torch.Size([50, 20])
<class 'torch.Tensor'> torch.Size([50, 20])
<class 'torch.Tensor'> torch.Size([50])
<class 'torch.Tensor'> torch.Size([500, 800])
<class 'torch.Tensor'> torch.Size([500])
<class 'torch.Tensor'> torch.Size([10, 500])
<class 'torch.Tensor'> torch.Size([10])

So I am curious why I cannot delete .cuda(), since you mentioned that nn.Parameter already put the filters on GPU. Thanks a lot!