copyrightly:

I found out that when I dropped .cuda() , self.filter1 and self.filter2 appear in model.parameters() , that’s where I got those extra shape.


filter1 and filter2 shouldn’t be returned by model.parameters() regardless of the cuda() call, as they are defined as tensors, not nn.Parameters.
If you want to register them as parameters, warp them in nn.Parameter.



 copyrightly:

Further in this case, self.bias1 and self.bias2 have value nan , although self.theta1, self.Lambda1, self.psi1 have their value. This makes me quite confused.


I removed the cuda() call, and changed both filters to be parameters:
    def whole_filter(self, in_channels, out_channels, kernel_size, theta_column, Lambda_column, psi_column):
        result = torch.zeros(out_channels, in_channels, kernel_size, kernel_size) # \text{out\_channels} , \frac{\text{in\_channels}}{\text{groups}} , kH , kW
        for i in range(out_channels):
            result[i] = self.one_filter(in_channels, kernel_size, theta_column[i], Lambda_column[i], psi_column[i])
        return nn.Parameter(result)

It seems the model is working for at least one epoch:
...
Train Epoch: 1 [59520/60000 (99%)]      Loss: 0.054967

Test set: Average loss: 0.1043, Accuracy: 9678/10000 (97%)