That happens when I dropped .cuda() at line 16 and 19:
self.filter1 = self.whole_filter(in_channels, channel1, kernel_size, self.theta1, self.Lambda1, self.psi1).cuda()
self.filter2 = self.whole_filter(channel1, channel2, kernel_size, self.theta2, self.Lambda2, self.psi2).cuda()

My guess is that after I dropped .cuda(), when I call x = F.conv2d(x, self.filter1, bias=self.bias1) on line 25, self.filter1 is not on GPU, so pytorch generate a filter of the corresponding size and do the convolution.