I think I got it, make filter size = size of the image, thus, there’s no scanning, each set of weights from the filter will act only on corresponding pixel, only.
But now I think this is too much parameteres, and I really want to do depth-wise convolution to save some parameters. Groups is not helping at all, it is just separating the filters, no real depth-wise conv is taking place. How can I do that?