I was thinking to write a C++ implementation in LibTorch of a customized convolution, that iterates over the pixels of the image and performs the convolution, without the time-efficient / memory-unefficient unfold trick.
Alternatively, I can work in Python by cuting the input image and performing the convolution space-wise (was thinking to use the group parameter of Conv2d but I don’t think it’s possible), taking care of correct padding between the image patches.
The last resort could be to implement a pytorch_sparse unfold operator, considering that the images have lot of pixels set to 0.