You would have to share a bit more about what you’re trying to do to be sure, but one bridge to more bespoke convolution-like operations can be the torch.nn.Unfold 1 module or the corresponding torch.nn.functional.unfold. What it does is reducing the convolution to Unfold + batch matrix multiplication with the kernel reshaped to a matrix form - the linked documentation has a demonstration.
This allows to replace the linear transformation in the convolution with other ops.
Note, though, that this is quite memory (and compute) intensive – each pixel is duplicated to all filters it appears in, but it might work for prototyping.
Best regards
Thomas