I think the easiest approach would be to unfold the inputs, apply the weights to each patch, and use a matrix multiplication approach for the convolution, which will most likely use a lot of memory and be slow.
Checking the native conv implementation in C++ could yield a speedup, but I also understand this wouldn’t be interesting to you.