Hi,
There are a few things that you can do to speed up your code, but this is expected to the slower and much more memory hungry than the conv2d module itself.
This is because the conv2d can use other conv implementation (based on fft for example) depending on the input shape to reduce memory use.
I would advise using the functional version of unfold as well from here 53 so that you only need one call.
You can rewrite:
    patches = patches * weight
    patches = patches.sum(-1).sum(-1).sum(-1)

to be more efficient by collapsing the last 3 dimensions and doing a matrix matrix multiplications between patches and weight.transpose(-1, -2) to get the result of these two lines with a single op.