Both gets slower right?
Regular:  0.21726393699645996
Regular backward:  0.42275333404541016
 Unfold:  0.357952356338501
 Mm:  0.4160928726196289
 Fold:  0.08525800704956055
Custom:  0.8644123077392578
 Back fold:  0.07413196563720703
 Back mm:  0.2995262145996094
 Back unfold:  0.44156312942504883
Custom backward:  0.8373520374298096

But the custom is still between 2 and 4 times slower.

Thank you so much for helping me! In that case, do you have ideas that can speed up fold and unfold on GPU?

I am not sure if there is anything you can do.
The thing is that on GPU, cudnn has very fine tuned algorithms to do convolution that are significantly better than MM.
Can you try comparing the runtimes on GPU when cudnn is disabled via torch.backends.cudnn.enabled = False?