So I’m still not quite sure how to handle kernel fails and properly print the error. I thought what I had worked, but I’ve run into issues where it just says “kernel failed: no error”. Probably an out of memory issue? The GPU I tested on has ~12GB of RAM. I also have CUDA_NUM_THREADS=512. Maybe that value needs to change for your setup? It’s all hardcoded, unfortunately.
Also, I fixed those imports shortly after posting–this was all tied up in a different module at one point. It should work now.
In any case, I ran your better profiler script locally and un-commented custom code, which I ran as well. Here are the results on a GTX 1080T with CUDA 10.1 (still looking for the cudnn version) and PyTorch 1.2
######################### CUDNN FALSE


Various patch convolution methods:
  Using-unfold avg. time: 0.054104 seconds
  Rolled-into-batch avg. time: 0.996649 seconds


Compare to traditional convolution with B x C x H x W inputs with similar number (actually more) of elements:
  nn.Conv2d: 0.041167 seconds
######################### CUDNN TRUE


Various patch convolution methods:
  Using-unfold avg. time: 0.044344 seconds
  Rolled-into-batch avg. time: 0.072312 seconds
  Custom-patch-im2col avg. time: 0.037970 seconds


Compare to traditional convolution with B x C x H x W inputs with similar number (actually more) of elements:
  nn.Conv2d: 0.038860 seconds
######################### CUDNN TRUE BENCH


Various patch convolution methods:
  Using-unfold avg. time: 0.044354 seconds
  Rolled-into-batch avg. time: 0.040456 seconds
  Custom-patch-im2col avg. time: 0.038056 seconds


Compare to traditional convolution with B x C x H x W inputs with similar number (actually more) of elements:
  nn.Conv2d: 0.044387 seconds
######################### CUDNN TRUE BENCH


Various patch convolution methods:
  Using-unfold avg. time: 0.044427 seconds
  Rolled-into-batch avg. time: 0.035127 seconds
  Custom-patch-im2col avg. time: 0.037969 seconds


Compare to traditional convolution with B x C x H x W inputs with similar number (actually more) of elements:
  nn.Conv2d: 0.038883 seconds


It’s also possible that the unfold approach is taking advantage of not re-allocating memory each time? My method treats each layer independently and forward and backward passes re-allocate intermediate storage. Perhaps there’s some optimization going on in the PyTorch backend to not re-allocate and destroy GPU memory so much. Perhaps the intermediate storage is fixed through a full network pass somehow? I’m speculating now, but it might explain the performance gains.