I’ve adopted your code and did some test. Your problem is that you are using too small data to take the fft an effect. I need to do big convolutions i. e. 30x30 px over “full HD” images 1920x1920. In this setup your implementations brings some fruit over when comapring your fft convolution ran on GPU and basic numpy fftconvolution.
kernel 30x30 img 1920x1920
‘fft_conv2d_numpy_test’      52.11 ms  36.56 ms  42.81 ms  42.81 ms
‘conv2d_torch_cpu_all’      429.19 ms 415.87 ms 422.51 ms 416.33 ms
‘conv2d_torch_gpu_all’     1910.89 ms 111.67 ms 111.00 ms 135.16 ms
‘fft_conv2d_torch_cpu_all’  282.97 ms 250.37 ms 255.89 ms 266.99 ms
‘fft_conv2d_torch_gpu_all’  224.72 ms  22.27 ms  20.70 ms  21.17 ms