I profiled the code using the torch.autograd.profiler.profile(use_cuda=True).
Torch.mul and sub take the large majority of the time. I find this odd considering the FFT should be the most expensive process.
mul   Total Cuda Time: 13374.464us
sub   Total Cuda Time: 12881.920us
fft      Total Cuda Time:  377.856us
A single multiplication takes more time than the entire 2dconv.  Is there any reason for this?
FFT Conv

------------------  ---------------  ---------------  ---------------  ---------------  ---------------
Name                       CPU time        CUDA time            Calls        CPU total       CUDA total
------------------  ---------------  ---------------  ---------------  ---------------  ---------------
ConstantPadNd             137.393us        912.160us                1        137.393us        912.160us
tensor                     13.192us         12.704us                1         13.192us         12.704us
fill_                      20.303us        271.840us                1         20.303us        271.840us
narrow                     10.064us          3.072us                1         10.064us          3.072us
as_strided                  4.841us          1.024us                1          4.841us          1.024us
narrow                      6.956us          3.072us                1          6.956us          3.072us
as_strided                  3.490us          1.024us                1          3.490us          1.024us
fft                       366.340us       1125.376us                1        366.340us       1125.376us
reshape                     6.930us          3.072us                1          6.930us          3.072us
as_strided                  2.757us          1.024us                1          2.757us          1.024us
_fft_with_size            341.994us       1115.136us                1        341.994us       1115.136us
reshape                     7.154us          3.072us                1          7.154us          3.072us
as_strided                  2.890us          1.024us                1          2.890us          1.024us
select                      6.358us          3.072us                1          6.358us          3.072us
as_strided                  2.261us          1.024us                1          2.261us          1.024us
select                      5.708us          3.072us                1          5.708us          3.072us
as_strided                  2.252us          1.024us                1          2.252us          1.024us
ConstantPadNd              69.969us         98.304us                1         69.969us         98.304us
tensor                      3.763us          1.024us                1          3.763us          1.024us
fill_                       8.095us         81.952us                1          8.095us         81.952us
narrow                      6.727us          3.072us                1          6.727us          3.072us
as_strided                  2.766us          1.024us                1          2.766us          1.024us
narrow                      6.890us          3.072us                1          6.890us          3.072us
as_strided                  3.516us          1.024us                1          3.516us          1.024us
fft                       327.536us        377.856us                1        327.536us        377.856us
reshape                     6.212us          3.072us                1          6.212us          3.072us
as_strided                  2.527us          1.024us                1          2.527us          1.024us
_fft_with_size            305.552us        367.616us                1        305.552us        367.616us
reshape                     6.970us          3.072us                1          6.970us          3.072us
as_strided                  2.821us          1.024us                1          2.821us          1.024us
select                      6.008us          3.072us                1          6.008us          3.072us
as_strided                  2.276us          1.024us                1          2.276us          1.024us
select                      6.037us          3.072us                1          6.037us          3.072us
as_strided                  2.459us          1.024us                1          2.459us          1.024us
expand                      3.891us          0.000us                1          3.891us          0.000us
expand                      2.262us          1.024us                1          2.262us          1.024us
mul                        13.018us      13373.440us                1         13.018us      13373.440us
expand                      2.868us          1.024us                1          2.868us          1.024us
expand                      3.245us          2.048us                1          3.245us          2.048us
mul                         7.824us      13374.464us                1          7.824us      13374.464us
add                        10.083us        130.049us                1         10.083us        130.049us
add                         7.389us        405.502us                1          7.389us        405.502us
expand                      2.945us          1.024us                1          2.945us          1.024us
expand                      2.206us          1.024us                1          2.206us          1.024us
mul                         7.154us      10663.937us                1          7.154us      10663.937us
sub                         9.580us      12881.920us                1          9.580us      12881.920us
sub                         6.989us      12941.307us                1          6.989us      12941.307us
sub                         6.710us      12944.382us                1          6.710us      12944.382us
sum                        17.342us       4509.697us                1         17.342us       4509.697us
_sum                       12.580us       4507.652us                1         12.580us       4507.652us
sum                        12.929us       4544.518us                1         12.929us       4544.518us
_sum                        7.680us       4542.458us                1          7.680us       4542.458us
unsqueeze                   4.507us          1.022us                1          4.507us          1.022us
unsqueeze                   2.375us          1.022us                1          2.375us          1.022us
cat                        15.456us       1828.865us                1         15.456us       1828.865us
ifft                      332.527us       3326.973us                1        332.527us       3326.973us
reshape                     7.441us          3.075us                1          7.441us          3.075us
as_strided                  3.722us          1.022us                1          3.722us          1.022us
_fft_with_size            308.962us       3316.734us                1        308.962us       3316.734us
reshape                     7.042us          3.075us                1          7.042us          3.075us
as_strided                  2.849us          1.030us                1          2.849us          1.030us
slice                       6.009us          3.075us                1          6.009us          3.075us
as_strided                  2.182us          1.022us                1          2.182us          1.022us
slice                       5.469us          3.075us                1          5.469us          3.075us
as_strided                  2.191us          1.030us                1          2.191us          1.030us
select                      5.729us          3.075us                1          5.729us          3.075us
as_strided                  2.198us          1.030us                1          2.198us          1.030us

F.conv2d
---------------------  ---------------  ---------------  ---------------  ---------------  ---------------
Name                          CPU time        CUDA time            Calls        CPU total       CUDA total
---------------------  ---------------  ---------------  ---------------  ---------------  ---------------
conv2d                        62.204us       1486.848us                1         62.204us       1486.848us
convolution                   58.547us       1483.776us                1         58.547us       1483.776us
_convolution                  55.039us       1482.272us                1         55.039us       1482.272us
clone                         11.581us        390.144us                1         11.581us        390.144us
tensor                         3.375us          1.024us                1          3.375us          1.024us
cudnn_convolution             29.480us       1085.440us                1         29.480us       1085.440us