Hi,
The runtimes you see is that it takes ~2ms to run a convolution. This is most likely not bound by how much compute your cpu can do. And since the particular algorithm used here scales very well with the number of output channels, you don’t see a big difference between the two.
Note that if you use a GPU, setting cudnn.benchmark=True will be most likely very useful here as a different algorithm can be chosen.