I see. My concern is regarding small conv. I heard that if we compare the runtimes of a big Conv layer and a series of cascaded small Conv layers, we will find that the cascaded small Conv layers takes more times to run even if the two settings have the same number of floating point operations. This is because small Conv layers cannot make full use of the GPU.
So I wonder if PyTorch has a mechanism to parallelize the small Conv layers that runs independently to accelerate it.