Ahh, thank you, that makes sense.
Now the real question is: how does PyTorch launch the jobs so efficiently? For the learning experience, I’ve re-implemented convolution with my own version of im2col. Using mine, the kernel launch time is an order of magnitude slower, but the actual running time (after synchronizing) is only 1.25x slower. Is there some funky stuff going on under the hood to launch the job?