From what I recall, there is a fixed overhead to the cuda runtime. And the an overhead that depends on the amount of cuda code there is in the whole codebase (not sure why). And we do have a large number of cuda kernels as we have pytorch’s own ones, cudnn, magma etc.