Thanks for the reply! @ptrblck
The size means the “<<<X,Y,Z>>>” before the cuda kernel function call. As I went through some of the code, I now think the size for the same function is the same across different runs, but always computed again and again at runtime for the same cuda kernel function call each time. Is this correct(after benchmarking and picking the actual kernels for a static model)?
And for cudnn kernel, PyTorch has no way to decide the size right? “<<<X,Y,Z>>>” is determined inside cudnn like a blackbox.