I’m not understanding what you mean ‘directly on GPU using constants’? There are no constants in Python and any variable in Python exists in the CPU and has to be transferred at some point to the GPU. You can make sure that those tensors are defined and moved to GPU outside of your high frequency loops and not modified, that is as close to constant as you get.
I’m not an expert on the nitty gritty details of CUDA kernels and the specific Pytorch mechanics relating to their handling. I believe a typical kernel launch latency is approx 10us, down to about 5us and up to I’m not sure where. But I’ve seen traces with 50us, etc.
So, if you are doing convolutions or mm with tens of thousands, hundreds of thousand elements, you don’t notice that. But for a tight loop of small operations that could only partially leverage the full parallalelization ability on a modern CPU, you could probably crank out  thousands of iterations of your loop (C/C++ optimized) in the time it takes to launch a kernel on the GPU.
If you want to keep it a little higher level, maybe use Eigen, you’d get a bit of cross platform ability in the sense that it may be able to leverage a bit of NEON or SSE depending on what platform you compile for.