That code is too high level, so the inner loop sucks.

Dispatch by dtype, like built-in kernels do.
Instead of Tensor.operator[] indexing, use either raw pointers (this implies contiguous tensors and doing address arithmetic) or “accessors” (torch API).
Compute dot product manually