For loops add the Python overhead in each iteration and call into the dispatcher for a small workload.

The for loops add very little overhead as far as I could tell from profiling using line_profiler.
Is the pytorch dispatch really enough to explain it being 77 times slower?

A lot of operations are using vectorization inside, which won’t be fully used.

What does vectorization really mean at a low level?
Do you mean that pytorch uses avx2 style SIMD parallelism on CPU?