I don’t know how you’ve installed PyTorch, but would assume you’ve built it from source (or did you find pip wheels for aarch64?).
If so, did you install NEON on your system and was PyTorch detecting it during the build? This should use vectorized operations and could yield a speedup. Also, OpenBLAS should be able to use all your cores.