Hi,
I am not experienced in this area and just wanted to discuss about it.
Intuitively, I can say that when you enable multi-threading, you are adding some additional costs such as context switching between threads. So, for any operation on a small tensor, the additional costs of multi-threading compensate the speed up. Same thing applies in using GPU vs. CPU.
For instance, using GPU for operations on small tensors has almost same performance on CPU too but when tensor size grows exponentially, the distance between the performances increases enormously.