For small matrix operations on CPU (e.g., 3x3 matrices), NumPy seems to be approximately 15x faster than PyTorch. This is relevant for operations in the data loader or (geometric) transformations where I could achieve great speedups by using NumPy arrays instead tensors.
Here is a small performance test which just adds a constant to a 3x3 matrix:
>python -mtimeit -s"import numpy as np; test = np.random.rand(3,3)" "test + 1"
500000 loops, best of 5: 765 nsec per loop

> python -mtimeit -s "import torch; test = torch.rand((3, 3))" "test + 1"
20000 loops, best of 5: 11.2 usec per loop

I assume that the overhead for PyTorch tensors is much higher than for NumPy arrays. When using larger arrays and tensors, the overhead will be less relevant and I have seen contributions in this forum and on GitHub where the performances seem to be comparable.
I am working with PyTorch 1.3, NumPy 1.16.4 and Python 3.7.4 on a 64 bit Windows machine.
(Btw: I found some articles in this forum comparing the speed of specific operators and discussing the processing speed of large matrices but not about the overhead of the matrix classes itself. If I have missed an article – please let me know!)