I would trust the terminal python -mtimeit results since they reflect my experiences in my training framework: By replacing small PyTorch matrix operations by NumPy operations, I could achieve a great speed-up. (This mainly affects data loading and augmentation, which is not vectorizable in my case. The large training vector operations are of course carried out on the GPU.)
@peterjc123: I do not think that this has something to do with SIMD instructions since both of the above tests run on a single CPU as my task monitor indicates. Additionally, you could re-run the tests with a 1x1 matrix yielding approximately the same results.
Here is an alternative timing measurement delivering similar results:
import torch
import time
iterations = 10000
test_tensor = torch.zeros((1, 1))
test_array = test_tensor.numpy().copy()
time_start = time.time()
for i in range(iterations):
    test_tensor = test_tensor + 1
t_torch = (time.time() - time_start) / iterations
time_start = time.time()
for i in range(iterations):
    test_array = test_array + 1
t_numpy = (time.time() - time_start) / iterations
print('Time elapsed (PyTorch tensor): {0:.3f} µs'.format(t_torch * 1E6))
print('Time elapsed (NumPy array): {0:.3f} µs'.format(t_numpy * 1E6))

Time elapsed (PyTorch tensor): 9.798 µs
Time elapsed (NumPy array): 0.800 µs