Besides the additional checks for numpy array inputs, number of channels etc. in the torchvision implementations, you are not creating a contiguous output (TF.to_tensor does it here 1) and would also see a performance penalty due to the needed copy after the permute operation.
You can check it via:
out = to_tensor(img)
print(out.is_contiguous())
> False

Next operations, which might require a contiguous input, would then trigger the copy and you would slow them down instead.