If you don’t want to transform your images, you could load them as tensors.
However, if you do need a transformation, you would have to transform the tensors back to PIL.Images, apply the augmentation/transformation, and transform them back to tensors.
This shouldn’t be too bad regarding the performance, but can be annoying.
On the other hand do you want to load single images/tensors?
If so, I think you won’t get any performance improvement:
x = torch.empty(3, 224, 224).uniform_(0, 1)
img = transforms.ToPILImage()(x)

# save
torch.save(x, 'tensor_image.pth')
img.save('pil_image.png')

%timeit torch.load('tensor_image.pth')
> 1000 loops, best of 3: 206 µs per loop
%timeit Image.open('pil_image.png')
> 10000 loops, best of 3: 86.6 µs per loop

I think you’ll be faster if you load a whole batch of tensors.