Apparently numpy has (fast) memory mapping:
img = np.memmap(filename, dtype='int16', mode='r').__array__()

That gives me a numpy array of int16.  I can’t use .numpy() to immediately convert that object into a torch tensor, because conversion from int16 is not supported.  That’s okay in my case – I can do the preprocessing in numpy – but thought it was worth pointing out.