Alternatively to @albanD’s solution, you could also use DatasetFolder 297, which basically is the underlying class of ImageFolder.
Using this class you can provide your own files extensions and loader to load the samples.
def npy_loader(path):
    sample = torch.from_numpy(np.load(path))
    return sample
    

dataset = datasets.DatasetFolder(
    root='PATH',
    loader=npy_loader,
    extensions=['.npy']
)

If you want to use transformations, you would need to convert the sample tensors to PIL.Images in your loader.