Cool! I think there’s a huge need for this stuff… and no framework has quite converged on the optimal solution yet (see tensorflow is planning to clean up their disaster of an input pipeline now: https://github.com/tensorflow/tensorflow/issues/7951 29). Pytorch has a good opportunity to improve on everyone else’s mistakes from the start…
Another benefit of a common api for sampling is that you would remove the need to create a new data.Dataset subclass for each public dataset and instead would just need to download and pre-process the data… Would greatly increase your productivity in that area and allow for easily augmenting those datasets.
PS: the ImageFolder class seems kinda on an island in torchvision with the rest of the relevant sampling code (e.g. TensorDataset) in the main repo. I love the idea of torchvision for loading public datasets and pre-trained models… but maybe it would make more sense to separate the general purpose sampling code into a third torchvision folder (e.g. models, datasets, sampling) or develop the sampling solely in the main repo… just my very humble opinion of course.