I believe the easiest way is to implement a Dataset class with the csv file you described. The class only needs to provide a method to fetch data a particular given index. http://pytorch.org/docs/master/data.html#torch.utils.data.Dataset 58