Solved by macwiatrak in post #5 
              
              
                Hey Yin, spark to torch dataloader does require some custom work but is fairly easy to build. How I do it is I use torch.utils.data.IterableDataset. Firstly I load all the avro/parquet (as you are working with spark) to a DataReader object which is a generator (where I do some of my custom processin…