I’m struggling to find an elegant way to do this. Keras allows this functionality by simply passing an argument to the generator.
I’ve been following this suggestion Manually set number of batches in DataLoader 41
However, I believe this solution is loading extra samples that end up not being used. I’d like to cycle through all the samples, across different epochs . For example if I have 1000 samples and I want 100 steps per epoch (batch size 1) , in 10 epochs the loader will cycle through all the samples and on epoch 11 it will load the first sample again. However following the solution from the other topic I believe some samples will be skipped and will never be used, because the dataloader already has loaded them but they end up not being trained on.