Logan_Nielsen:

I’m assuming __getitem__ eventually gets called on the dataset so as long as what gets returned there is the desired result then preprocessing is occurring.


Usually you would add the proprocessing in the __getitem__ and would return the already preprocessed batch of samples. Note that this thread had a specific requirement:

Consequently, I do not want to preload future batches. In other words, I want the method __next__ of dataloader.batch_sampler to be called only when the same method of dataloader itself is explicitly called.

So I’m unsure if you are working with a similar requirement or if it’s a general question.



 Logan_Nielsen:

I’m thinking about using multiple threads to help the word go faster. Do I need to worry about how these processes affect what Pytorch is doing in preparing the data? (e.g. Pytorch thinks a thread isn’t being used when really I’ve just kicked up a process on it.)


You should be able to use multi-threading inside a process, i.e. in each worker of your DataLoader. Could you describe your concern a bit more, please? I.e. are you seeing any specific issues in your current use case?