ptrblck:

I’m not sure I understand the use case completely.
Are you using an “outside” DataLoader with a custom collate_fn , where you are creating additional DataLoaders ?
If so, what the use case and why do you need to initialize DataLoaders inside a DataLoader ?


Yes! I have a data loader outside and the collate function is creating data loaders. That is exactly correct.
I have 64 image classes (with 600 examples) and I am sampling 20 images from 5 classes at a time (N-way K-shot few shot classification). 1 meta-batch is 20 images from 5 classes. So I am trying to create multiple meta-batches in parallel so that it’s quicker but something I did made this really slow. Now it’s taking over a day to run compared to my old single meta-batch code that ran in 4 hours.
I’m not sure why it’s so slow. I tried using the pin memory to speed things up but I get the error mentioned above.