Hello guys¡
Today I am preparing birds and cars dataset from stanford university to finetune some pretrained imagenet models. I am preparing the dataset in different folders as keeping the entire dataset in RAM needs more than 20 GB and use torch.datasets.ImageFolder
I want to use several threads to upload the dataset as reading from disk is slow so I do not want to have a bottleneck on data loading (I am pretty sure i will have it because cannot use several threads to load as I will be reading from a hard disk).
I have been reading tutorials and other posts. I want to know two things:
-first: Does each worker load a batch or each workers loads samples from the next batch? I think each worker load a batch but I am not very used to multiprocessing package and after reading the code I cannot be certain about that.
-second: I read this in the tutorial https://pytorch.org/docs/stable/notes/faq.html#dataloader-workers-random-seed 24 but I am not really sure on what should I do to not get replicated data.
Thanks.