Vivek_Reddy:

Right now, I have my Dataset set up so it does an os.walk across all the directories and retrieves a list of all filenames in the init function. This takes several hours.


This seems quite unexpected. I’ve just tested os.walk on a full ImageNet (~1000 folders) and it finishes in ~2s.