yonti:

then compared the same configuration(batch size 256 / 10 workers) with file descriptor vs file system


So you see a 2x slowdown using a DataLoader with batch_size=256 and num_workers=10 in comparison to “file system”?
Could you post some more information on your use cases as well as your system and when you are seeing a slowdown (how large etc.)?