I’ve been doing some tests on the same dataset.
First, I tried to make parallel requests for image loading and decoding. I used batched mode to retrieve a number of items sequentially, and then a simple Pool to load the images from the remote fsspec URLs that point to zip files. Unfortunately, it didn’t work: processes stalled and deadlocked.
Seeing that the backend of the fsspec URLs is HTTP, and that the fsspec library is capable of concurrent downloading of HTTP files, I tried to load several images at once using cat, as demonstrated here 3. That doesn’t work either: since the fsspec file is zip::, the filesystem is resolved as a Zip Archive even though the backend is HTTP. cat is not supported for lists in zip archives.
Then I used URLs that point to extracted images instead of entries inside zip files. I set up an http server pointing to an unarchived checkpoint of the dataset, and updated my loading script to return those URLs. This worked and was very fast, but the times are not comparable because the server was in the same LAN. More interestingly, a parallel Pool was able to retrieve images concurrently in this setup.
I’m going to repeat this last test in a more realistic scenario, loading images from a distant server, and then I’ll post some figures.
Meanwhile, I have the following questions on how to proceed.

Are there any hints to make fsspec work in parallel? We could implement an async zip filesystem, but that looks like a lot of hassle.
Does it make sense to store the files in the dataset in unarchived format, instead of inside zips?

Thank you!