A couple of figures from my dataset loading tests using streaming mode from a remote server:




Dataset structure
Sequential Iteration
Parallel iteration




Zip shards
5m 53s
Unsupported


Unarchived images
6m 20s
35s





Dataset volume
20 small shards with a total of just 1398 images.


Server
4 processors, 8 GB RAM, SSD disk, nginx http server, hosted in Amsterdam. Same for both tests.
The “Zip shards” dataset loading script returns compound fsspec URLs that refer to entries inside the zip files, whereas the “Unarchived images” script returns direct URLs to the extracted image files, stored in the same server. Both versions use the same metadata archives in JSON lines format.


Parallel configuration
16 parallel Pools. Pre-processing batch size of 128 items.
The dataset map function is invoked like dataset.map(p_function, batched=True, batch_size=128). The mapping function p_function uses a Pool of 16 workers.


Client: computer with 16 processors, not a TPU.


The conclusions are the same I alluded to in my previous message: dataset streaming could be improved if fsspec retrieval of zip contents could work in parallel, but I couldn’t make it work. If the dataset server stores unarchived files, then parallelization works fine, but file management becomes inconvenient for huge datasets. (In addition, parallelization needs to be handcrafted as part of the pre-processing code, but this could be abstracted into a class).