I did a few tests on a TPU v3-8 and 8,320 images split over 50 shards (see my loading script 1).




Mode
load data + map (extract img dimensions)
Iterate through dataset




Local
24.2 s
1.62 s


Local (cached)
0.023 s
same


Streaming
0.072 s
26m 47s



Obviously in streaming mode the task is performed only when iterating.
Then I checked the time in streaming if I didn’t open the file.




Mode
open img (extract img dimensions)
Time to iterate
Iterations per second




Streaming
No
27.4 s
303.6


Streaming
Yes
26m 47s
5.2



In streaming mode, we don’t take advantage of multiprocessing during mapping. So then I tested the following:

local dataset where img is fsspec format pointing to url
map to stream files from multiple processes

Results:

it’s actually not faster, probably because it uses requests which reads only from one stream at a time
it sometimes results in errors (similar to when I use datasets-cli to download lots of large archives) due to streams having been closed. I think they are open concurrently but read only 1 at a time, resulting sometimes in timeout.

Conclusions:

streaming data is currently a bottleneck for our application (we could only iterate through 0.5M images per day)
using local copies require very large disks. Also data is duplicated as we have archive + uncompressed data + arrow files. It could be improved a bit by removing compressed files (maybe after this line)
there’s also the webdataset that looks pretty cool (the videos are nice as well). Interestingly they use the tar format. I’m gonna test it out.