Doing ds_all_utts.set_format("numpy", columns=["embedding"]) doesn’t help much:
fetching idx 0 of 2732013 took 0.0039942264556884766 seconds
fetching idx 100000 of 2732013 took 0.03906583786010742 seconds
fetching idx 200000 of 2732013 took 0.0478971004486084 seconds
fetching idx 300000 of 2732013 took 0.06415033340454102 seconds
fetching idx 400000 of 2732013 took 0.08065438270568848 seconds
fetching idx 500000 of 2732013 took 0.0943913459777832 seconds
fetching idx 600000 of 2732013 took 0.08204221725463867 seconds
fetching idx 700000 of 2732013 took 0.11144042015075684 seconds
fetching idx 800000 of 2732013 took 0.10394906997680664 seconds
fetching idx 900000 of 2732013 took 0.11541366577148438 seconds
fetching idx 1000000 of 2732013 took 0.1507728099822998 seconds
fetching idx 1100000 of 2732013 took 0.1657857894897461 seconds
fetching idx 1200000 of 2732013 took 0.1772749423980713 seconds
fetching idx 1300000 of 2732013 took 0.17113184928894043 seconds
fetching idx 1400000 of 2732013 took 0.2014932632446289 seconds
fetching idx 1500000 of 2732013 took 0.21463871002197266 seconds
fetching idx 1600000 of 2732013 took 0.2281172275543213 seconds
fetching idx 1700000 of 2732013 took 0.21286630630493164 seconds
fetching idx 1800000 of 2732013 took 0.22928833961486816 seconds
fetching idx 1900000 of 2732013 took 0.25535106658935547 seconds
fetching idx 2000000 of 2732013 took 0.26118993759155273 seconds
fetching idx 2100000 of 2732013 took 0.3325386047363281 seconds
fetching idx 2200000 of 2732013 took 0.3626432418823242 seconds
fetching idx 2300000 of 2732013 took 0.37558627128601074 seconds
fetching idx 2400000 of 2732013 took 0.3952178955078125 seconds
fetching idx 2500000 of 2732013 took 0.3839607238769531 seconds
fetching idx 2600000 of 2732013 took 0.35515308380126953 seconds
fetching idx 2700000 of 2732013 took 0.38485121726989746 seconds

Is this behaviour expected - slower lookup for rows later in the dataset? It says here that random access is O(1) :

So caching the dataset directly on disk can use memory-mapping and pay effectively zero cost with O(1) random access. The default in Datasets is thus to always memory-map dataset on drive.

Does the fact that I’m loading my dataset using load_from_disk have any effect on this?