Hi @lhoestq,
I tried raising num_proc but this didn’t seem to help.
Generally, I think the problem can be abstracted away from the .map function, or even from the need to write data to a new dataset. A simplified version of my problem just requires me to fetch a certain ~300k rows from my large ds_all_utts dataset (and just hold the result in memory - I have enough). If I could do that efficiently, then my problem would be solved.
As you noted in this github issue 1, querying rows of a dataset gets slower as the dataset gets bigger, so that’s my fundamental problem.
Do you have any ideas for how I could more efficiently query my large dataset? You mentioned that you can access 6 million wikipedia articles in less than a minute on your laptop - what format are you saving that dataset, and how are you accessing it?