I have set up an Import.io bulk extract that works great with say, 50 URLs. It literally zips through all of them in seconds. However, when I try to do an extract of 40,000 URLs, the extractor starts very fast for the first thousand or so, and then progressively keeps getting slower every incremental URL. By 5,000 it literally is taking about 4-5 seconds per URL.
One solution that seems to work is breaking them into chunks of 1,000 URLs at a time and doing a separate bulk extract for each. However, this is very time consuming, and requires splicing back together all of the data at the end.
Has anyone experienced this, and if so do they have a more elegant solution?
Thanks,
Mike
Mike, would you mind trying again?
We have worked on the Bulk Extract, now it should be slightly slower at the beginning, but more constant
Possibly 40k are still too many, in which case you may try to split, but I did run 5k+ in a single run
Let me know how it goes!
Import.io bulk extract slows down when more URLs are in list