I have set up an Import.io bulk extract that works great with say, 50 URLs. It literally zips through all of them in seconds. However, when I try to do an extract of 40,000 URLs, the extractor starts very fast for the first thousand or so, and then progressively keeps getting slower every incremental URL. By 5,000 it literally is taking about 4-5 seconds per URL.
One solution that seems to work is breaking them into chunks of 1,000 URLs at a time and doing a separate bulk extract for each. However, this is very time consuming, and requires splicing back together all of the data at the end.
Has anyone experienced this, and if so do they have a more elegant solution?
Thanks,
Mike
One slightly less elegant solution would be to create a crawler. And before you run it insert the the 10k URLs in the "where to start crawling" box.
Under advanced options set the crawl depth to zero, that way you will only get the pages you put in the where to start crawling input box.
That should do the trick. Plus the cawler has a bunch of other options like wait between pages and concurrent pages etc.
Import.io bulk extract slows down when more URLs are in list