Lessons learned scraping 100B product pages
> As most companies need to extract product data on a daily basis, waiting a couple days for your engineering team to fix any broken spiders isn’t an option. When these situations arise, Scrapinghub uses a machine learning based data extraction tool that we’ve developed as a fallback until the spider has been repaired.<p>I once worked on a spider that crawled article content and I ran into the same problem. I always wanted to try the following solution to it but never had the chance.<p>Assume you have a database of URLs and the fields you've scraped from them in the past (title, author, date, etc). If you ever fail to scrape one of those values from a <i>new</i> URL, here's what you do:<p>- Go back to one of the old URLs where you already have the correct value (let's say it's the title).<p>- Walk through the whole DOM until you find that known title. At each node you will have to remove child nodes except for text, to deal with titles like "Foo <span>Bar</span>" which you want to match against "Foo Bar". So this is going to be an expensive search.<p>- Generate several possible selectors which match the node you walked to (maybe you have ".title", ".title h2", ".content .top h2", etc).<p>- Test each new selector on several other already-crawled pages. If any of the selectors work 100% of the time, there's your new selector.<p>Any thoughts on whether something like this would work?