Lessons learned scraping 100B product pages
> As most companies need to extract product data on a daily basis, waiting a couple days for your engineering team to fix any broken spiders isn’t an option. When these situations arise, Scrapinghub uses a machine learning based data extraction tool that we’ve developed as a fallback until the spider has been repaired.I once worked on a spider that crawled article content and I ran into the same problem. I always wanted to try the following solution to it but never had the chance.Assume you have a database of URLs and the fields you've scraped from them in the past (title, author, date, etc). If you ever fail to scrape one of those values from a new URL, here's what you do:- Go back to one of the old URLs where you already have the correct value (let's say it's the title).- Walk through the whole DOM until you find that known title. At each node you will have to remove child nodes except for text, to deal with titles like "Foo Bar" which you want to match against "Foo Bar". So this is going to be an expensive search.- Generate several possible selectors which match the node you walked to (maybe you have ".title", ".title h2", ".content .top h2", etc).- Test each new selector on several other already-crawled pages. If any of the selectors work 100% of the time, there's your new selector.Any thoughts on whether something like this would work?