I'd like to scrape contact info from about 1000-2000 different restaurant websites. Almost all of them have contact information either on the homepage or on some kind of "contact" page, but no two websites are exactly alike (i.e., there's no common pattern to exploit). How can I reliably scrape email/phone # info from sites like these without specifically pointing the Python script to a particular element on the page (i.e., the script needs to be structure agnostic, since each site has a unique HTML structure, they don't all have, e.g., their contact info in a "contact" div).
I know there's no way to write a program that will be 100% effective, I'd just like to maximize my hit rate.
Any guidance on this—where to start, what to read—would be much appreciated.
Thanks.
In most countries the telephone number follows one of a very few well defined patterns that can be matched with a simple regexp - likewise email addresses have an internationally recognised format - simply scrape the homepage, contacts or contact us page and then parse with regular expressions - you should easily achieve better than 90% accuracy.
Alternatively of course you simply submit the restaurant name and town to the local equivalent of the Yellow Pages web site.
Scraping Contact Information from Several Unique Sites with Python