I need to scrape about 100 websites that are very similar in the content that they provide.
My first doubt. Should be possible to write a generic script to scrape all the 100 websites or in scraping techniques is only possible to write scripts for particular websites. (Dumb question.). I think I should ask what possibility is easier. Write 100 different scripts for each website is hard.
Second question. My primary language is PHP, but after searching here on Stackoverflow I found that one of the most advanced scrapers is "Beautiful Soup" in Python. Should be possible to make calls in PHP to "Beautiful Soup" in Python? Or should be better to do all the script in Python?
Give me some clues on how should I go.
Sorry for my weak english.
Best Regards,
We do something sort of like this with RSS feeds using Python -- we use ElementTree since RSS is usually guaranteed to be well-formed.  Beautiful Soup is probably better suited for parsing HTML.
Insofar as dealing with 100 different sites, try to write an abstraction that works on most of them and transforms the page into a common data-structure you can work with.  Then override parts of the abstraction to handle individual sites which differ from the norm.
Scrapers are usually I/O bound -- look into coroutine libraries like eventlet or gevent to exploit some I/O parallelism and speed up the whole process.
Webscraping Techniques using PHP or Python