Scraping products data with categories from e-commerce
Categories and subcategories are usually in the breadcrumbs.
In general the css selector for those will be .breadcrumb a and that will probably work for 80% of modern e-commerce websites.
I need to develop an application that takes as input an url of an e-commerce website and scrap the products titles, prices with the categories and sub-categories.
Scrapy seems like a good solution for scraping data, so my question is how can I tell scrapy where the titles, prices, cat and sub categories are to extract them knowing that websites have different structures and don't really use the same tags?
EDIT: I gotta change my question to this, can't we write a generic spider that takes the start url, allowed domains, and xpath or css selectors as arguments?