I need to develop an application that takes as input an url of an e-commerce website and scrap the products titles, prices with the categories and sub-categories.
Scrapy seems like a good solution for scraping data, so my question is how can I tell scrapy where the titles, prices, cat and sub categories are to extract them knowing that websites have different structures and don't really use the same tags?
EDIT: I gotta change my question to this, can't we write a generic spider that takes the start url, allowed domains, and xpath or css selectors as arguments?
Categories and subcategories are usually in the breadcrumbs.
In general the css selector for those will be .breadcrumb a and that will probably work for 80% of modern e-commerce websites.
Scraping products data with categories from e-commerce