I am using scrapy to scrape reviews about books from a site. Till now i have made a crawler and scraped comments of a single book by giving its url as starting url by myself and i even had to give tags of comments about that book by myself after finding it from page's source code. Ant it worked. But the problem is that  till now the work i've done manually i want it to be done automatically. i.e. I want some way that crawler should  be able to find book's page in the website and scrape its comments. I am extracting comments from goodreads and it doesn't provide a uniform method for url's or even tags are also different for different books. Plus i don't want to use Api. I want to do all work by myself. Any help would be appreciated.
HtmlAgilityPack helped me in parsing and reading Xpath for the reviews. It worked :)
Comments scraping without using Api