title: How to use Scrapy to crawl data on the second level of a Page
body: <p>I want to use scrapy spider to get data (question title + content &amp; answer) from all posts of the following website:</p>
  
  <p><a href="https://forums.att.com/t5/custom/page/page-id/latest-activity/category-id/Customer_Care/page/1?page-type=latest-solutions-topics" rel="nofollow noreferrer">https://forums.att.com/t5/custom/page/page-id/latest-activity/category-id/Customer_Care/page/1?page-type=latest-solutions-topics</a></p>
  
  <p>The problem is I just dont know how to make it first to follow the link of the post and then to crawl the data of all 15 posts/site. </p>
  
  <p>{import scrapy</p>
  
  <p>class ArticleSpider(scrapy.Spider):
      name = "post"
      start_urls = ['<a href="https://forums.att.com/t5/Data-Messaging-Features-Internet/Throttling-for-unlimited-data/m-p/4805201#M73235" rel="nofollow noreferrer">https://forums.att.com/t5/Data-Messaging-Features-Internet/Throttling-for-unlimited-data/m-p/4805201#M73235</a>']</p>
  
  <pre><code>def parse(self, response):
      SET_SELECTOR = 'body'
      for post in response.css(SET_SELECTOR):
  
          # Selector for title, content and answer
          TITLE_SELECTOR = '.lia-message-subject h5 ::text'
          CONTENT_SELECTOR = '.lia-message-body-content'
          ANSWER_SELECTOR = '.lia-message-body-content'
  
          yield {
  
              # [0].extract() = extract_first()
              'Qtitle': post.css(TITLE_SELECTOR)[0].extract(),
              'Qcontent': post.css(CONTENT_SELECTOR)[0].extract(),
              'Answer': post.css(ANSWER_SELECTOR)[1].extract(),
          }
      # Running through all 173 pages
      NEXT_PAGE_SELECTOR = '.lia-paging-page-next a ::attr(href)'
      next_page = response.css(NEXT_PAGE_SELECTOR).extract_first()
      if next_page:
          yield scrapy.Request(
              response.urljoin(next_page),
              callback=self.parse
          )}
  </code></pre>
  
  <p>I hope u can help me out. Thanks in advance!</p>