title: How to use Scrapy to crawl Data which needs to be parsed before?
body: <p>I want to use scrapy spider to get data about Dota 2 from <a href="http://opendota.com" rel="nofollow noreferrer">opendota.com</a>.</p>
  
  <p>I would like to start for example at <code>matchID = 3610252548</code>, crawl all the data(<code>matchid</code>, <code>duration</code>, <code>region</code>, <code>totalkills</code> Radiant and <code>totalkillsDire</code>) i need and then increase the <code>matchID +1</code> and crawl and so on.</p>
  
  <p>Problems:</p>
  
  <ol>
  <li>The data needs to be requested over a <code>MatchID</code> (<a href="https://www.opendota.com/request" rel="nofollow noreferrer">https://www.opendota.com/request</a>) so it takes a few seconds (ca. 15-20) till the data is available.</li>
  <li><p>I would like to restrict the amount of match requests to 3/min to keep the load small on the Server. </p>
  
  <pre><code>import scrapy
  
  class OpendotaSpider(scrapy.Spider):
  # name der Spinne
  name = "opendota"
  start_urls = ['https://www.opendota.com/matches/3607703223']
  
  
  def parse(self, response):
  for post in response.css('body'):
  
  yield {
  
      # [0].extract() = extract_first()
      'Winner': post.css('.matchinfo span ::text').extract_first(),
      'Matchid': post.css('.additionalInfo li ::text').extract_first(),
      'Duration': post.css('.duration span ::text').extract()
  }
  </code></pre>
  
  <p>This is the little bit of Code i have until now. Thanks for your help!</p></li>
  </ol>