title: Scrapy: Dynamically generate rules for each start_url
body: <p>I have created a spider which is supposed to crawl multiple websites and I need to define different rules for each URL in the start_url list.</p>
  
  <pre><code>start_urls = [
      "http://URL1.com/foo"
      "http://URL2.com/bar"
  ]
  
  rules = [
      Rule (LinkExtractor(restrict_xpaths=("//" + xpathString+"/a")), callback="parse_object", follow=True)
  ]
  </code></pre>
  
  <p>The only thing that needs to change in the rule is the xpath string for restrict_xpath. I've already come up with a function that can get the xpath I want dynamically from any website.
  I figured I can just get the current URL that the spider will be scraping and pass it through the function and then pass the resulting xpath to the rule.</p>
  
  <p>Unfortunately, I've been searching and it seems that this isn't possible since scrapy utilizes a scheduler and compiles all the start_urls and rules right from the start. Is there any workaround to achieve what I'm trying to do?</p>