title: How to pass parameters to scrapy spiders in program?
body: <p>I am a newbie of python and scrapy. I used the method in this blog <a href="http://kirankoduru.github.io/python/multiple-scrapy-spiders.html" rel="nofollow">Running multiple scrapy spiders programmatically</a> to run my spiders in a flask app.Here is the code:</p>
  
  <pre><code># list of crawlers
  TO_CRAWL = [DmozSpider, EPGDspider, GDSpider]
  
  # crawlers that are running 
  RUNNING_CRAWLERS = []
  
  def spider_closing(spider):
      """
      Activates on spider closed signal
      """
      log.msg("Spider closed: %s" % spider, level=log.INFO)
      RUNNING_CRAWLERS.remove(spider)
      if not RUNNING_CRAWLERS:
          reactor.stop()
  
  # start logger
  log.start(loglevel=log.DEBUG)
  
  # set up the crawler and start to crawl one spider at a time
  for spider in TO_CRAWL:
      settings = Settings()
  
      # crawl responsibly
      settings.set("USER_AGENT", "Kiran Koduru (+http://kirankoduru.github.io)")
      crawler = Crawler(settings)
      crawler_obj = spider()
      RUNNING_CRAWLERS.append(crawler_obj)
  
      # stop reactor when spider closes
      crawler.signals.connect(spider_closing, signal=signals.spider_closed)
      crawler.configure()
      crawler.crawl(crawler_obj)
      crawler.start()
  
  # blocks process; so always keep as the last statement
  reactor.run()
  </code></pre>
  
  <p>Here is my spiders code:</p>
  
  <pre><code>class EPGDspider(scrapy.Spider):
  name = "EPGD"
  allowed_domains = ["epgd.biosino.org"]
  term = "man"
  start_urls = ["http://epgd.biosino.org/EPGD/search/textsearch.jsp?textquery="+term+"&amp;submit=Feeling+Lucky"]
  MONGODB_DB = name + "_" + term
  MONGODB_COLLECTION = name + "_" + term
  
  def parse(self, response):
      sel = Selector(response)
      sites = sel.xpath('//tr[@class="odd"]|//tr[@class="even"]')
      url_list = []
      base_url = "http://epgd.biosino.org/EPGD"
  
      for site in sites:
          item = EPGD()
          item['genID'] = map(unicode.strip, site.xpath('td[1]/a/text()').extract())
          item['genID_url'] = base_url+map(unicode.strip, site.xpath('td[1]/a/@href').extract())[0][2:]
          item['taxID'] = map(unicode.strip, site.xpath('td[2]/a/text()').extract())
          item['taxID_url'] = map(unicode.strip, site.xpath('td[2]/a/@href').extract())
          item['familyID'] = map(unicode.strip, site.xpath('td[3]/a/text()').extract())
          item['familyID_url'] = base_url+map(unicode.strip, site.xpath('td[3]/a/@href').extract())[0][2:]
          item['chromosome'] = map(unicode.strip, site.xpath('td[4]/text()').extract())
          item['symbol'] = map(unicode.strip, site.xpath('td[5]/text()').extract())
          item['description'] = map(unicode.strip, site.xpath('td[6]/text()').extract())
          yield item
  
      sel_tmp = Selector(response)
      link = sel_tmp.xpath('//span[@id="quickPage"]')
  
      for site in link:
          url_list.append(site.xpath('a/@href').extract())
  
      for i in range(len(url_list[0])):
          if cmp(url_list[0][i], "#") == 0:
              if i+1 &lt; len(url_list[0]):
                  print url_list[0][i+1]
                  actual_url = "http://epgd.biosino.org/EPGD/search/"+ url_list[0][i+1]
                  yield Request(actual_url, callback=self.parse)
                  break
              else:
                  print "The index is out of range!"
  </code></pre>
  
  <p>As you can see, there is a parameter <code>term = 'man'</code> in my code, and it's part of my <code>start urls</code>. I don't want this parameter to be fixed, so I wonder how can I give the <code>start url</code> or the parameter <code>term</code> dynamically in my program? Just like running a spider in command line there is a way can pass parameter as below:</p>
  
  <pre><code>class MySpider(BaseSpider):
  
      name = 'my_spider'    
  
      def __init__(self, *args, **kwargs): 
        super(MySpider, self).__init__(*args, **kwargs) 
  
        self.start_urls = [kwargs.get('start_url')] 
  And start it like: scrapy crawl my_spider -a start_url="http://some_url"
  </code></pre>
  
  <p>Can anybody tell me how to deal with this? </p>