title: Scrapy: Modify rules for scraping web page
body: <p>I've started to use scrapy for a project of mine to scrape data off a tennis website. <a href="http://www.tennisinsight.com/player_activity.php?player_id=1" rel="nofollow">Here</a> is an example page that I want to scrape data off. As you can see, I want to scrape data for a tennis player. I need to recursively go through the entire page and gather 'Match Stats' (Theres a link titled 'Match Stats' next to every match) for a player's matches. I've already written code to parse data from the opened match stats popup. All I need to do now is open these match stats pages through the initial spider.</p>
  
  <p>In all the examples I've read up on, we can write rules to navigate scrapy to the different urls that need scraping. In my case, I just want to write a rule to the different match stats links. However, if you saw the page I want to scrape, 'Match Stats' links are in the following format: <code>javascript:makePopup('match_stats_popup.php?matchID=183704502')</code>. As I've read online (I might be wrong!), scrapy can't deal with javascript and hence cant 'click' on that link. However, since the links are javascript popups, its possible to add the <code>match_stats_popup.php?matchID=183704502</code> part of the link to the main url to get a standard html page:</p>
  
  <pre><code>http://www.tennisinsight.com/match_stats_popup.php?matchID=183704502
  </code></pre>
  
  <p>I am hoping I could modify the rules before scraping. In summary, I just want to find the links that are of the type: <code>javascript:makePopup('match_stats_popup.php?matchID=183704502</code>, and modify them so that they are now of the type <code>http://www.tennisinsight.com/match_stats_popup.php?matchID=183704502</code></p>
  
  <p>This is what I've written in the rules so far, which doesnt open any pages:</p>
  
  <pre><code>rules = (
      Rule(SgmlLinkExtractor(allow='/match_stats_popup.php?matchID=\d+'),
          'parse_match', follow=True,
      ),
  )
  </code></pre>
  
  <p><code>parse_match</code> is the method which parses data from the opened match stats popup.</p>
  
  <p>Hope my problem is clear enough!</p>