title: Downloading XML results with LWP::UserAgent in PERL
body: <p>I'm hoping for some assistance with a Perl issue.  </p>
  
  <p>I need to download an XML file that is the result of a query, parse the results, grab the next link from the XML file, download &amp; repeat.</p>
  
  <p>I have been able to download and parse the first result set fine.  </p>
  
  <p>I grab the next URL, but it seems that returned result never changes. I.e.: the second time through the loop, <code>$res-&gt;content</code> is the same as the first time.  Therefore, the value of <code>$url</code> never changes after the first download.  </p>
  
  <p>I'm suspecting it is a scope problem, but I just cannot seem to get a handle on this. </p>
  
  <pre><code>use LWP::UserAgent;
  use HTTP::Cookies;
  use Data::Dumper;
  use XML::LibXML;
  use strict;
  
  my $url = "http://quod.lib.umich.edu/cgi/f/findaid/findaid-idx?c=bhlead&amp;cc=bhlead&amp;type=simple&amp;rgn=Entire+Finding+Aid&amp;q1=civil+war&amp;Submit=Search;debug=xml";
  
  while ($url ne ""){
  
      my $ua = LWP::UserAgent-&gt;new();    
      $ua-&gt;agent('Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)');
      $ua-&gt;timeout(30);
      $ua-&gt;default_header('pragma' =&gt; "no-cache", 'max-age' =&gt; '0');
  
      print "Download URL:\n$url\n\n";
  
      my $res = $ua-&gt;get($url);
  
      if ($res-&gt;is_error) {
          print STDERR __LINE__, " Error: ", $res-&gt;status_line, " ", $res;
          exit;
      } 
  
      my $parser = XML::LibXML-&gt;new(); 
      my $doc = $parser-&gt;load_xml(string=&gt;$res-&gt;content);
  
      #grab the url of the next result set
      $url = $doc-&gt;findvalue('//ResultsLinks/SliceNavigationLinks/NextHitsLink');
  
      print "NEXT URL:\n$url\n\n";
  
  }
  </code></pre>