title: How do I do paginating in Nokogiri when the number of pages is not readily available?
body: <p>I'm trying to learn how to build a custom screen-scraper that downloads to a local directory, and I'm following "The Great Bastards Guide" by Danny Ngyuen. I have a question that was not addressed in the book: What if the page I am trying to scrape doesn't have the last page number visible in their pagination scheme?</p>
  
  <p>Several sites limit the number of pages that are shown in their results until you get near the end, like Google. How would I define <code>last_page_number</code> when it isn't visible on the main index. Alternatively, is there a better method to take Nokogiri through the pagination when all I have is a next page link? </p>
  
  <pre><code>require 'rubygems'
  require 'nokogiri'
  require 'open-uri'
  require 'fileutils'
  BASE_URL = 'http://explore.data.gov'
  BASE_DIR = '/catalog/raw/?&amp;limit=100&amp;page='
  LOCAL_DIR = 'data-hold/datagov-pages'
  
  FileUtils.makedirs(LOCAL_DIR) unless File.exists?LOCAL_DIR
  
  # get metainfo from first page:
  page = Nokogiri::HTML(open(BASE_URL+BASE_DIR + '1'))
  
  # write the HTML for page 1 to disk
  File.open("#{LOCAL_DIR}/1.html", 'w'){|f| f.write(page.to_html)}
  
  last_page_number = page.css("a.end.lastLink.button")[0]['href'].match(/page=(\d+)/)[1].to_i
  
  puts "Iterating from 2 to #{last_page_number}"
  
  for pg_number in 2..last_page_number do
    puts "Getting #{pg_number}"
    File.open("#{LOCAL_DIR}/#{pg_number}.html", 'w') do |f| 
      f.write( open("#{BASE_URL}#{BASE_DIR}#{pg_number}").read )
    end
  end
  </code></pre>