title: Sequentially run through a MYSQL table with 1,000,000 records?
body: <p>I've built a site that has about 1,000,000 products in it. I want to build a set of site maps which will contain links to all the products. I understand that site maps may contain up to 50,000, links, so my plan is to build 20 such site maps.</p>
  
  <p>I want to do accomplish this with a series of SQL statements. My plan: 1,000 SQL statements, each of which would pull up 1,000 products. The SQL statements would be along the following lines:</p>
  
  <pre><code>SELECT name, category, catalog_number FROM product ORDER BY id LIMIT    0, 1000
  SELECT name, category, catalog_number FROM product ORDER BY id LIMIT 1000, 1000
  SELECT name, category, catalog_number FROM product ORDER BY id LIMIT 2000, 1000
  SELECT name, category, catalog_number FROM product ORDER BY id LIMIT 3000, 1000
  </code></pre>
  
  <p>And so on.</p>
  
  <p>The problem is that as the first argument to LIMIT (the offset) grows, the amount of time each sql takes to execute grows very quickly! To be specific, when the first argument to LIMIT was anything less than 28000, the statement executed in under .3 seconds. But when I tried:</p>
  
  <pre><code>SELECT name, category, catalog_number FROM product ORDER BY id LIMIT 29000, 1000
  </code></pre>
  
  <p>the time taken to execute the statement jumped to 30 seconds! And every statement after that took a similarly long amount of time to execute.</p>
  
  <p>Another thing I tried was to increase the <em>second</em> argument to LIMIT -- but the same problem occurred.</p>
  
  <p>Any assistance would be welcome. Thank you.</p>