title: Loop through a PYTHON API call with a limited Row Count per request and Total Rows in the HTTP header?
body: <p>I am very new to Python.  I've written an API call for a large amount of electricity settlement data largely from the API instructions.  The secured API limits requests to 50,000 rows per request.  The API instructions offer a HTTP response header "X-TotalRows" to assist in looping through the entire data set which may be millions of rows.</p>
  
  <p><strong>How do I write the loop for the Python call to append all data, 50k rows at a time?</strong>  I've included my code for the initial dataset (rows 1-50,000) but do not know how to use the HTML Header "X-TotalRows" to append the entire data set, 50k at a time.</p>
  
  <p>The instructions recommend looping through the data using the HTML Header "X-TotalRows" and altering the Start Row parameter to be "1 + rowCount".</p>
  
  <p>This may seem elementary but I've searched and experimented for hours and hours trying to crack this code.  Any help is appreciated.</p>
  
  <pre><code>import http.client, urllib.request, urllib.parse, urllib.error, base64
  
  headers = {
      # Request headers
      'Ocp-Apim-Subscription-Key': 'xyz', 'content-type': 'application/json'
  }
  
  params = urllib.parse.urlencode({
      # Request parameters
      'download': 'true',
      'rowCount': '50000',
      'sort': 'datetime_beginning_ept',
      'order': 'asc',
      'startRow': 1,
      'isActiveMetadata': 'true',
      'fields': 'datetime_beginning_utc, datetime_beginning_ept, pnode_id, pnode_name, voltage, equipment, type, zone, system_energy_price_da, total_lmp_da, congestion_price_da, marginal_loss_price_da, row_is_current, version_nbr',
      #'datetime_beginning_utc': '{string}',
      'datetime_beginning_ept': '1-1-2018 00:00 to 1-31-2018 23:00',
      #'pnode_id': '{number}',
      #'pnode_name': '{string}',
      #'voltage': '{string}',
      #'equipment': '{string}',
      #'type': '{string}',
      'zone': 'aep;comed;pseg'
  })
  
  try:
      conn = http.client.HTTPSConnection('api.pjm.com')
      conn.request("GET", "/api/v1/da_hrl_lmps?%s" % params, "{body}", headers)
      response = conn.getresponse()
      print(response.status, response.reason)
      data = response.read()
      #print(data)
      conn.close()
      file = open('output.txt', 'w')
      s=str(data)
      file.write(s)
      file.close()
      print("Go to ouput.txt")
  except Exception as e:
      print("[Errno {0}] {1}".format(e.errno, e.strerror))
  </code></pre>