title: Pulling paginated json data from API using Python
body: <p><strong>The problem:</strong>
  I have an online CRM system with limited reporting functionality. There is a restful API available and I am able to pull the data I need from within PowerBI. The issue is that API paginates the data in either 100s or 200s. At the current state there are over 45k records I need to pull every time I need to refresh the data. In PowerBI it takes over 30 mins. I was thinking (maybe incorrectly) that Python script may be a solution. As I am no Python expert I aided myself with google search and came up with the below simple script to pull all the records.</p>
  <pre><code>import json
  import requests
  
  base_url = &quot;https:##############&quot;
  entity = &quot;payment&quot;
  session = requests.Session()
  session.headers.update({'Authorization' : 'Bearer TOKEN',
              'Content-Type' : 'application/json',
              'Beacon-Application': 'developer_api'})
  
  
  def main_request(base_url, entity):
      r = session.get(base_url + entity)
      return r.json()
  
  def get_pages(response):
      pages = round(response['total'] / 100)
      return pages  
  
  
  data = main_request(base_url, entity)
  results = main_request(base_url, entity)['results']
  
  pages = get_pages(data)
  
  #print(pages)
  #print(results[0])
  
  
  
  payments = []
  
  for page_num in range(pages + 1):
       full_url =base_url + entity + '?page=' + f&quot;{page_num+1}&quot;
       data_ = session.get(full_url).json()
  
       payments.extend(data_['results'])
  
  
  
  print(len(payments))
  
  with open('payments.json', 'w') as f:
         json.dump(payments, f)
  </code></pre>
  <p>The problem is that it still takes relatively long time but also creates a large json file ~300Mb for 45k records.
  Can someone point me in the right direction? What are the best practices?</p>