title: How to deal with a large queryset in Django
body: <p>I need to process a queryset in Django of about 15 000 rows. I basically loop through the queryset and create a new object from each object in the queryset. The new object contains a number of calculated fields, as shown below. The result of the report is then saved as an Excel file, with one row for every CalcEntry. </p>
  
  <pre><code>def get_report(report_date):
      for db_entry in my_large_queryset:
          yield CalcEntry(db_entry, report_date)
  
  class CalcEntry(object):
      def __init__(self, db_entry, report_date):
          self.db_entry = db_entry
          self.report_date = report_date
  
      @property
      def calc1(self):
          if self.db_entry.value_date &gt; self.report_date
              return self.db_entry.value
          return 0
  
      @property
      def calc2(self):
          #... There's about 20 of these calcs in the CalcEntry class
  </code></pre>
  
  <p>Creating the report takes about 20s, so I'm close to getting a timeout error since it is running on Heroku (Heroku times out after 30s). However, <strong>just looping through the queryset takes a lot of time</strong>, without even creating the CalcEntry class. I read that looping through large querysets are not recommended, but I thought large would be more than 15 000.</p>
  
  <p>It looks like instead of looping through the queryset, I should use the values() method. Consequently, I thought of the following solutions, but all of them requires a lot of work so I'm hoping there is another better way of dealing with this problem:</p>
  
  <ol>
  <li>Try to calculate the fields using F expressions and then use values() on the queryset. This will result in a monster query though.</li>
  <li>Run the calcs in the background and save it in a new db table at a specific report date. Again in this case I can then use the values() method to get a list of dictionaries that can be used to create the Excel file.</li>
  <li>Do some sort of raw SQL calc, but I would want to avoid this if possible (I'm using postgreSQL)</li>
  </ol>
  
  <p>How do people generally deal with large querysets and what other alternative to looping over the queryset are available?</p>