title: Is it possible to write a QuerySet method that modifies the dataset but delays evaluation (similar to prefetch_related)?
body: <p>I'm working on a QuerySet class that does something similar to <code>prefetch_related</code> but allows the query to link data that's in an unconnected database (basically, linking records from django apps's database to records in a legacy system, using a shared unique key, something along the links of:</p>
  
  <pre><code>class UserFoo(models.Model):
      ''' Uses the django database &amp; can link to User model '''
      user = models.OneToOneField(User, related_name='userfoo')
      foo_record = models.CharField(
          max_length=32, 
          db_column="foo",
          unique=True
      ) # uuid pointing to legacy db table
  
      @property
      def foo(self):
          if not hasattr(self, '_foo'):
              self._foo = Foo.objects.get(uuid=self.foo_record)
          return self._foo
  
      @foo.setter
      def foo(self, foo_obj):
          self._foo = foo_obj
  </code></pre>
  
  <p>and then</p>
  
  <pre><code>class Foo(models.Model):
      '''Uses legacy database'''
      id = models.AutoField(primary_key=True)
      uuid = models.CharField(max_length=32) # uuid for Foo legacy db table
      …
  
      @property
      def user(self):
          if not hasattr(self, '_user'):
              self._user = User.objects.get(userfoo__foo_record=self.uuid)
          return self._user
  
      @user.setter
      def user(self, user_obj):
          self._user = user_obj
  </code></pre>
  
  <p>Run normally, a query that matches 100 foos (each with, say, 1 user record) will end up requiring 101 queries: one to get the foos, and a hundred for each user record (by doing a look up for the user record by calling the <code>user</code> property on each food).</p>
  
  <p>To get around this, I am making something <em>similar</em> to <code>prefetch_related</code> which pulls all of the matching records for a query by the key, which means I just need one additional query to get the remaining records.</p>
  
  <p>My code looks something like this:</p>
  
  <pre><code>class FooWithUserQuerySet(models.query.QuerySet):
      def with_foo(self):
          qs = self._clone()
          foo_idx = {}
          for record in self.all():
              foo_idx.setdefault(record.uuid, []).append(record)
          users = User.objects.filter(
              userfoo__foo_record__in=foo_idx.keys()
          ).select_related('django','relations','here')
          user_idx = {}
          for user in users:
              user_idx[user.userfoo.foo_record] = user
          for fid, frecords in foo_idx.items():
              user = user_idx.get(fid)
              for frecord in frecords:
                  if user:
                      setattr(frecord, 'user', user)
          return qs
  </code></pre>
  
  <p>This <em>works</em>, but any extra data saved to a <code>foo</code> is lost if the query is later modified — that is, if the queryset is re-ordered or filtered in any way.</p>
  
  <p>I would like a way to create a method that does exactly what I am doing now, but waits until the moment that adjusts whenever the query is evaluated, so that <code>foo</code> records always have a User record.</p>
  
  <p>Some notes:</p>
  
  <ul>
  <li>the example has been highly simplified. There are actually a lot of tables that link up to the legacy data, and so for example although there is a one-to-on relationship between Foo and User, there will be some cases where a queryset will have multiple Foo records with the same key.</li>
  <li>the legacy database is on a different server and server platform, so I can't link the two tables using a database server itself</li>
  <li>ideally I'd like the User data to be cached, so that even if the records are sorted or sliced I don't have to re-run the foo query a second time.</li>
  </ul>
  
  <p>Basically, I don't know enough about the internals of how the lazy evaluation of querysets works in order to do the necessary coding. I have jumped back and forth on the <a href="https://github.com/django/django/blob/master/django/db/models/query.py" rel="nofollow">source code for django.db.models.query</a> but it really is a fairly dense read and I'm hoping someone out there who's worked with this already can offer some pointers.</p>