Does Python garbage collect when Heroku warns about memory quota vastly exceeded (R15)?
Looks like the problem is that I'm not using .iterator() to iterate over the main queryset. Even though I'm freeing the data structures I'm creating after each iteration, the actual query results are all cached.
Unfortunately, I can't use .iterator(), because I use prefetch_related extensively.
I need some kind of hybrid method. I think it will involve processing the top-level queryset in batches. It won't completely have the advantage of a finite number of queries that prefetch_related has, but it will be better than one query per model object.
I have a background task running under Celery on Heroku, that is getting "Error R14 (Memory quota exceeded)" frequently and "Error R15 (Memory quota vastly exceeded)" occasionally. I am loading a lot of stuff from the database (via Django on Postgres), but it should be loading up a big object, processing it, then disposing of the reference and loading up the next big object.
My question is, does the garbage collector know to run before hitting Heroku's memory limit? Should I manually run the gc?
Another thing is that my task sometimes fails, and then Celery automatically retries it, and it succeeds. It should be deterministic. I wonder if something is hanging around in memory after the task is done, and still takes up space when the next task starts. Restarting the worker process clears the memory and lets it succeed. Maybe Django or the DB has some caches that are not cleared?
I'm using standard-2x size. I could go to performance-m or performance-l, but trying to avoid that as it would cost more money.