For Want of a JOIN
I&#x27;m not sure I understand how a JOIN would have fixed this problem.  That is, if each chunk is fetching 1k rows, and you&#x27;re doing 50 simultaneous chunks, then you&#x27;re doing a 50,000 row query, and that&#x27;s ALSO going to be extremely slow, in terms of exclusive database contention (less of an issue with bigquery) and result set memory usage (definitely still a huge issue for python). In fact, one of my most frequent pieces of feedback to junior engineers who are just working on a larger backend for the first time is &quot;this query tries to fetch too much data at once, it will take too long and use way more memory then it needs to, please use find_each to automatically batch the query so that we balance memory usage and database contention&quot;. Indeed, Rails by default will use the exact same batching strategy the junior engineer chose in this case:  fetch 1k items, process those items, and move on to the next 1k items. I understand that the author rankles about things not being done the &quot;right way&quot; with JOIN, but I question whether their focus on &quot;best practices&quot; is preventing them from seeing the optimization forest (split things up into parallel background tasks, don&#x27;t try to keep the entire dataset in memory at once) for the &quot;doing things right&quot; trees (use JOIN)