Answers.Body: <p>After looking through this, it would probably be better off as a <a href="https://github.com/craftcms/cms/issues" rel="nofollow noreferrer">Github issue</a> than a Stack Exchange post, but answering here for posterity.</p>
  
  <p>In short, this comes down to a <a href="https://github.com/craftcms/cms/commit/95f62074ebbcdd328b5403320be22e9c9756e3fd" rel="nofollow noreferrer">change</a> we had to make in Craft 3 to fix some other caching issues where we started to serialize element queries when saving them, so that when retrieving them later to determine if a cache should be busted, we would unserialize and execute the saved element query.</p>
  
  <p>It turns out that serialization requires significantly more memory than not serializing.</p>
  
  <p>In the case of your database, with your custom fields and plugins on both <code>shows</code> and <code>movies</code> sections, it takes about 1.1MB of memory to save each element query inside of your <code>{% cache %}</code> tags.</p>
  
  <p>Given your current template code, with the nested <code>for</code> loops and the number of elements you're accessing in the cache tag, you're looking at about 7,000 element queries that Craft has to track, or 7.7GB of memory - hence the error you're hitting.</p>
  
  <p>It's on our list to completely revamp the template caching engine to help mitigate scenarios like this (as well as other template caching issues people have <a href="https://github.com/craftcms/cms/issues?utf8=%E2%9C%93&amp;q=is%3Aissue+is%3Aopen+cache" rel="nofollow noreferrer">come across</a>)</p>
  
  <p>In the meantime, you've got a few options.</p>
  
  <ol>
  <li><p>Revamp your template logic so you're not executing <a href="https://rob-bell.net/2009/06/a-beginners-guide-to-big-o-notation/" rel="nofollow noreferrer">O(n2)</a> database queries.  That could involve pure Twig/templates or a custom plugin where you can grab exactly the data your template needs from PHP/Querybuilder.</p></li>
  <li><p>Set the <a href="https://docs.craftcms.com/v3/config/config-settings.html#cacheelementqueries" rel="nofollow noreferrer">cacheElementQueries</a> config setting to <code>false</code>. Craft will no long attempt to record/bust template caches, so you can come up with your own logic for busting them (cron job, plugin, etc.).</p></li>
  <li><p>Use a plugin like <a href="https://github.com/mmikkel/CacheFlag-Craft3" rel="nofollow noreferrer">Cache Flag</a> (or a similar one).  It also doesn't attempt to save element queries, but provides a way to bust caches based on keywords.</p></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-22T14:59:58.070Z
Answers.Text: Thanks a lot, Brad, and sorry the late reply. I had no idea that it could rack up like that! I will create a GitHub issue for it.
  
  In the meantime it seems that cacheFlag does the trick.
  
  Out of curiosity – am I going at the structure wrong, are the queries themselves inefficient, or is it just a problem for caching?
Answers.UserDisplayName: KSP
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-22T15:00:57.410Z
Answers.Text: As I'm basically looping through a cinema program that I load from an external XML, I keep the Movies and the actual Shows as different entry types, and relate the shows to the movies. That way I can update the movie information in only one place.
  
  But as I both list the shows in a daily program (fetching titles from the movies they are related to), and list the shows for a movie in a movie view, it's useful to have them separate. This also allows me to override information on specific shows (e.g. if there's Q&A or similar).
Answers.UserDisplayName: KSP
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-18T19:10:01.463Z
Answers.Text: I just updated a theater site to Craft 3 where we use a very similar setup and ran into the same cache issues.  
  
  We have a temporary solution utilizing Cache Flag, but it's far from ideal.I'm in the process of exploring a major rework to how we handle cache on this site to resolve these issues and get things working like before, but I don't want to dig in too far if some changes are coming soon.  
  
  Searched through Github issues and milestones and couldn't find anything directly related to an update on this. Any ETA on cache engine revamp?
Answers.UserDisplayName: jsun
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-17T23:16:28.390Z
Answers.LastActivityDate: 2018-07-17T23:16:28.390Z
Answers.UserDisplayName: Brad Bell
Body: <p>I have a problem with using <code>{% cache %}</code>in Craft 3, although the code runs fine without caching, and it works in Craft 2.</p>
  
  <p>I have narrowed the problem down to the following code:</p>
  
  <pre><code>{% cache %}
    {% set upcomingShows = craft.entries.section('shows').orderBy('showTime asc').limit(null).all() %}
    {% for date, shows in upcomingShows|group("showTime|date_modify('-3 hours')|date('Y-m-d')") %}
      {% for show in shows %}
        {% set movie = craft.entries.relatedTo({ targetElement: show }).one() %}
      {% endfor %}
    {% endfor %}
  {% endcache %}
  </code></pre>
  
  <p>It gives me the following fatal error:</p>
  
  <pre><code>Allowed memory size of 268435456 bytes exhausted (tried to allocate 71002178 bytes)
  </code></pre>
  
  <p>If I skip the Twig <code>|group</code>filter, the cache works again:</p>
  
  <pre><code>{% cache %}
    {% set upcomingShows = craft.entries.section('shows').orderBy('showTime asc').limit(null).all() %}
    {% for shows in upcomingShows %}
      {% for show in shows %}
        {% set movie = craft.entries.relatedTo({ targetElement: show }).one() %}
      {% endfor %}
    {% endfor %}
  {% endcache %}
  </code></pre>
  
  <p>(I first thought the problem was the <code>relatedTo()</code> function, but it turns out it's the <code>|group</code> is what triggers it, without it it works also with the  <code>relatedTo()</code>.)</p>
  
  <p>Is this a bug in the cache, or is there a way around it?</p>
  
  <p><strong>UPDATE:</strong> After some more testing, it seems like it works <em>either</em> with the <code>|group</code> filter or the <code>relatedTo()</code> function, but not both at the same time.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-14T20:27:01.277Z
Comments.Text: How many entries are returned by the `upcomingShows` query?
Comments.UserDisplayName: Mats Mikkel Rummelhoff
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-15T10:25:16.467Z
Comments.Text: Approximately 50 or so, nothing massive
Comments.UserDisplayName: KSP
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-16T23:12:02.433Z
Comments.Text: If you want to send a database dump over to support@craftcms.com and reference this SE post, we can look into it.
Comments.UserDisplayName: Brad Bell
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-16T23:24:19.427Z
Comments.Text: @BradBell Thanks a million, I’ll send it over tomorrow.
Comments.UserDisplayName: KSP
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-17T08:57:00.993Z
Comments.Text: @BradBell I get an error trying to back up the database from the admin. Will a dump from phpMyAdmin do?
Comments.UserDisplayName: KSP
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-17T18:47:12.453Z
Comments.Text: @KSP That's actually the same
Comments.UserDisplayName: Robin Schambach
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-17T19:28:10.793Z
Comments.Text: Thanks, @RobinSchambach, figured it out and sent both versions + sample output from the part I’m trying to cache.
Comments.UserDisplayName: KSP
Tags: <templating><craft3><cache>
Title: Caching in Craft 3 causes out of memory when including grouped for loop
ThreadText: # Caching in Craft 3 causes out of memory when including grouped for loop
  
  (Asked by: username_0 on 2018-07-14)
  
  I have a problem with using `{% cache %}`in Craft 3, although the code runs fine without caching, and it works in Craft 2.
  
  I have narrowed the problem down to the following code:
  
  ```
  {% cache %}
    {% set upcomingShows = craft.entries.section('shows').orderBy('showTime asc').limit(null).all() %}
    {% for date, shows in upcomingShows|group("showTime|date_modify('-3 hours')|date('Y-m-d')") %}
      {% for show in shows %}
        {% set movie = craft.entries.relatedTo({ targetElement: show }).one() %}
      {% endfor %}
    {% endfor %}
  {% endcache %}
  
  ```
  
  It gives me the following fatal error:
  
  ```
  Allowed memory size of 268435456 bytes exhausted (tried to allocate 71002178 bytes)
  
  ```
  
  If I skip the Twig `|group`filter, the cache works again:
  
  ```
  {% cache %}
    {% set upcomingShows = craft.entries.section('shows').orderBy('showTime asc').limit(null).all() %}
    {% for shows in upcomingShows %}
      {% for show in shows %}
        {% set movie = craft.entries.relatedTo({ targetElement: show }).one() %}
      {% endfor %}
    {% endfor %}
  {% endcache %}
  
  ```
  
  (I first thought the problem was the `relatedTo()` function, but it turns out it's the `|group` is what triggers it, without it it works also with the `relatedTo()`.)
  
  Is this a bug in the cache, or is there a way around it?
  
  **UPDATE:** After some more testing, it seems like it works *either* with the `|group` filter or the `relatedTo()` function, but not both at the same time.
  
  **Tags:** templating, craft3, cache
  
  > **Comments:**
  > **username_1**: How many entries are returned by the `upcomingShows` query?  
  > **username_0**: Approximately 50 or so, nothing massive  
  > **username_2**: If you want to send a database dump over to support@craftcms.com and reference this SE post, we can look into it.  
  > **username_0**: @username_2 Thanks a million, I’ll send it over tomorrow.  
  > **username_0**: @username_2 I get an error trying to back up the database from the admin. Will a dump from phpMyAdmin do?  
  > **username_3**: @username_0 That's actually the same  
  > **username_0**: Thanks, @username_3, figured it out and sent both versions + sample output from the part I’m trying to cache.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-07-17. Score: 2 | Accepted answer
  
  After looking through this, it would probably be better off as a [Github issue](https://github.com/craftcms/cms/issues) than a Stack Exchange post, but answering here for posterity.
  
  In short, this comes down to a [change](https://github.com/craftcms/cms/commit/95f62074ebbcdd328b5403320be22e9c9756e3fd) we had to make in Craft 3 to fix some other caching issues where we started to serialize element queries when saving them, so that when retrieving them later to determine if a cache should be busted, we would unserialize and execute the saved element query.
  
  It turns out that serialization requires significantly more memory than not serializing.
  
  In the case of your database, with your custom fields and plugins on both `shows` and `movies` sections, it takes about 1.1MB of memory to save each element query inside of your `{% cache %}` tags.
  
  Given your current template code, with the nested `for` loops and the number of elements you're accessing in the cache tag, you're looking at about 7,000 element queries that Craft has to track, or 7.7GB of memory - hence the error you're hitting.
  
  It's on our list to completely revamp the template caching engine to help mitigate scenarios like this (as well as other template caching issues people have [come across](https://github.com/craftcms/cms/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+cache))
  
  In the meantime, you've got a few options.
  
  1. Revamp your template logic so you're not executing [O(n2)](https://rob-username_2.net/2009/06/a-beginners-guide-to-big-o-notation/) database queries. That could involve pure Twig/templates or a custom plugin where you can grab exactly the data your template needs from PHP/Querybuilder.
  2. Set the [cacheElementQueries](https://docs.craftcms.com/v3/config/config-settings.html#cacheelementqueries) config setting to `false`. Craft will no long attempt to record/bust template caches, so you can come up with your own logic for busting them (cron job, plugin, etc.).
  3. Use a plugin like [Cache Flag](https://github.com/mmikkel/CacheFlag-Craft3) (or a similar one). It also doesn't attempt to save element queries, but provides a way to bust caches based on keywords.
  
  > **Comments:**
  > **username_0**: Thanks a lot, username_2, and sorry the late reply. I had no idea that it could rack up like that! I will create a GitHub issue for it.  In the meantime it seems that cacheFlag does the trick.  Out of curiosity – am I going at the structure wrong, are the queries themselves inefficient, or is it just a problem for caching?  
  > **username_0**: As I'm basically looping through a cinema program that I load from an external XML, I keep the Movies and the actual Shows as different entry types, and relate the shows to the movies. That way I can update the movie information in only one place.  But as I both list the shows in a daily program (fetching titles from the movies they are related to), and list the shows for a movie in a movie view, it's useful to have them separate. This also allows me to override information on specific shows (e.g. if there's Q&A or similar).  
  > **username_4**: I just updated a theater site to Craft 3 where we use a very similar setup and ran into the same cache issues.    We have a temporary solution utilizing Cache Flag, but it's far from ideal.I'm in the process of exploring a major rework to how we handle cache on this site to resolve these issues and get things working like before, but I don't want to dig in too far if some changes are coming soon.    Searched through Github issues and milestones and couldn't find anything directly related to an update on this. Any ETA on cache engine revamp?  
  
  ---
  
  