Answers.Body: <p>We're also seeing this issue on Laravel Forge. For us it mostly happens after a Craft update is deployed, so it might have something to do with the queue runner still having the previous Craft version loaded.</p>
  <p>While I'm not sure what causes it, as a workaround you can ensure that the queue listener restarts periodically and/or after deployments.</p>
  <p>There are some problems with this approach. For sites with a lot of traffic and queue jobs, restarting the queue runner may result in the queue getting backed up. Also, when a queue job is running while the queue runner is being forcefully restarted, it can cause the job to fail. To address this, we also have a nightly cron job that runs <code>queue/retry</code> to automatically retry failed queue jobs. But for small and medium sites that don't have tons of queue jobs coming in 24/7, this approach works quite well.</p>
  <h2>Restarting the queue runner periodically</h2>
  <p>Create a cron job in Forge, in the server settings under <strong>Scheduler</strong>:</p>
  <ul>
  <li><strong>Command:</strong> <code>sudo supervisorctl restart all</code></li>
  <li><strong>User:</strong> <code>forge</code></li>
  <li><strong>Frequency:</strong>  Nightly (or choose whatever you like)</li>
  </ul>
  <p>On newly provisioned servers, this should work without any further adjustments, because Forge now allows the <code>forge</code> user to execute <code>supervisorctl</code> commands without a password. On older servers, make sure the following snippet is in your <code>/etc/sudoers.d/supervisorctl</code>:</p>
  <pre><code>forge ALL=NOPASSWD: /usr/bin/supervisorctl reload
  forge ALL=NOPASSWD: /usr/bin/supervisorctl reread
  forge ALL=NOPASSWD: /usr/bin/supervisorctl restart *
  forge ALL=NOPASSWD: /usr/bin/supervisorctl start *
  forge ALL=NOPASSWD: /usr/bin/supervisorctl status *
  forge ALL=NOPASSWD: /usr/bin/supervisorctl status
  forge ALL=NOPASSWD: /usr/bin/supervisorctl stop *
  forge ALL=NOPASSWD: /usr/bin/supervisorctl update *
  forge ALL=NOPASSWD: /usr/bin/supervisorctl update
  </code></pre>
  <h2>Restarting the queue runner after deployments</h2>
  <p>To restart the queue runner after every deployment, add this to the end of your deployment script:</p>
  <pre><code>sudo supervisorctl restart all
  </code></pre>
  <p>This will restart <strong>all</strong> daemons. Alternatively, you can also find the ID of your queue runner (use <code>sudo supervisorctl status</code> to list all active daemons) and only restart that in your deployment (replace <code>&lt;daemon-id&gt;</code> with the actual ID):</p>
  <pre><code>sudo supervisorctl restart &lt;daemon-id&gt;
  </code></pre>
  <p>On newer servers, if you're not using website isolation, this should work by default. If you <em>are</em> using website isolation, you need to allow the website user to run <code>supervisorctl</code> without a password, since you can't (or rather, shouldn't) provide the password in the deployment script.</p>
  <p>Add this to your <code>/etc/sudoers.d/supervisorctl</code>:</p>
  <pre><code>websiteuser ALL=NOPASSWD: /usr/bin/supervisorctl *
  </code></pre>
  <p>Replace <code>websiteuser</code> with the website's (Ubuntu) user in Forge.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-08T14:14:44.263Z
Answers.Text: Thanks Moritz for a good and thoughtful answer.
  
  Unfortunately in my case, deployments doesn't seem to be connected with the issue.
  
  It has happened to pages that have been without deployments for weeks, and suddenly the daemon fails.
  
  The tips are still very useful though. Maybe regular restarting is necessary and will help ‍♂️
Answers.UserDisplayName: Eddie
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-08T14:28:45.160Z
Answers.Text: @Eddie Yeah some people on Discord said the same. In this case I would try the nightly restart. Of course, it's only a workaround, but this should work well enough for sites without huge amounts of traffic, so those won't pile up queue jobs while nobody is looking.
Answers.UserDisplayName: MoritzLost
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-11T23:53:03.953Z
Answers.Text: Although this doesn't technically answer OPs question, I awarded the bounty to @MoritzLost as their answer at least offers some well-explained, concrete actions to mitigate the problem in the interim.
Answers.UserDisplayName: Mats Mikkel Rummelhoff
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-13T09:23:06.540Z
Answers.Text: @MatsMikkelRummelhoff Yeah, unfortunate that for now there's only a workaround – hopefully someone can figure out why this is happening in the first place! Might also be something the Craft team needs to address, not sure if it's an infrastructure problem or a bug in the queue handler.
Answers.UserDisplayName: MoritzLost
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-06T10:05:31.220Z
Answers.LastActivityDate: 2023-11-06T10:05:31.220Z
Answers.UserDisplayName: MoritzLost
Answers.Body: <p>What about the <a href="https://www.kolide.com/features/checks/ubuntu-unattended-upgrades" rel="nofollow noreferrer">unattended-upgrades</a>?</p>
  <p>We had a similar problem with daemons (and still do) but not with the database.
  In our case, whenever an update occurs, with some chance daemons die with the same error(<code>FATAL Exited too quickly</code>). The supervisor is restarted, but somehow the daemon doesn’t seem to work.</p>
  <p>To test this theory, look at when specific service (supervisor, mysql etc) crashed and compare it with the unattended-upgrade.log(everything should be in the /var/log/)</p>
  <p>E.g. <a href="https://www.reddit.com/r/docker/comments/k4kk97/docker_daemon_crash_with_unattendedupgrades/" rel="nofollow noreferrer">https://www.reddit.com/r/docker/comments/k4kk97/docker_daemon_crash_with_unattendedupgrades/</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-05T22:17:10.817Z
Answers.Text: We're experiencing the same issue as OP, and Forge's auto updates (which I believe use `unattended-upgrades`) has definitely been floated as a possible culprit. I'm going to look closer into that; though if this is somehow related I still find it strange that the problem appears to exclusively occur on boxes w/ MySQL 8 (we haven't seen it occur on any of our MariaDB boxes to date, of which we have many).
Answers.UserDisplayName: Mats Mikkel Rummelhoff
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-05T16:36:42.243Z
Answers.LastActivityDate: 2023-11-05T16:36:42.243Z
Answers.UserDisplayName: RomanAvr
Body: <p>Somehow the <code>queue/listen</code> occasionally loses access to the database:</p>
  <pre><code>FATAL     Exited too quickly
  </code></pre>
  <p>.</p>
  <pre><code>Database Connection Error: Craft CMS can’t connect to the database.
  </code></pre>
  <p>As the connection is lost, it seems like the daemon is not able to recover, and needs to be manually restarted.</p>
  <p><strong>Additional info:</strong></p>
  <ul>
  <li>Database: MySQL 8.0.35</li>
  <li>There was no deployment being done at the time of failure.</li>
  <li>I don’t currently have any automatic restarting of the daemon after deployments.</li>
  <li>There are no other indications of the database being down at the time the daemon dropped out.</li>
  <li>It has happened across multiple sites, and across hosts.</li>
  <li>One common factor is that Laravel Forge is used for provisioning and setting up the Daemon.</li>
  <li>It has happened to custom VPS’s, and Digital Ocean Droplets.</li>
  <li>All affected sites as far as I know are Craft 4.5.5 or later.</li>
  <li>This issue is new to me the last 3-5 months, even though I've made Craft sites with this same setup for a couple of years.</li>
  <li>The issue has also been discussed in the Discord channel &quot;Devops&quot; late August 2023. <a href="https://discord.com/channels/456442477667418113/456448961679589386/1144029528394580099" rel="noreferrer">https://discord.com/channels/456442477667418113/456448961679589386/1144029528394580099</a></li>
  <li>It seems to have a tendency to happen at night time. Which is also roundabout the same time doing database backups. It doesn't correlate greatly, so it seems farfetched to be linked. Often linked by +- 2 hours.</li>
  </ul>
  <p>Most of the time this happens, its quite critical to get up and running again, and I don't know how to really get to the bottom of the problem.</p>
  <p>Daemon Supervisor Configuration</p>
  <pre><code>[program:daemon-758141]
  command=/usr/bin/nice -n 10 /usr/bin/php /home/forge/www.site.com/craft queue/listen --verbose
  
  process_name=%(program_name)s_%(process_num)02d
  autostart=true
  autorestart=true
  user=forge
  numprocs=2
  startsecs=1
  redirect_stderr=true
  stdout_logfile=/home/forge/.forge/daemon-758141.log
  stopwaitsecs=10
  stopsignal=SIGTERM
  stopasgroup=true
  killasgroup=true
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-03T11:24:43.887Z
Comments.Text: This seems like a rather complicated bug based on the lengthy Discord discussion, but I'd like to just leave these links here as they are in my documentation for an issue that I fixed which might _possibly_ be related.
  
  https://discord.com/channels/456442477667418113/456442477667418115/940287266771832892
  https://www.freedesktop.org/software/systemd/man/systemd.unit.html#StartLimitIntervalSec=interval
  https://unix.stackexchange.com/questions/289629/systemd-restart-always-is-not-honored
Comments.UserDisplayName: James Smith
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-03T11:25:10.897Z
Comments.Text: Basically, with the default settings, even a 500ms DB outage would have caused the queue daemon to fail and not restart.
Comments.UserDisplayName: James Smith
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-03T11:39:46.783Z
Comments.Text: Thanks for good links and points. Unfortunately they seem to be only systemd related, but still gives some guidance.
  In my case Supervisor is used, and not systemd, and I dont seem to find a similar restart-timing configuration in Supervisor ‍♂️
Comments.UserDisplayName: Eddie
Tags: <mysql><hosting><services><queue>
Title: Craft loses contact with database, and stops daemon
ThreadText: # Craft loses contact with database, and stops daemon
  
  (Asked by: username_0 on 2023-11-03)
  
  Somehow the `queue/listen` occasionally loses access to the database:
  
  ```
  FATAL     Exited too quickly
  
  ```
  
  .
  
  ```
  Database Connection Error: Craft CMS can’t connect to the database.
  
  ```
  
  As the connection is lost, it seems like the daemon is not able to recover, and needs to be manually restarted.
  
  **Additional info:**
  
  * Database: MySQL 8.0.35
  * There was no deployment being done at the time of failure.
  * I don’t currently have any automatic restarting of the daemon after deployments.
  * There are no other indications of the database being down at the time the daemon dropped out.
  * It has happened across multiple sites, and across hosts.
  * One common factor is that Laravel Forge is used for provisioning and setting up the Daemon.
  * It has happened to custom VPS’s, and Digital Ocean Droplets.
  * All affected sites as far as I know are Craft 4.5.5 or later.
  * This issue is new to me the last 3-5 months, even though I've made Craft sites with this same setup for a couple of years.
  * The issue has also been discussed in the Discord channel "Devops" late August 2023. <https://discord.com/channels/456442477667418113/456448961679589386/1144029528394580099>
  * It seems to have a tendency to happen at night time. Which is also roundabout the same time doing database backups. It doesn't correlate greatly, so it seems farfetched to be linked. Often linked by +- 2 hours.
  
  Most of the time this happens, its quite critical to get up and running again, and I don't know how to really get to the bottom of the problem.
  
  Daemon Supervisor Configuration
  
  ```
  [program:daemon-758141]
  command=/usr/bin/nice -n 10 /usr/bin/php /home/forge/www.site.com/craft queue/listen --verbose
  
  process_name=%(program_name)s_%(process_num)02d
  autostart=true
  autorestart=true
  user=forge
  numprocs=2
  startsecs=1
  redirect_stderr=true
  stdout_logfile=/home/forge/.forge/daemon-758141.log
  stopwaitsecs=10
  stopsignal=SIGTERM
  stopasgroup=true
  killasgroup=true
  
  ```
  
  **Tags:** mysql, hosting, services, queue
  
  > **Comments:**
  > **username_1**: This seems like a rather complicated bug based on the lengthy Discord discussion, but I'd like to just leave these links here as they are in my documentation for an issue that I fixed which might _possibly_ be related.  https://discord.com/channels/456442477667418113/456442477667418115/940287266771832892 https://www.freedesktop.org/software/systemd/man/systemd.unit.html#StartLimitIntervalSec=interval https://unix.stackexchange.com/questions/289629/systemd-restart-always-is-not-honored  
  > **username_1**: Basically, with the default settings, even a 500ms DB outage would have caused the queue daemon to fail and not restart.  
  > **username_0**: Thanks for good links and points. Unfortunately they seem to be only systemd related, but still gives some guidance. In my case Supervisor is used, and not systemd, and I dont seem to find a similar restart-timing configuration in Supervisor ‍♂️  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-11-06. Score: 3
  
  We're also seeing this issue on Laravel Forge. For us it mostly happens after a Craft update is deployed, so it might have something to do with the queue runner still having the previous Craft version loaded.
  
  While I'm not sure what causes it, as a workaround you can ensure that the queue listener restarts periodically and/or after deployments.
  
  There are some problems with this approach. For sites with a lot of traffic and queue jobs, restarting the queue runner may result in the queue getting backed up. Also, when a queue job is running while the queue runner is being forcefully restarted, it can cause the job to fail. To address this, we also have a nightly cron job that runs `queue/retry` to automatically retry failed queue jobs. But for small and medium sites that don't have tons of queue jobs coming in 24/7, this approach works quite well.
  
  ## Restarting the queue runner periodically
  
  Create a cron job in Forge, in the server settings under **Scheduler**:
  
  * **Command:** `sudo supervisorctl restart all`
  * **User:** `forge`
  * **Frequency:** Nightly (or choose whatever you like)
  
  On newly provisioned servers, this should work without any further adjustments, because Forge now allows the `forge` user to execute `supervisorctl` commands without a password. On older servers, make sure the following snippet is in your `/etc/sudoers.d/supervisorctl`:
  
  ```
  forge ALL=NOPASSWD: /usr/bin/supervisorctl reload
  forge ALL=NOPASSWD: /usr/bin/supervisorctl reread
  forge ALL=NOPASSWD: /usr/bin/supervisorctl restart *
  forge ALL=NOPASSWD: /usr/bin/supervisorctl start *
  forge ALL=NOPASSWD: /usr/bin/supervisorctl status *
  forge ALL=NOPASSWD: /usr/bin/supervisorctl status
  forge ALL=NOPASSWD: /usr/bin/supervisorctl stop *
  forge ALL=NOPASSWD: /usr/bin/supervisorctl update *
  forge ALL=NOPASSWD: /usr/bin/supervisorctl update
  
  ```
  
  ## Restarting the queue runner after deployments
  
  To restart the queue runner after every deployment, add this to the end of your deployment script:
  
  ```
  sudo supervisorctl restart all
  
  ```
  
  This will restart **all** daemons. Alternatively, you can also find the ID of your queue runner (use `sudo supervisorctl status` to list all active daemons) and only restart that in your deployment (replace `<daemon-id>` with the actual ID):
  
  ```
  sudo supervisorctl restart <daemon-id>
  
  ```
  
  On newer servers, if you're not using website isolation, this should work by default. If you *are* using website isolation, you need to allow the website user to run `supervisorctl` without a password, since you can't (or rather, shouldn't) provide the password in the deployment script.
  
  Add this to your `/etc/sudoers.d/supervisorctl`:
  
  ```
  websiteuser ALL=NOPASSWD: /usr/bin/supervisorctl *
  
  ```
  
  Replace `websiteuser` with the website's (Ubuntu) user in Forge.
  
  > **Comments:**
  > **username_0**: Thanks Moritz for a good and thoughtful answer.  Unfortunately in my case, deployments doesn't seem to be connected with the issue.  It has happened to pages that have been without deployments for weeks, and suddenly the daemon fails.  The tips are still very useful though. Maybe regular restarting is necessary and will help ‍♂️  
  > **username_2**: @username_0 Yeah some people on Discord said the same. In this case I would try the nightly restart. Of course, it's only a workaround, but this should work well enough for sites without huge amounts of traffic, so those won't pile up queue jobs while nobody is looking.  
  > **username_3**: Although this doesn't technically answer OPs question, I awarded the bounty to @username_2 as their answer at least offers some well-explained, concrete actions to mitigate the problem in the interim.  
  > **username_2**: @username_3 Yeah, unfortunate that for now there's only a workaround – hopefully someone can figure out why this is happening in the first place! Might also be something the Craft team needs to address, not sure if it's an infrastructure problem or a bug in the queue handler.  
  
  ---
  
  ### Answer by username_4 on 2023-11-05. Score: 2
  
  What about the [unattended-upgrades](https://www.kolide.com/features/checks/ubuntu-unattended-upgrades)?
  
  We had a similar problem with daemons (and still do) but not with the database.
  In our case, whenever an update occurs, with some chance daemons die with the same error(`FATAL Exited too quickly`). The supervisor is restarted, but somehow the daemon doesn’t seem to work.
  
  To test this theory, look at when specific service (supervisor, mysql etc) crashed and compare it with the unattended-upgrade.log(everything should be in the /var/log/)
  
  E.g. <https://www.reddit.com/r/docker/comments/k4kk97/docker_daemon_crash_with_unattendedupgrades/>
  
  > **Comments:**
  > **username_3**: We're experiencing the same issue as OP, and Forge's auto updates (which I believe use `unattended-upgrades`) has definitely been floated as a possible culprit. I'm going to look closer into that; though if this is somehow related I still find it strange that the problem appears to exclusively occur on boxes w/ MySQL 8 (we haven't seen it occur on any of our MariaDB boxes to date, of which we have many).  
  
  ---
  
  