<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Enhancement%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical enhancement to an existing feature.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22Issue+Type%3A+Enhancement%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed enhancements.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the if the same enhancement was already implemented in the<br>
master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues in this issue<br>
(If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="697564497" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/6348" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/6348/hovercard" href="https://github.com/celery/celery/issues/6348">#6348</a>
<ul dir="auto">
<li>The situation is similar, but this issue is based on Redis Sentinel and we use AWS ElastiCache. Error message is also a bit different.</li>
</ul>
</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h1 dir="auto">Brief Summary</h1>

<p dir="auto">When AWS ElastiCache for Redis has a replication structure and causes a failover, Celery doesn't failover and the main process tries graceful shutdown. It means,</p>
<ul dir="auto">
<li>main process exits due to Unrecoverable error when there is no remaining task</li>
<li>main process hangs until the remaining task finishes</li>
</ul>
<p dir="auto">The second one is critical for us because we have many long ongoing tasks. The first one is better than it because we automatically reboot the process, but we'd like to avoid it because Redis failover is a kind of expected behavior.</p>
<p dir="auto">If some change is acceptable, we will try to create a pull request by ourselves.</p>
<h1 dir="auto">Detail</h1>
<p dir="auto">We got the following error when ElastiCache failover happened.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[CRITICAL - 2022-11-04 10:45:27 - 21 worker:207] Unrecoverable error: ReadOnlyError(&quot;You can't write against a read only replica.&quot;)
Traceback (most recent call last):
  File &quot;/var/www/venv/lib/python3.7/site-packages/celery/worker/worker.py&quot;, line 203, in start
    self.blueprint.start(self)
  File &quot;/var/www/venv/lib/python3.7/site-packages/celery/bootsteps.py&quot;, line 116, in start
    step.start(parent)
  File &quot;/var/www/venv/lib/python3.7/site-packages/celery/bootsteps.py&quot;, line 365, in start
    return self.obj.start()
  File &quot;/var/www/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py&quot;, line 326, in start
    blueprint.start(self)
  File &quot;/var/www/venv/lib/python3.7/site-packages/celery/bootsteps.py&quot;, line 116, in start
    step.start(parent)
  File &quot;/var/www/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py&quot;, line 618, in start
    c.loop(*c.loop_args())
  File &quot;/var/www/venv/lib/python3.7/site-packages/celery/worker/loops.py&quot;, line 97, in asynloop
    next(loop)
  File &quot;/var/www/venv/lib/python3.7/site-packages/kombu/asynchronous/hub.py&quot;, line 362, in create_loop
    cb(*cbargs)
  File &quot;/var/www/venv/lib/python3.7/site-packages/kombu/transport/redis.py&quot;, line 1266, in on_readable
    self.cycle.on_readable(fileno)
  File &quot;/var/www/venv/lib/python3.7/site-packages/kombu/transport/redis.py&quot;, line 504, in on_readable
    chan.handlers[type]()
  File &quot;/var/www/venv/lib/python3.7/site-packages/kombu/transport/redis.py&quot;, line 898, in _brpop_read
    **options)
  File &quot;/var/www/venv/lib/python3.7/site-packages/redis/client.py&quot;, line 1189, in parse_response
    response = connection.read_response()
  File &quot;/var/www/venv/lib/python3.7/site-packages/redis/connection.py&quot;, line 817, in read_response
    raise response
redis.exceptions.ReadOnlyError: You can't write against a read only replica."><pre class="notranslate"><code class="notranslate">[CRITICAL - 2022-11-04 10:45:27 - 21 worker:207] Unrecoverable error: ReadOnlyError("You can't write against a read only replica.")
Traceback (most recent call last):
  File "/var/www/venv/lib/python3.7/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/var/www/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/var/www/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 365, in start
    return self.obj.start()
  File "/var/www/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 326, in start
    blueprint.start(self)
  File "/var/www/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/var/www/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 618, in start
    c.loop(*c.loop_args())
  File "/var/www/venv/lib/python3.7/site-packages/celery/worker/loops.py", line 97, in asynloop
    next(loop)
  File "/var/www/venv/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 362, in create_loop
    cb(*cbargs)
  File "/var/www/venv/lib/python3.7/site-packages/kombu/transport/redis.py", line 1266, in on_readable
    self.cycle.on_readable(fileno)
  File "/var/www/venv/lib/python3.7/site-packages/kombu/transport/redis.py", line 504, in on_readable
    chan.handlers[type]()
  File "/var/www/venv/lib/python3.7/site-packages/kombu/transport/redis.py", line 898, in _brpop_read
    **options)
  File "/var/www/venv/lib/python3.7/site-packages/redis/client.py", line 1189, in parse_response
    response = connection.read_response()
  File "/var/www/venv/lib/python3.7/site-packages/redis/connection.py", line 817, in read_response
    raise response
redis.exceptions.ReadOnlyError: You can't write against a read only replica.
</code></pre></div>
<p dir="auto">As far as we checked, an error during connection becomes <code class="notranslate">ConnectionError</code> and regarded as "Recoverable", but <code class="notranslate">ReadOnlyError</code> (and its parent <code class="notranslate">ResponseError</code>) is regarded as "Unrecoverable".</p>
<h1 dir="auto">Design</h1>
<h2 dir="auto">Architectural Considerations</h2>

<p dir="auto">Maybe this part in kombu is related<br>
<a href="https://github.com/celery/kombu/blob/v5.2.3/kombu/transport/redis.py#L128-L141">https://github.com/celery/kombu/blob/v5.2.3/kombu/transport/redis.py#L128-L141</a></p>
<h2 dir="auto">Proposed Behavior</h2>

<p dir="auto">When an error happens due to switching master to replica, retry connection and connect to the new master.</p>
<h2 dir="auto">Proposed UI/UX</h2>

<ul dir="auto">
<li>Adding some option to regard ResponseError as "Recoverable".  Configuration key like <code class="notranslate">redis_retry_on_response_error</code> or <code class="notranslate">redis_retry_on_failover</code>?</li>
</ul>
<p dir="auto">But we're not sure if regarding ResponseError as "Recoverable" has a critical impact on existing behavior. We'd really appreciate advice from the community.</p>
<h2 dir="auto">Diagrams</h2>

<p dir="auto">N/A</p>
<h2 dir="auto">Alternatives</h2>

<p dir="auto">None</p>
<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?utf8=%E2%9C%93&amp;q=is%3Aissue+label%3A%22Category%3A+Documentation%22+">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues in this issue<br>
(If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h1 dir="auto">Description</h1>

<p dir="auto">The migration command stated in the documentation doesn't work. The following works instead:<br>
<code class="notranslate">python manage.py migrate django_celery_results</code></p>
<h1 dir="auto">Suggestions</h1>