<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Enhancement%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical enhancement to an existing feature.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22Issue+Type%3A+Enhancement%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed enhancements.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the if the same enhancement was already implemented in the<br>
master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues in this issue<br>
(If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h1 dir="auto">Brief Summary</h1>

<p dir="auto">When there is a connection error with Redis while executing a command, in most cases, the redis client will <a href="https://github.com/andymccurdy/redis-py/blob/272d3139e1c82c2d89551f87d12df7c18d938ea2/redis/client.py#L885-L886">discard the connection</a>, causing the next command sent to Redis to open a new connection. This allows applications to recover from connection errors by simply retrying, a property that is used in Celery, for example when setting keys in the Redis result backend: </p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/celery/celery/blob/d0563058f8f47f347ac1b56c44f833f569764482/celery/backends/redis.py#L324-L325">celery/celery/backends/redis.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
        Lines 324 to 325
      in
      <a data-pjax="true" class="commit-tease-sha" href="/celery/celery/commit/d0563058f8f47f347ac1b56c44f833f569764482">d056305</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L324" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="324"></td>
          <td id="LC324" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">def</span> <span class="pl-en">set</span>(<span class="pl-s1">self</span>, <span class="pl-s1">key</span>, <span class="pl-s1">value</span>, <span class="pl-c1">**</span><span class="pl-s1">retry_policy</span>): </td>
        </tr>

        <tr class="border-0">
          <td id="L325" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="325"></td>
          <td id="LC325" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-k">return</span> <span class="pl-s1">self</span>.<span class="pl-en">ensure</span>(<span class="pl-s1">self</span>.<span class="pl-s1">_set</span>, (<span class="pl-s1">key</span>, <span class="pl-s1">value</span>), <span class="pl-c1">**</span><span class="pl-s1">retry_policy</span>) </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<p dir="auto">This is not the case however when the <a href="https://github.com/andymccurdy/redis-py/blob/272d3139e1c82c2d89551f87d12df7c18d938ea2/redis/client.py#L3397-L3414">connection to Redis is in a pubsub state</a>. The reason for that is that some state associated with the connection (namely the list of keys subscibed to). The Redis client doesn't keep track of this state, so it can't possibly restore it when creating a new connection and leaves the connection handling to the application code.</p>
<p dir="auto">The Celery Redis result consumer uses pubsub in order to be notified when results are available, but doesn't handle connection errors at all, causing a result consumer to end up in a state where it can't connect to the result backend any more after a single connection error, as any further attempt will reuse the same faulty connection.</p>
<p dir="auto">The solution would be to add error handling logic to the result consumer, so it will recreate the connection on connection errors and initialize it to the proper state.</p>
<h1 dir="auto">Design</h1>
<h2 dir="auto">Architectural Considerations</h2>

<p dir="auto">None</p>
<h2 dir="auto">Proposed Behavior</h2>

<p dir="auto">Add error handling in all places the Redis result consumer sends a Redis command in a pubsub context:</p>
<ul dir="auto">
<li><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/celery/celery/blob/d0563058f8f47f347ac1b56c44f833f569764482/celery/backends/redis.py#L127">celery/celery/backends/redis.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 127
      in
      <a data-pjax="true" class="commit-tease-sha" href="/celery/celery/commit/d0563058f8f47f347ac1b56c44f833f569764482">d056305</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L127" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="127"></td>
          <td id="LC127" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">message</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-s1">_pubsub</span>.<span class="pl-en">get_message</span>(<span class="pl-s1">timeout</span><span class="pl-c1">=</span><span class="pl-s1">timeout</span>) </td>
        </tr>
    </tbody></table>
  </div>
</div>
</li>
<li><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/celery/celery/blob/d0563058f8f47f347ac1b56c44f833f569764482/celery/backends/redis.py#L142">celery/celery/backends/redis.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 142
      in
      <a data-pjax="true" class="commit-tease-sha" href="/celery/celery/commit/d0563058f8f47f347ac1b56c44f833f569764482">d056305</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L142" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="142"></td>
          <td id="LC142" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">self</span>.<span class="pl-s1">_pubsub</span>.<span class="pl-en">subscribe</span>(<span class="pl-s1">key</span>) </td>
        </tr>
    </tbody></table>
  </div>
</div>
</li>
<li><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/celery/celery/blob/d0563058f8f47f347ac1b56c44f833f569764482/celery/backends/redis.py#L148">celery/celery/backends/redis.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 148
      in
      <a data-pjax="true" class="commit-tease-sha" href="/celery/celery/commit/d0563058f8f47f347ac1b56c44f833f569764482">d056305</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L148" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="148"></td>
          <td id="LC148" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">self</span>.<span class="pl-s1">_pubsub</span>.<span class="pl-en">unsubscribe</span>(<span class="pl-s1">key</span>) </td>
        </tr>
    </tbody></table>
  </div>
</div>
</li>
</ul>
<p dir="auto">We should catch all Redis connection errors, and call a new method that will reinitialize a pubsub connection in the proper state (discard the current connection from the pool, start the pubsub context, subscribe to all keys in <code class="notranslate">ResultConsumer.subscribed_to</code>) using the retry policy. If in <code class="notranslate">drain_events</code>, we should try to get new messages again.</p>
<p dir="auto">This will take care of most issues with connection errors. I see two remaining issues:<br>
1.Some message might have been lost (sent between losing the connection and reconnecting). We could read all keys subscribed to right after reconnecting and before starting the pubsub context and call <code class="notranslate">on_state_change</code> for each existing key, but this might cause some messages to be delivered twice and I don't know how Celery will react to that.<br>
2. If the connection can't be re-established despite the retries and reaches max-retries, the result consumer will end up with a faulty connection that can't be recovered from. This should be communicated somehow to the user (documentation, logging an explicit error message, custom exception).</p>
<h2 dir="auto">Proposed UI/UX</h2>

<p dir="auto">None</p>
<h2 dir="auto">Diagrams</h2>

<p dir="auto">N/A</p>
<h2 dir="auto">Alternatives</h2>

<p dir="auto">None</p>
<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> This has already been asked to the <a href="https://github.com/celery/celery/discussions">discussions forum</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="https://docs.celeryq.dev/en/master/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li>[] I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>: 5.3.0a1 (dawn-chorus)</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"><code class="notranslate"></code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: Python 3.10.4</li>
<li><strong>Minimal Celery Version</strong>: 5.3.0a1</li>
<li><strong>Minimal Kombu Version</strong>: 5.3.0a1</li>
<li><strong>Minimal Broker Version</strong>: RabbitMQ 3.9.1</li>
<li><strong>Minimal Result Backend Version</strong>: Memory Cache</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: Ubuntu 22.04</li>
<li><strong>Minimal Broker Client Version</strong>: 3.9.1</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<p dir="auto">software -&gt; celery:5.3.0a1 (dawn-chorus) kombu:5.3.0a1 py:3.10.4<br>
billiard:3.6.4.0 py-amqp:5.1.1<br>
platform -&gt; system:Linux arch:64bit, ELF<br>
kernel version:5.15.0-25-generic imp:CPython<br>
loader   -&gt; celery.loaders.default.Loader<br>
settings -&gt; transport:amqp results:cache</p>
<p dir="auto">broker_pool_limit: 32768<br>
broker_url: 'amqp://guest:********<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/localhost/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/localhost">@localhost</a>:1928//'<br>
cache_backend: 'memory'<br>
enable_utc: True<br>
imports:<br>
('services',)<br>
result_backend: 'cache'<br>
result_expires: 10800<br>
task_acks_late: True<br>
task_annotations: {<br>
'services.a_service': {'rate_limit': '7000/m'},<br>
'services.b_service': {'rate_limit': '7000/m'},<br>
'services.c_service': {'rate_limit': '7000/m'},<br>
'services.d_service': {'rate_limit': '7000/m'},<br>
'services.e_service': {'rate_limit': '7000/m'},<br>
'services.f_service': {'rate_limit': '1/m'},<br>
'services.g_service': {'rate_limit': '1/m'},<br>
'services.h_service': {'rate_limit': '1/m'},<br>
'services.i_service': {'rate_limit': '7000/m'},<br>
'services.j_service': {'rate_limit': '7000/m'},<br>
'services.xx_service': {'rate_limit': '7000/m'},<br>
'services.zz_service': {'rate_limit': '7000/m'},<br>
'services.vc_service': {'rate_limit': '7000/m'}}<br>
task_ignore_result: True<br>
deprecated_settings: None</p>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="aiocron==1.8
aiofiles==0.8.0
aiohttp==3.8.1
aiohttp-socks==0.7.1
aioredis==2.0.1
aiosignal==1.2.0
amqp==5.1.1
anyio==3.5.0
asgiref==3.5.0
async-timeout==4.0.2
asyncpg==0.25.0
attrs==21.4.0
beautifulsoup4==4.11.1
billiard==3.6.4.0
blessed==1.19.0
bs4==0.0.1
celery @ https://github.com/celery/celery/archive/refs/heads/master.zip
certifi==2021.10.8
chardet==4.0.0
charset-normalizer==2.0.12
click==8.1.2
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.2.0
coverage==6.3.2
crc16==0.1.1
croniter==1.3.4
cryptg==0.3.1
DateTime==4.5
dbus-python==1.2.18
Deprecated==1.2.13
distro-info===1.1build1
Django==4.0.4
dnspython==2.2.1
eventlet==0.33.1
Faker==13.3.5
fastapi==0.75.2
flower==1.0.0
frozenlist==1.3.0
future==0.18.2
gevent==21.12.0
greenlet==1.1.2
gyp==0.1
h11==0.13.0
humanize==0.0.0
idna==3.3
image==1.5.33
iniconfig==1.1.1
jdatetime==4.1.0
kombu==5.3.0a1
lxml==4.8.0
motor @ https://github.com/mongodb/motor/archive/refs/heads/master.zip
multidict==6.0.2
netifaces==0.11.0
nose==1.3.7
packaging==21.3
persiantools==3.0.0
pg-activity==2.2.1
phonenumbers==8.12.47
Pillow==9.1.0
pluggy==1.0.0
pprintpp==0.4.0
prometheus-client==0.14.1
prompt-toolkit==3.0.29
psutil==5.9.1
psycopg2==2.9.3
py==1.11.0
pyaes==1.6.1
pyArango==2.0.1
pyasn1==0.4.8
pycountry==22.3.5
pycountry-convert==0.7.2
pydantic==1.9.0
Pygments==2.11.2
PyGObject==3.42.0
pymongo==4.1.1
pyparsing==3.0.8
Pyrogram==2.0.19
PySocks==1.7.1
pytest==7.1.1
pytest-cov==3.0.0
pytest-mock==3.7.0
python-apt==2.3.0+ubuntu2
python-dateutil==2.8.2
python-debian===0.1.43ubuntu1
python-dotenv==0.20.0
python-multipart==0.0.5
python-socks==2.0.3
pytz==2022.1
pytz-deprecation-shim==0.1.0.post0
PyYAML==5.4.1
qrcode==7.3.1
Random-Word==1.0.7
redis==4.2.2
repoze.lru==0.7
requests==2.27.1
rsa==4.8
six==1.16.0
sniffio==1.2.0
socks==0
soupsieve==2.3.2.post1
speedtest-cli==2.1.3
SQLAlchemy==1.4.35
sqlalchemy2-stubs==0.0.2a22
sqlmodel==0.0.6
sqlparse==0.4.2
starlette==0.17.1
TgCrypto==1.2.3
tomli==2.0.1
tornado==6.1
typing_extensions==4.2.0
tzdata==2022.1
tzlocal==4.2
ubuntu-advantage-tools==27.9
ufw==0.36.1
urllib3==1.26.9
uvicorn==0.17.6
vine==5.0.0
wcwidth==0.2.5
wrapt==1.14.0
yarl==1.7.2
zope.event==4.5.0
zope.interface==5.4.0"><pre class="notranslate"><code class="notranslate">aiocron==1.8
aiofiles==0.8.0
aiohttp==3.8.1
aiohttp-socks==0.7.1
aioredis==2.0.1
aiosignal==1.2.0
amqp==5.1.1
anyio==3.5.0
asgiref==3.5.0
async-timeout==4.0.2
asyncpg==0.25.0
attrs==21.4.0
beautifulsoup4==4.11.1
billiard==3.6.4.0
blessed==1.19.0
bs4==0.0.1
celery @ https://github.com/celery/celery/archive/refs/heads/master.zip
certifi==2021.10.8
chardet==4.0.0
charset-normalizer==2.0.12
click==8.1.2
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.2.0
coverage==6.3.2
crc16==0.1.1
croniter==1.3.4
cryptg==0.3.1
DateTime==4.5
dbus-python==1.2.18
Deprecated==1.2.13
distro-info===1.1build1
Django==4.0.4
dnspython==2.2.1
eventlet==0.33.1
Faker==13.3.5
fastapi==0.75.2
flower==1.0.0
frozenlist==1.3.0
future==0.18.2
gevent==21.12.0
greenlet==1.1.2
gyp==0.1
h11==0.13.0
humanize==0.0.0
idna==3.3
image==1.5.33
iniconfig==1.1.1
jdatetime==4.1.0
kombu==5.3.0a1
lxml==4.8.0
motor @ https://github.com/mongodb/motor/archive/refs/heads/master.zip
multidict==6.0.2
netifaces==0.11.0
nose==1.3.7
packaging==21.3
persiantools==3.0.0
pg-activity==2.2.1
phonenumbers==8.12.47
Pillow==9.1.0
pluggy==1.0.0
pprintpp==0.4.0
prometheus-client==0.14.1
prompt-toolkit==3.0.29
psutil==5.9.1
psycopg2==2.9.3
py==1.11.0
pyaes==1.6.1
pyArango==2.0.1
pyasn1==0.4.8
pycountry==22.3.5
pycountry-convert==0.7.2
pydantic==1.9.0
Pygments==2.11.2
PyGObject==3.42.0
pymongo==4.1.1
pyparsing==3.0.8
Pyrogram==2.0.19
PySocks==1.7.1
pytest==7.1.1
pytest-cov==3.0.0
pytest-mock==3.7.0
python-apt==2.3.0+ubuntu2
python-dateutil==2.8.2
python-debian===0.1.43ubuntu1
python-dotenv==0.20.0
python-multipart==0.0.5
python-socks==2.0.3
pytz==2022.1
pytz-deprecation-shim==0.1.0.post0
PyYAML==5.4.1
qrcode==7.3.1
Random-Word==1.0.7
redis==4.2.2
repoze.lru==0.7
requests==2.27.1
rsa==4.8
six==1.16.0
sniffio==1.2.0
socks==0
soupsieve==2.3.2.post1
speedtest-cli==2.1.3
SQLAlchemy==1.4.35
sqlalchemy2-stubs==0.0.2a22
sqlmodel==0.0.6
sqlparse==0.4.2
starlette==0.17.1
TgCrypto==1.2.3
tomli==2.0.1
tornado==6.1
typing_extensions==4.2.0
tzdata==2022.1
tzlocal==4.2
ubuntu-advantage-tools==27.9
ufw==0.36.1
urllib3==1.26.9
uvicorn==0.17.6
vine==5.0.0
wcwidth==0.2.5
wrapt==1.14.0
yarl==1.7.2
zope.event==4.5.0
zope.interface==5.4.0
</code></pre></div>
<p dir="auto"></p>

<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
N/A
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>

<details>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="
[2022-08-08 03:58:25,725: DEBUG/ForkPoolWorker-522] Closing current connection...
[2022-08-08 03:58:25,725: DEBUG/ForkPoolWorker-522] Cancelling 0 pending message(s)...
[2022-08-08 03:58:25,725: INFO/ForkPoolWorker-522] Disconnection from 149.154.175.58:443/TcpFull complete!
[2022-08-08 03:58:25,725: INFO/ForkPoolWorker-522] Task services.a_service[64f77116-f3f6-4bc1-ace2-5b994258cfd1] succeeded in 2.5671544598881155s: None
[2022-08-08 03:58:25,833: DEBUG/ForkPoolWorker-513] Handling RPC result for message 7129318888505874252
[2022-08-08 03:58:25,833: DEBUG/ForkPoolWorker-513] Receiving items from the network...
[2022-08-08 03:58:25,833: INFO/ForkPoolWorker-513] Disconnecting from 149.154.175.54:443/TcpFull...
[2022-08-08 03:58:25,833: DEBUG/ForkPoolWorker-513] Closing current connection...
[2022-08-08 03:58:25,834: DEBUG/ForkPoolWorker-513] Cancelling 0 pending message(s)...
[2022-08-08 03:58:25,834: INFO/ForkPoolWorker-513] Disconnection from 149.154.175.54:443/TcpFull complete!
[2022-08-08 03:58:25,834: INFO/ForkPoolWorker-513] Task services.a_service[f83856d0-bdae-43f0-8f5f-0272c2f5fc0f] succeeded in 2.6757159510161728s: None
[2022-08-08 03:58:34,700: DEBUG/MainProcess] heartbeat_tick : for connection 158072e042fd43e2982921507c4090df
[2022-08-08 03:58:34,700: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 529/751, now - 605/775, monotonic - 1686196.873902797, last_heartbeat_sent - 1686196.873901675, heartbeat int. - 60 for connection 158072e042fd43e2982921507c4090df
[2022-08-08 03:58:54,706: DEBUG/MainProcess] heartbeat_tick : for connection 158072e042fd43e2982921507c4090df
[2022-08-08 03:58:54,706: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 605/775, now - 605/805, monotonic - 1686216.879871576, last_heartbeat_sent - 1686196.873901675, heartbeat int. - 60 for connection 158072e042fd43e2982921507c4090df
[2022-08-08 03:59:14,711: DEBUG/MainProcess] heartbeat_tick : for connection 158072e042fd43e2982921507c4090df
[2022-08-08 03:59:14,711: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 605/805, now - 605/835, monotonic - 1686236.885361101, last_heartbeat_sent - 1686196.873901675, heartbeat int. - 60 for connection 158072e042fd43e2982921507c4090df
[2022-08-08 03:59:34,716: DEBUG/MainProcess] heartbeat_tick: sending heartbeat for connection 158072e042fd43e2982921507c4090df
[2022-08-08 03:59:54,720: DEBUG/MainProcess] heartbeat_tick : for connection 158072e042fd43e2982921507c4090df
[2022-08-08 03:59:54,720: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 605/865, now - 606/895, monotonic - 1686276.894446629, last_heartbeat_sent - 1686276.894445928, heartbeat int. - 60 for connection 158072e042fd43e2982921507c4090df
[2022-08-08 04:00:14,726: DEBUG/MainProcess] heartbeat_tick : for connection 158072e042fd43e2982921507c4090df
[2022-08-08 04:00:14,726: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 606/895, now - 606/925, monotonic - 1686296.899726603, last_heartbeat_sent - 1686276.894445928, heartbeat int. - 60 for connection 158072e042fd43e2982921507c4090df
"><pre class="notranslate"><code class="notranslate">
[2022-08-08 03:58:25,725: DEBUG/ForkPoolWorker-522] Closing current connection...
[2022-08-08 03:58:25,725: DEBUG/ForkPoolWorker-522] Cancelling 0 pending message(s)...
[2022-08-08 03:58:25,725: INFO/ForkPoolWorker-522] Disconnection from 149.154.175.58:443/TcpFull complete!
[2022-08-08 03:58:25,725: INFO/ForkPoolWorker-522] Task services.a_service[64f77116-f3f6-4bc1-ace2-5b994258cfd1] succeeded in 2.5671544598881155s: None
[2022-08-08 03:58:25,833: DEBUG/ForkPoolWorker-513] Handling RPC result for message 7129318888505874252
[2022-08-08 03:58:25,833: DEBUG/ForkPoolWorker-513] Receiving items from the network...
[2022-08-08 03:58:25,833: INFO/ForkPoolWorker-513] Disconnecting from 149.154.175.54:443/TcpFull...
[2022-08-08 03:58:25,833: DEBUG/ForkPoolWorker-513] Closing current connection...
[2022-08-08 03:58:25,834: DEBUG/ForkPoolWorker-513] Cancelling 0 pending message(s)...
[2022-08-08 03:58:25,834: INFO/ForkPoolWorker-513] Disconnection from 149.154.175.54:443/TcpFull complete!
[2022-08-08 03:58:25,834: INFO/ForkPoolWorker-513] Task services.a_service[f83856d0-bdae-43f0-8f5f-0272c2f5fc0f] succeeded in 2.6757159510161728s: None
[2022-08-08 03:58:34,700: DEBUG/MainProcess] heartbeat_tick : for connection 158072e042fd43e2982921507c4090df
[2022-08-08 03:58:34,700: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 529/751, now - 605/775, monotonic - 1686196.873902797, last_heartbeat_sent - 1686196.873901675, heartbeat int. - 60 for connection 158072e042fd43e2982921507c4090df
[2022-08-08 03:58:54,706: DEBUG/MainProcess] heartbeat_tick : for connection 158072e042fd43e2982921507c4090df
[2022-08-08 03:58:54,706: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 605/775, now - 605/805, monotonic - 1686216.879871576, last_heartbeat_sent - 1686196.873901675, heartbeat int. - 60 for connection 158072e042fd43e2982921507c4090df
[2022-08-08 03:59:14,711: DEBUG/MainProcess] heartbeat_tick : for connection 158072e042fd43e2982921507c4090df
[2022-08-08 03:59:14,711: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 605/805, now - 605/835, monotonic - 1686236.885361101, last_heartbeat_sent - 1686196.873901675, heartbeat int. - 60 for connection 158072e042fd43e2982921507c4090df
[2022-08-08 03:59:34,716: DEBUG/MainProcess] heartbeat_tick: sending heartbeat for connection 158072e042fd43e2982921507c4090df
[2022-08-08 03:59:54,720: DEBUG/MainProcess] heartbeat_tick : for connection 158072e042fd43e2982921507c4090df
[2022-08-08 03:59:54,720: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 605/865, now - 606/895, monotonic - 1686276.894446629, last_heartbeat_sent - 1686276.894445928, heartbeat int. - 60 for connection 158072e042fd43e2982921507c4090df
[2022-08-08 04:00:14,726: DEBUG/MainProcess] heartbeat_tick : for connection 158072e042fd43e2982921507c4090df
[2022-08-08 04:00:14,726: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 606/895, now - 606/925, monotonic - 1686296.899726603, last_heartbeat_sent - 1686276.894445928, heartbeat int. - 60 for connection 158072e042fd43e2982921507c4090df

</code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>

<h1 dir="auto">Actual Behavior</h1>

<p dir="auto">Hello<br>
i must to handle 5 million tasks daily<br>
i have 100 Queues and number of concurrency in worker is 1000 and i use multi quques like celery -A proj worker -Q Qa,Qb,Qc ... ( i dont know if its effective to say )<br>
sometimes my worker stuck<br>
and this is my logs<br>
worker stuck for 60 ~ 100 seconds then continue working<br>
sometimes it can handle it very fast and sometimes i faced with this and my worker stucking every 60 ~ 100 seconds<br>
did i panic ? is evenrything is ok<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/75903249/183329812-6766390c-7586-46b4-841e-05b1e2ed9396.png"><img width="1507" alt="Screen Shot 2022-08-08 at 7 30 10 AM" src="https://user-images.githubusercontent.com/75903249/183329812-6766390c-7586-46b4-841e-05b1e2ed9396.png" style="max-width: 100%;"></a></p>