<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> This has already been asked to the <a href="https://groups.google.com/forum/#!forum/celery-users" rel="nofollow">discussion group</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>: 5.1.2</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="celery -A myapp report

software -&gt; celery:5.1.2 (sun-harmonics) kombu:5.1.0 py:3.9.0
            billiard:3.6.4.0 py-amqp:5.0.6
platform -&gt; system:Linux arch:64bit, ELF
            kernel version:5.11.0-22-generic imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:amqp results:disabled

broker_url: 'amqp://guest:********@localhost:5672//'
deprecated_settings: None"><pre class="notranslate"><code class="notranslate">celery -A myapp report

software -&gt; celery:5.1.2 (sun-harmonics) kombu:5.1.0 py:3.9.0
            billiard:3.6.4.0 py-amqp:5.0.6
platform -&gt; system:Linux arch:64bit, ELF
            kernel version:5.11.0-22-generic imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:amqp results:disabled

broker_url: 'amqp://guest:********@localhost:5672//'
deprecated_settings: None
</code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Celery Version</strong>: 1.x</li>
<li><strong>Minimal Kombu Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Version</strong>: N/A or Unknown</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Client Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="aiohttp==3.7.4.post0
alabaster==0.7.12
amqp==5.0.6
appdirs==1.4.4
async-timeout==3.0.1
attrs==20.3.0
autoflake==1.4
autopep8==1.5.7
aws-xray-sdk==0.95
azure-common==1.1.5
azure-nspkg==3.0.2
azure-storage==0.36.0
azure-storage-common==1.1.0
azure-storage-nspkg==3.1.0
Babel==2.8.0
backcall==0.2.0
billiard==3.6.4.0
bleach==3.2.1
boto==2.49.0
boto3==1.16.9
botocore==1.19.9
bump2version==1.0.1
bumpversion==0.6.0
case==1.5.3
cassandra-driver==3.20.2
-e git+git@github.com:celery/celery.git@7b5a44d646f43288fb546da10a1141347b01543b#egg=celery
Cerberus==1.3.2
certifi==2020.12.5
cffi==1.14.5
cfgv==3.2.0
chardet==4.0.0
click==7.1.2
click-didyoumean==0.0.3
click-plugins==1.1.1
click-repl==0.1.6
codecov==2.1.11
colorama==0.4.4
couchbase==3.1.3
coverage==5.4
cryptography==3.4.6
DateTime==4.3
decorator==4.4.2
deepmerge==0.1.1
dephell-archive==0.1.7
dephell-argparse==0.1.3
dephell-changelogs==0.0.1
dephell-discover==0.2.10
dephell-licenses==0.1.7
dephell-links==0.1.5
dephell-markers==1.0.3
dephell-pythons==0.1.15
dephell-setuptools==0.2.4
dephell-shells==0.1.5
dephell-specifier==0.2.2
dephell-venvs==0.1.18
dephell-versioning==0.1.2
diff-cover==4.2.3
distlib==0.3.1
dnspython==1.16.0
docker==4.3.1
docutils==0.16
durationpy==0.5
ecdsa==0.16.0
elasticsearch==7.9.1
ephem==3.7.7.1
eventlet==0.29.1
filelock==3.0.12
flower==0.9.5
future==0.18.2
gevent==21.1.2
greenlet==0.4.17
humanize==3.1.0
identify==1.5.6
idna==2.10
imagesize==1.2.0
importlib-metadata==4.0.1
inflect==3.0.2
iniconfig==1.1.1
ipython==7.19.0
ipython-genutils==0.2.0
isort==5.9.1
jedi==0.17.2
jeepney==0.6.0
Jinja2==2.11.2
jinja2-pluralize==0.3.0
jmespath==0.10.0
jsondiff==1.1.1
jsonpickle==1.4.1
keyring==21.5.0
kombu==5.1.0
linecache2==1.0.0
m2r==0.2.1
MarkupSafe==1.1.1
mistune==0.8.4
mock==4.0.2
moto==1.3.7
msgpack==1.0.0
multidict==5.1.0
mypy-extensions==0.4.3
nodeenv==1.5.0
nose==1.3.7
packaging==20.9
parso==0.7.1
pbr==5.5.1
pexpect==4.8.0
pickleshare==0.7.5
pkginfo==1.6.1
pluggy==0.13.1
pre-commit==2.8.2
prometheus-client==0.8.0
prompt-toolkit @ file:///home/thedrow/.cache/pypoetry/artifacts/85/ed/40/229a34dee2081fd935c85f600bfcddb0974a96a88f099412c62b58d4fa/prompt_toolkit-3.0.3-py3-none-any.whl
ptyprocess==0.6.0
py==1.10.0
pyaml==20.4.0
pyArango==1.3.4
pycodestyle==2.7.0
pycouchdb==1.14.1
pycparser==2.20
pycryptodome==3.9.8
pycurl==7.43.0.5
pydocumentdb==2.3.2
pyflakes==2.2.0
Pygments==2.7.3
pylibmc==1.6.1
pymongo==3.11.0
pyparsing==2.4.7
pyrsistent==0.17.3
pytest==6.2.2
pytest-celery==0.0.0a1
pytest-cov==2.11.1
pytest-rerunfailures==9.1.1
pytest-subtests==0.3.2
pytest-timeout==1.4.2
pytest-travis-fold==1.3.0
python-consul==1.1.0
python-dateutil==2.8.1
python-jose==2.0.2
python-memcached==1.59
pytz @ file:///home/thedrow/.cache/pypoetry/artifacts/b0/a7/8d/54de3ab4d1ff29abbbca1e9ccbaefdc2a1b290138311b84f73bee16de1/pytz-2021.1-py2.py3-none-any.whl
pyupgrade==2.19.1
pyupgrade-directories==0.2.0
PyYAML==5.3.1
readme-renderer==28.0
redis==3.5.3
requests==2.25.1
requests-toolbelt==0.9.1
responses==0.12.0
rfc3986==1.4.0
ruamel.yaml==0.16.13
ruamel.yaml.clib==0.2.2
s3transfer==0.3.3
SecretStorage==3.3.0
shellingham==1.4.0
simplejson==3.17.2
six==1.15.0
snowballstemmer==2.0.0
softlayer-messaging==1.0.3
Sphinx==3.3.0
sphinx-celery==2.0.0
sphinx-click==2.5.0
sphinx-testing==0.7.2
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==1.0.3
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.4
SQLAlchemy==1.3.23
tblib==1.7.0
tokenize-rt==4.1.0
toml==0.10.2
toml-sort==0.18.0
tomlkit==0.7.0
tornado==6.1
tqdm==4.54.1
traceback2==1.4.0
traitlets==5.0.5
twine==3.4.1
typing-extensions==3.7.4.3
unittest2==1.1.0
urllib3==1.26.3
vine==5.0.0
virtualenv==20.1.0
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==0.57.0
Werkzeug==1.0.1
wrapt==1.12.1
xmltodict==0.12.0
yarl==1.6.3
yaspin==1.4.1
zipp==3.4.0
zope.event==4.5.0
zope.interface==5.1.2"><pre class="notranslate"><code class="notranslate">aiohttp==3.7.4.post0
alabaster==0.7.12
amqp==5.0.6
appdirs==1.4.4
async-timeout==3.0.1
attrs==20.3.0
autoflake==1.4
autopep8==1.5.7
aws-xray-sdk==0.95
azure-common==1.1.5
azure-nspkg==3.0.2
azure-storage==0.36.0
azure-storage-common==1.1.0
azure-storage-nspkg==3.1.0
Babel==2.8.0
backcall==0.2.0
billiard==3.6.4.0
bleach==3.2.1
boto==2.49.0
boto3==1.16.9
botocore==1.19.9
bump2version==1.0.1
bumpversion==0.6.0
case==1.5.3
cassandra-driver==3.20.2
-e git+git@github.com:celery/celery.git@7b5a44d646f43288fb546da10a1141347b01543b#egg=celery
Cerberus==1.3.2
certifi==2020.12.5
cffi==1.14.5
cfgv==3.2.0
chardet==4.0.0
click==7.1.2
click-didyoumean==0.0.3
click-plugins==1.1.1
click-repl==0.1.6
codecov==2.1.11
colorama==0.4.4
couchbase==3.1.3
coverage==5.4
cryptography==3.4.6
DateTime==4.3
decorator==4.4.2
deepmerge==0.1.1
dephell-archive==0.1.7
dephell-argparse==0.1.3
dephell-changelogs==0.0.1
dephell-discover==0.2.10
dephell-licenses==0.1.7
dephell-links==0.1.5
dephell-markers==1.0.3
dephell-pythons==0.1.15
dephell-setuptools==0.2.4
dephell-shells==0.1.5
dephell-specifier==0.2.2
dephell-venvs==0.1.18
dephell-versioning==0.1.2
diff-cover==4.2.3
distlib==0.3.1
dnspython==1.16.0
docker==4.3.1
docutils==0.16
durationpy==0.5
ecdsa==0.16.0
elasticsearch==7.9.1
ephem==3.7.7.1
eventlet==0.29.1
filelock==3.0.12
flower==0.9.5
future==0.18.2
gevent==21.1.2
greenlet==0.4.17
humanize==3.1.0
identify==1.5.6
idna==2.10
imagesize==1.2.0
importlib-metadata==4.0.1
inflect==3.0.2
iniconfig==1.1.1
ipython==7.19.0
ipython-genutils==0.2.0
isort==5.9.1
jedi==0.17.2
jeepney==0.6.0
Jinja2==2.11.2
jinja2-pluralize==0.3.0
jmespath==0.10.0
jsondiff==1.1.1
jsonpickle==1.4.1
keyring==21.5.0
kombu==5.1.0
linecache2==1.0.0
m2r==0.2.1
MarkupSafe==1.1.1
mistune==0.8.4
mock==4.0.2
moto==1.3.7
msgpack==1.0.0
multidict==5.1.0
mypy-extensions==0.4.3
nodeenv==1.5.0
nose==1.3.7
packaging==20.9
parso==0.7.1
pbr==5.5.1
pexpect==4.8.0
pickleshare==0.7.5
pkginfo==1.6.1
pluggy==0.13.1
pre-commit==2.8.2
prometheus-client==0.8.0
prompt-toolkit @ file:///home/thedrow/.cache/pypoetry/artifacts/85/ed/40/229a34dee2081fd935c85f600bfcddb0974a96a88f099412c62b58d4fa/prompt_toolkit-3.0.3-py3-none-any.whl
ptyprocess==0.6.0
py==1.10.0
pyaml==20.4.0
pyArango==1.3.4
pycodestyle==2.7.0
pycouchdb==1.14.1
pycparser==2.20
pycryptodome==3.9.8
pycurl==7.43.0.5
pydocumentdb==2.3.2
pyflakes==2.2.0
Pygments==2.7.3
pylibmc==1.6.1
pymongo==3.11.0
pyparsing==2.4.7
pyrsistent==0.17.3
pytest==6.2.2
pytest-celery==0.0.0a1
pytest-cov==2.11.1
pytest-rerunfailures==9.1.1
pytest-subtests==0.3.2
pytest-timeout==1.4.2
pytest-travis-fold==1.3.0
python-consul==1.1.0
python-dateutil==2.8.1
python-jose==2.0.2
python-memcached==1.59
pytz @ file:///home/thedrow/.cache/pypoetry/artifacts/b0/a7/8d/54de3ab4d1ff29abbbca1e9ccbaefdc2a1b290138311b84f73bee16de1/pytz-2021.1-py2.py3-none-any.whl
pyupgrade==2.19.1
pyupgrade-directories==0.2.0
PyYAML==5.3.1
readme-renderer==28.0
redis==3.5.3
requests==2.25.1
requests-toolbelt==0.9.1
responses==0.12.0
rfc3986==1.4.0
ruamel.yaml==0.16.13
ruamel.yaml.clib==0.2.2
s3transfer==0.3.3
SecretStorage==3.3.0
shellingham==1.4.0
simplejson==3.17.2
six==1.15.0
snowballstemmer==2.0.0
softlayer-messaging==1.0.3
Sphinx==3.3.0
sphinx-celery==2.0.0
sphinx-click==2.5.0
sphinx-testing==0.7.2
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==1.0.3
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.4
SQLAlchemy==1.3.23
tblib==1.7.0
tokenize-rt==4.1.0
toml==0.10.2
toml-sort==0.18.0
tomlkit==0.7.0
tornado==6.1
tqdm==4.54.1
traceback2==1.4.0
traitlets==5.0.5
twine==3.4.1
typing-extensions==3.7.4.3
unittest2==1.1.0
urllib3==1.26.3
vine==5.0.0
virtualenv==20.1.0
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==0.57.0
Werkzeug==1.0.1
wrapt==1.12.1
xmltodict==0.12.0
yarl==1.6.3
yaspin==1.4.1
zipp==3.4.0
zope.event==4.5.0
zope.interface==5.1.2
</code></pre></div>
<p dir="auto"></p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
N/A
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>

<p dir="auto">To reproduce the issue you must force close the connection and cause AsynPool to be flushed.<br>
Missing a heartbeat due to a breakpoint does the trick.<br>
See </p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/celery/celery/blob/681e72edb918c8ff315665a6abbfc6dd99f303e2/celery/concurrency/asynpool.py#L978-L1038">celery/celery/concurrency/asynpool.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
        Lines 978 to 1038
      in
      <a data-pjax="true" class="commit-tease-sha" href="/celery/celery/commit/681e72edb918c8ff315665a6abbfc6dd99f303e2">681e72e</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L978" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="978"></td>
          <td id="LC978" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">def</span> <span class="pl-en">flush</span>(<span class="pl-s1">self</span>): </td>
        </tr>

        <tr class="border-0">
          <td id="L979" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="979"></td>
          <td id="LC979" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-k">if</span> <span class="pl-s1">self</span>.<span class="pl-s1">_state</span> <span class="pl-c1">==</span> <span class="pl-v">TERMINATE</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L980" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="980"></td>
          <td id="LC980" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         <span class="pl-k">return</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L981" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="981"></td>
          <td id="LC981" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-c"># cancel all tasks that haven't been accepted so that NACK is sent.</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L982" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="982"></td>
          <td id="LC982" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-k">for</span> <span class="pl-s1">job</span> <span class="pl-c1">in</span> <span class="pl-s1">self</span>.<span class="pl-s1">_cache</span>.<span class="pl-en">values</span>(): </td>
        </tr>

        <tr class="border-0">
          <td id="L983" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="983"></td>
          <td id="LC983" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s1">job</span>.<span class="pl-s1">_accepted</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L984" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="984"></td>
          <td id="LC984" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">             <span class="pl-s1">job</span>.<span class="pl-en">_cancel</span>() </td>
        </tr>

        <tr class="border-0">
          <td id="L985" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="985"></td>
          <td id="LC985" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">  </td>
        </tr>

        <tr class="border-0">
          <td id="L986" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="986"></td>
          <td id="LC986" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-c"># clear the outgoing buffer as the tasks will be redelivered by</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L987" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="987"></td>
          <td id="LC987" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-c"># the broker anyway.</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L988" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="988"></td>
          <td id="LC988" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-k">if</span> <span class="pl-s1">self</span>.<span class="pl-s1">outbound_buffer</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L989" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="989"></td>
          <td id="LC989" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         <span class="pl-s1">self</span>.<span class="pl-s1">outbound_buffer</span>.<span class="pl-en">clear</span>() </td>
        </tr>

        <tr class="border-0">
          <td id="L990" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="990"></td>
          <td id="LC990" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">  </td>
        </tr>

        <tr class="border-0">
          <td id="L991" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="991"></td>
          <td id="LC991" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">self</span>.<span class="pl-en">maintain_pool</span>() </td>
        </tr>

        <tr class="border-0">
          <td id="L992" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="992"></td>
          <td id="LC992" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">  </td>
        </tr>

        <tr class="border-0">
          <td id="L993" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="993"></td>
          <td id="LC993" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-k">try</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L994" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="994"></td>
          <td id="LC994" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         <span class="pl-c"># ...but we must continue writing the payloads we already started</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L995" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="995"></td>
          <td id="LC995" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         <span class="pl-c"># to keep message boundaries.</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L996" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="996"></td>
          <td id="LC996" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         <span class="pl-c"># The messages may be NACK'ed later if synack is enabled.</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L997" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="997"></td>
          <td id="LC997" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         <span class="pl-k">if</span> <span class="pl-s1">self</span>.<span class="pl-s1">_state</span> <span class="pl-c1">==</span> <span class="pl-v">RUN</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L998" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="998"></td>
          <td id="LC998" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">             <span class="pl-c"># flush outgoing buffers</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L999" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="999"></td>
          <td id="LC999" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">             <span class="pl-s1">intervals</span> <span class="pl-c1">=</span> <span class="pl-en">fxrange</span>(<span class="pl-c1">0.01</span>, <span class="pl-c1">0.1</span>, <span class="pl-c1">0.01</span>, <span class="pl-s1">repeatlast</span><span class="pl-c1">=</span><span class="pl-c1">True</span>) </td>
        </tr>

        <tr class="border-0">
          <td id="L1000" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1000"></td>
          <td id="LC1000" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">             <span class="pl-s1">owned_by</span> <span class="pl-c1">=</span> {} </td>
        </tr>

        <tr class="border-0">
          <td id="L1001" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1001"></td>
          <td id="LC1001" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">             <span class="pl-k">for</span> <span class="pl-s1">job</span> <span class="pl-c1">in</span> <span class="pl-s1">self</span>.<span class="pl-s1">_cache</span>.<span class="pl-en">values</span>(): </td>
        </tr>

        <tr class="border-0">
          <td id="L1002" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1002"></td>
          <td id="LC1002" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                 <span class="pl-s1">writer</span> <span class="pl-c1">=</span> <span class="pl-en">_get_job_writer</span>(<span class="pl-s1">job</span>) </td>
        </tr>

        <tr class="border-0">
          <td id="L1003" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1003"></td>
          <td id="LC1003" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                 <span class="pl-k">if</span> <span class="pl-s1">writer</span> <span class="pl-c1">is</span> <span class="pl-c1">not</span> <span class="pl-c1">None</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L1004" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1004"></td>
          <td id="LC1004" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                     <span class="pl-s1">owned_by</span>[<span class="pl-s1">writer</span>] <span class="pl-c1">=</span> <span class="pl-s1">job</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L1005" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1005"></td>
          <td id="LC1005" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">  </td>
        </tr>

        <tr class="border-0">
          <td id="L1006" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1006"></td>
          <td id="LC1006" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">             <span class="pl-k">while</span> <span class="pl-s1">self</span>.<span class="pl-s1">_active_writers</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L1007" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1007"></td>
          <td id="LC1007" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                 <span class="pl-s1">writers</span> <span class="pl-c1">=</span> <span class="pl-en">list</span>(<span class="pl-s1">self</span>.<span class="pl-s1">_active_writers</span>) </td>
        </tr>

        <tr class="border-0">
          <td id="L1008" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1008"></td>
          <td id="LC1008" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                 <span class="pl-k">for</span> <span class="pl-s1">gen</span> <span class="pl-c1">in</span> <span class="pl-s1">writers</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L1009" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1009"></td>
          <td id="LC1009" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                     <span class="pl-k">if</span> (<span class="pl-s1">gen</span>.<span class="pl-s1">__name__</span> <span class="pl-c1">==</span> <span class="pl-s">'_write_job'</span> <span class="pl-c1">and</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L1010" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1010"></td>
          <td id="LC1010" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                             <span class="pl-en">gen_not_started</span>(<span class="pl-s1">gen</span>)): </td>
        </tr>

        <tr class="border-0">
          <td id="L1011" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1011"></td>
          <td id="LC1011" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                         <span class="pl-c"># hasn't started writing the job so can</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L1012" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1012"></td>
          <td id="LC1012" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                         <span class="pl-c"># discard the task, but we must also remove</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L1013" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1013"></td>
          <td id="LC1013" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                         <span class="pl-c"># it from the Pool._cache.</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L1014" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1014"></td>
          <td id="LC1014" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                         <span class="pl-k">try</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L1015" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1015"></td>
          <td id="LC1015" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                             <span class="pl-s1">job</span> <span class="pl-c1">=</span> <span class="pl-s1">owned_by</span>[<span class="pl-s1">gen</span>] </td>
        </tr>

        <tr class="border-0">
          <td id="L1016" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1016"></td>
          <td id="LC1016" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                         <span class="pl-k">except</span> <span class="pl-v">KeyError</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L1017" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1017"></td>
          <td id="LC1017" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                             <span class="pl-k">pass</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L1018" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1018"></td>
          <td id="LC1018" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                         <span class="pl-k">else</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L1019" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1019"></td>
          <td id="LC1019" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                             <span class="pl-c"># removes from Pool._cache</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L1020" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1020"></td>
          <td id="LC1020" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                             <span class="pl-s1">job</span>.<span class="pl-en">discard</span>() </td>
        </tr>

        <tr class="border-0">
          <td id="L1021" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1021"></td>
          <td id="LC1021" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                         <span class="pl-s1">self</span>.<span class="pl-s1">_active_writers</span>.<span class="pl-en">discard</span>(<span class="pl-s1">gen</span>) </td>
        </tr>

        <tr class="border-0">
          <td id="L1022" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1022"></td>
          <td id="LC1022" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                     <span class="pl-k">else</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L1023" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1023"></td>
          <td id="LC1023" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                         <span class="pl-k">try</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L1024" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1024"></td>
          <td id="LC1024" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                             <span class="pl-s1">job</span> <span class="pl-c1">=</span> <span class="pl-s1">owned_by</span>[<span class="pl-s1">gen</span>] </td>
        </tr>

        <tr class="border-0">
          <td id="L1025" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1025"></td>
          <td id="LC1025" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                         <span class="pl-k">except</span> <span class="pl-v">KeyError</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L1026" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1026"></td>
          <td id="LC1026" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                             <span class="pl-k">pass</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L1027" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1027"></td>
          <td id="LC1027" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                         <span class="pl-k">else</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L1028" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1028"></td>
          <td id="LC1028" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                             <span class="pl-s1">job_proc</span> <span class="pl-c1">=</span> <span class="pl-s1">job</span>.<span class="pl-s1">_write_to</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L1029" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1029"></td>
          <td id="LC1029" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                             <span class="pl-k">if</span> <span class="pl-s1">job_proc</span>.<span class="pl-en">_is_alive</span>(): </td>
        </tr>

        <tr class="border-0">
          <td id="L1030" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1030"></td>
          <td id="LC1030" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                                 <span class="pl-s1">self</span>.<span class="pl-en">_flush_writer</span>(<span class="pl-s1">job_proc</span>, <span class="pl-s1">gen</span>) </td>
        </tr>

        <tr class="border-0">
          <td id="L1031" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1031"></td>
          <td id="LC1031" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                 <span class="pl-c"># workers may have exited in the meantime.</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L1032" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1032"></td>
          <td id="LC1032" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                 <span class="pl-s1">self</span>.<span class="pl-en">maintain_pool</span>() </td>
        </tr>

        <tr class="border-0">
          <td id="L1033" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1033"></td>
          <td id="LC1033" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">                 <span class="pl-en">sleep</span>(<span class="pl-en">next</span>(<span class="pl-s1">intervals</span>))  <span class="pl-c"># don't busyloop</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L1034" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1034"></td>
          <td id="LC1034" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-k">finally</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L1035" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1035"></td>
          <td id="LC1035" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         <span class="pl-s1">self</span>.<span class="pl-s1">outbound_buffer</span>.<span class="pl-en">clear</span>() </td>
        </tr>

        <tr class="border-0">
          <td id="L1036" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1036"></td>
          <td id="LC1036" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         <span class="pl-s1">self</span>.<span class="pl-s1">_active_writers</span>.<span class="pl-en">clear</span>() </td>
        </tr>

        <tr class="border-0">
          <td id="L1037" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1037"></td>
          <td id="LC1037" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         <span class="pl-s1">self</span>.<span class="pl-s1">_active_writes</span>.<span class="pl-en">clear</span>() </td>
        </tr>

        <tr class="border-0">
          <td id="L1038" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1038"></td>
          <td id="LC1038" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">         <span class="pl-s1">self</span>.<span class="pl-s1">_busy_workers</span>.<span class="pl-en">clear</span>() </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<details>
<p dir="auto">
</p><div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# Sending any task would do
some_task.delay()"><pre class="notranslate"><span class="pl-c"># Sending any task would do</span>
<span class="pl-s1">some_task</span>.<span class="pl-en">delay</span>()</pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>

<p dir="auto">The pool's job cache should be completely empty after we flush it.</p>
<h1 dir="auto">Actual Behavior</h1>

<p dir="auto">Canceled jobs remain in the cache and are never cleaned up since the pool doesn't use the worker's synack capability (not sure what that is, actually).<br>
This is a problem since we keep the job's payload (which may be arbitrarily large) forever.<br>
Given enough uptime, whenever there are enough connection drops, we accumulate new jobs that are never garbage collected.</p>
<p dir="auto">This has been the case from the moment AsynPool has been written, so it seems.</p>
<h2 dir="auto">Environment &amp; Settings</h2>
<p dir="auto"><strong>Celery version</strong>:</p>
<ul dir="auto">
<li>celery==4.2.1 but also tried the latest master, where the problem also occurs:</li>
<li>-e git+<a href="https://github.com/celery/celery.git@c1d0bfea9ad98477cbc1def99157fe5109555500#egg=celery">https://github.com/celery/celery.git@c1d0bfea9ad98477cbc1def99157fe5109555500#egg=celery</a></li>
<li>django-celery-beat==1.4.0</li>
<li>kombu==4.2.2.post1</li>
</ul>
<p dir="auto"><strong>Report</strong>:</p>
<details>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="software -&gt; celery:4.2.0 (windowlicker) kombu:4.2.2-post1 py:3.6.6
            billiard:3.5.0.5 sqs:N/A
platform -&gt; system:Linux arch:64bit, ELF
            kernel version:3.13.0-139-generic imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:sqs results:disabled

broker_url: 'sqs://localhost//'
include: [...]
worker_hijack_root_logger: False
task_serializer: 'json'
result_expires: 3600
accept_content: ['json']
result_serializer: 'json'
timezone: 'Europe/Berlin'
enable_utc: True
broker_transport_options: {
    'polling_interval': 1,
    'region': 'eu-west-1',
    'visibility_timeout': 10860}
task_ignore_result: True
task_acks_late: True
worker_prefetch_multiplier: 1
worker_max_tasks_per_child: 10
worker_pool: 'celery.concurrency.prefork:TaskPool'
task_time_limit: 10800
worker_enable_remote_control: False
worker_send_task_events: False
task_default_queue: 'celery'"><pre class="notranslate"><code class="notranslate">software -&gt; celery:4.2.0 (windowlicker) kombu:4.2.2-post1 py:3.6.6
            billiard:3.5.0.5 sqs:N/A
platform -&gt; system:Linux arch:64bit, ELF
            kernel version:3.13.0-139-generic imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:sqs results:disabled

broker_url: 'sqs://localhost//'
include: [...]
worker_hijack_root_logger: False
task_serializer: 'json'
result_expires: 3600
accept_content: ['json']
result_serializer: 'json'
timezone: 'Europe/Berlin'
enable_utc: True
broker_transport_options: {
    'polling_interval': 1,
    'region': 'eu-west-1',
    'visibility_timeout': 10860}
task_ignore_result: True
task_acks_late: True
worker_prefetch_multiplier: 1
worker_max_tasks_per_child: 10
worker_pool: 'celery.concurrency.prefork:TaskPool'
task_time_limit: 10800
worker_enable_remote_control: False
worker_send_task_events: False
task_default_queue: 'celery'
</code></pre></div>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: 3.6</li>
<li><strong>Minimal Broker Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Version</strong>: N/A or Unknown</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: : N/A or Unknown</li>
</ul>
<h2 dir="auto">Minimally Reproducible Test Case</h2>
<ul dir="auto">
<li>edit <code class="notranslate">kombu/asynchronous/hub.py</code> to uncomment the print statements in <code class="notranslate">create_loop</code></li>
<li>fire ~500 very short tasks to run (so the main worker process has to exit and start new workers because of <code class="notranslate">worker_max_tasks_per_child: 10</code>)</li>
</ul>
<h1 dir="auto">Expected Behavior</h1>
<p dir="auto">The main worker process should settle to ~1% CPU usage, after the 500 tasks were run. The print statements should be run not so often because of the <code class="notranslate">sleep</code> call at the end of the <code class="notranslate">create_loop</code> method.</p>
<h1 dir="auto">Actual Behavior</h1>
<p dir="auto">The main worker hammers the CPU at constant 100% and the console output (due to the uncommented print statement) is flooded with <strong>lots</strong> (almost every microsecond) of these (never stopping):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="WARNING:celery.redirected: [[[HUB]]]: (31)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-32, started daemon)&gt;)-&gt;R!, (20)on_result_readable(20)-&gt;R!, (34)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-33, started daemon)&gt;)-&gt;R!, (16)on_result_readable(16)-&gt;R!, (42)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-34, started daemon)&gt;)-&gt;R!, (24)on_result_readable(24)-&gt;R!, (53)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-35, started daemon)&gt;)-&gt;R!, (28)on_result_readable(28)-&gt;R!, (38)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-36, started daemon)&gt;)-&gt;R!, (12)on_result_readable(12)-&gt;R!, (45)on_readable(45)-&gt;R!, (45)on_writable(45)-&gt;W (2019-01-22 17:09:04,502; /home/ubuntu/.virtualenvs/unchained-worker3.6/src/celery/celery/utils/log.py:235)
WARNING:celery.redirected: [EVENTS]: (GONE)(48)-&gt;R, (GONE)(46)-&gt;R, (GONE)(44)-&gt;R! (2019-01-22 17:09:04,503; /home/ubuntu/.virtualenvs/unchained-worker3.6/src/celery/celery/utils/log.py:235)
WARNING:celery.redirected: [[[HUB]]]: (31)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-32, started daemon)&gt;)-&gt;R!, (20)on_result_readable(20)-&gt;R!, (34)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-33, started daemon)&gt;)-&gt;R!, (16)on_result_readable(16)-&gt;R!, (42)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-34, started daemon)&gt;)-&gt;R!, (24)on_result_readable(24)-&gt;R!, (53)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-35, started daemon)&gt;)-&gt;R!, (28)on_result_readable(28)-&gt;R!, (38)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-36, started daemon)&gt;)-&gt;R!, (12)on_result_readable(12)-&gt;R!, (45)on_readable(45)-&gt;R!, (45)on_writable(45)-&gt;W (2019-01-22 17:09:04,505; /home/ubuntu/.virtualenvs/unchained-worker3.6/src/celery/celery/utils/log.py:235)
WARNING:celery.redirected: [EVENTS]: (GONE)(48)-&gt;R, (GONE)(46)-&gt;R, (GONE)(44)-&gt;R! (2019-01-22 17:09:04,506; /home/ubuntu/.virtualenvs/unchained-worker3.6/src/celery/celery/utils/log.py:235)
WARNING:celery.redirected: [[[HUB]]]: (31)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-32, started daemon)&gt;)-&gt;R!, (20)on_result_readable(20)-&gt;R!, (34)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-33, started daemon)&gt;)-&gt;R!, (16)on_result_readable(16)-&gt;R!, (42)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-34, started daemon)&gt;)-&gt;R!, (24)on_result_readable(24)-&gt;R!, (53)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-35, started daemon)&gt;)-&gt;R!, (28)on_result_readable(28)-&gt;R!, (38)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-36, started daemon)&gt;)-&gt;R!, (12)on_result_readable(12)-&gt;R!, (45)on_readable(45)-&gt;R!, (45)on_writable(45)-&gt;W (2019-01-22 17:09:04,507; /home/ubuntu/.virtualenvs/unchained-worker3.6/src/celery/celery/utils/log.py:235)
WARNING:celery.redirected: [EVENTS]: (GONE)(48)-&gt;R, (GONE)(46)-&gt;R, (GONE)(44)-&gt;R! (2019-01-22 17:09:04,508; /home/ubuntu/.virtualenvs/unchained-worker3.6/src/celery/celery/utils/log.py:235)
WARNING:celery.redirected: [[[HUB]]]: (31)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-32, started daemon)&gt;)-&gt;R!, (20)on_result_readable(20)-&gt;R!, (34)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-33, started daemon)&gt;)-&gt;R!, (16)on_result_readable(16)-&gt;R!, (42)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-34, started daemon)&gt;)-&gt;R!, (24)on_result_readable(24)-&gt;R!, (53)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-35, started daemon)&gt;)-&gt;R!, (28)on_result_readable(28)-&gt;R!, (38)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-36, started daemon)&gt;)-&gt;R!, (12)on_result_readable(12)-&gt;R!, (45)on_readable(45)-&gt;R!, (45)on_writable(45)-&gt;W (2019-01-22 17:09:04,509; /home/ubuntu/.virtualenvs/unchained-worker3.6/src/celery/celery/utils/log.py:235)
WARNING:celery.redirected: [EVENTS]: (GONE)(48)-&gt;R, (GONE)(46)-&gt;R, (GONE)(44)-&gt;R! (2019-01-22 17:09:04,510; /home/ubuntu/.virtualenvs/unchained-worker3.6/src/celery/celery/utils/log.py:235)
WARNING:celery.redirected: [[[HUB]]]: (31)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-32, started daemon)&gt;)-&gt;R!, (20)on_result_readable(20)-&gt;R!, (34)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-33, started daemon)&gt;)-&gt;R!, (16)on_result_readable(16)-&gt;R!, (42)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-34, started daemon)&gt;)-&gt;R!, (24)on_result_readable(24)-&gt;R!, (53)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-35, started daemon)&gt;)-&gt;R!, (28)on_result_readable(28)-&gt;R!, (38)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-36, started daemon)&gt;)-&gt;R!, (12)on_result_readable(12)-&gt;R!, (45)on_readable(45)-&gt;R!, (45)on_writable(45)-&gt;W (2019-01-22 17:09:04,511; /home/ubuntu/.virtualenvs/unchained-worker3.6/src/celery/celery/utils/log.py:235)
WARNING:celery.redirected: [EVENTS]: (GONE)(48)-&gt;R, (GONE)(46)-&gt;R, (GONE)(44)-&gt;R! (2019-01-22 17:09:04,513; /home/ubuntu/.virtualenvs/unchained-worker3.6/src/celery/celery/utils/log.py:235)"><pre class="notranslate"><code class="notranslate">WARNING:celery.redirected: [[[HUB]]]: (31)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-32, started daemon)&gt;)-&gt;R!, (20)on_result_readable(20)-&gt;R!, (34)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-33, started daemon)&gt;)-&gt;R!, (16)on_result_readable(16)-&gt;R!, (42)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-34, started daemon)&gt;)-&gt;R!, (24)on_result_readable(24)-&gt;R!, (53)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-35, started daemon)&gt;)-&gt;R!, (28)on_result_readable(28)-&gt;R!, (38)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-36, started daemon)&gt;)-&gt;R!, (12)on_result_readable(12)-&gt;R!, (45)on_readable(45)-&gt;R!, (45)on_writable(45)-&gt;W (2019-01-22 17:09:04,502; /home/ubuntu/.virtualenvs/unchained-worker3.6/src/celery/celery/utils/log.py:235)
WARNING:celery.redirected: [EVENTS]: (GONE)(48)-&gt;R, (GONE)(46)-&gt;R, (GONE)(44)-&gt;R! (2019-01-22 17:09:04,503; /home/ubuntu/.virtualenvs/unchained-worker3.6/src/celery/celery/utils/log.py:235)
WARNING:celery.redirected: [[[HUB]]]: (31)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-32, started daemon)&gt;)-&gt;R!, (20)on_result_readable(20)-&gt;R!, (34)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-33, started daemon)&gt;)-&gt;R!, (16)on_result_readable(16)-&gt;R!, (42)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-34, started daemon)&gt;)-&gt;R!, (24)on_result_readable(24)-&gt;R!, (53)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-35, started daemon)&gt;)-&gt;R!, (28)on_result_readable(28)-&gt;R!, (38)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-36, started daemon)&gt;)-&gt;R!, (12)on_result_readable(12)-&gt;R!, (45)on_readable(45)-&gt;R!, (45)on_writable(45)-&gt;W (2019-01-22 17:09:04,505; /home/ubuntu/.virtualenvs/unchained-worker3.6/src/celery/celery/utils/log.py:235)
WARNING:celery.redirected: [EVENTS]: (GONE)(48)-&gt;R, (GONE)(46)-&gt;R, (GONE)(44)-&gt;R! (2019-01-22 17:09:04,506; /home/ubuntu/.virtualenvs/unchained-worker3.6/src/celery/celery/utils/log.py:235)
WARNING:celery.redirected: [[[HUB]]]: (31)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-32, started daemon)&gt;)-&gt;R!, (20)on_result_readable(20)-&gt;R!, (34)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-33, started daemon)&gt;)-&gt;R!, (16)on_result_readable(16)-&gt;R!, (42)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-34, started daemon)&gt;)-&gt;R!, (24)on_result_readable(24)-&gt;R!, (53)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-35, started daemon)&gt;)-&gt;R!, (28)on_result_readable(28)-&gt;R!, (38)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-36, started daemon)&gt;)-&gt;R!, (12)on_result_readable(12)-&gt;R!, (45)on_readable(45)-&gt;R!, (45)on_writable(45)-&gt;W (2019-01-22 17:09:04,507; /home/ubuntu/.virtualenvs/unchained-worker3.6/src/celery/celery/utils/log.py:235)
WARNING:celery.redirected: [EVENTS]: (GONE)(48)-&gt;R, (GONE)(46)-&gt;R, (GONE)(44)-&gt;R! (2019-01-22 17:09:04,508; /home/ubuntu/.virtualenvs/unchained-worker3.6/src/celery/celery/utils/log.py:235)
WARNING:celery.redirected: [[[HUB]]]: (31)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-32, started daemon)&gt;)-&gt;R!, (20)on_result_readable(20)-&gt;R!, (34)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-33, started daemon)&gt;)-&gt;R!, (16)on_result_readable(16)-&gt;R!, (42)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-34, started daemon)&gt;)-&gt;R!, (24)on_result_readable(24)-&gt;R!, (53)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-35, started daemon)&gt;)-&gt;R!, (28)on_result_readable(28)-&gt;R!, (38)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-36, started daemon)&gt;)-&gt;R!, (12)on_result_readable(12)-&gt;R!, (45)on_readable(45)-&gt;R!, (45)on_writable(45)-&gt;W (2019-01-22 17:09:04,509; /home/ubuntu/.virtualenvs/unchained-worker3.6/src/celery/celery/utils/log.py:235)
WARNING:celery.redirected: [EVENTS]: (GONE)(48)-&gt;R, (GONE)(46)-&gt;R, (GONE)(44)-&gt;R! (2019-01-22 17:09:04,510; /home/ubuntu/.virtualenvs/unchained-worker3.6/src/celery/celery/utils/log.py:235)
WARNING:celery.redirected: [[[HUB]]]: (31)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-32, started daemon)&gt;)-&gt;R!, (20)on_result_readable(20)-&gt;R!, (34)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-33, started daemon)&gt;)-&gt;R!, (16)on_result_readable(16)-&gt;R!, (42)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-34, started daemon)&gt;)-&gt;R!, (24)on_result_readable(24)-&gt;R!, (53)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-35, started daemon)&gt;)-&gt;R!, (28)on_result_readable(28)-&gt;R!, (38)_event_process_exit(&lt;Hub@0x7fa987a1dc18: R:11 W:1&gt;, &lt;ForkProcess(ForkPoolWorker-36, started daemon)&gt;)-&gt;R!, (12)on_result_readable(12)-&gt;R!, (45)on_readable(45)-&gt;R!, (45)on_writable(45)-&gt;W (2019-01-22 17:09:04,511; /home/ubuntu/.virtualenvs/unchained-worker3.6/src/celery/celery/utils/log.py:235)
WARNING:celery.redirected: [EVENTS]: (GONE)(48)-&gt;R, (GONE)(46)-&gt;R, (GONE)(44)-&gt;R! (2019-01-22 17:09:04,513; /home/ubuntu/.virtualenvs/unchained-worker3.6/src/celery/celery/utils/log.py:235)
</code></pre></div>
<p dir="auto">also, the <code class="notranslate">strace</code> is full of these:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="epoll_ctl(9, EPOLL_CTL_DEL, 43, {EPOLLRDNORM|EPOLLRDBAND|EPOLLWRNORM|EPOLLERR|0x36f80000, {u32=32711, u64=4295000007}}) = -1 ENOENT (No such file or directory)
epoll_ctl(9, EPOLL_CTL_DEL, 46, {EPOLLERR|0x36ca1800, {u32=32711, u64=4295000007}}) = -1 EBADF (Bad file descriptor)
epoll_ctl(9, EPOLL_CTL_DEL, 44, {EPOLLERR|0x36ca1800, {u32=32711, u64=4295000007}}) = -1 EBADF (Bad file descriptor)
epoll_ctl(9, EPOLL_CTL_DEL, 19, {EPOLLRDNORM|EPOLLRDBAND|EPOLLWRBAND|EPOLLERR|EPOLLRDHUP|EPOLLONESHOT|0x1a0fd820, {u32=32711, u64=4295000007}}) = -1 ENOENT (No such file or directory)
epoll_ctl(9, EPOLL_CTL_DEL, 23, {EPOLLRDNORM|EPOLLRDBAND|EPOLLWRBAND|EPOLLERR|EPOLLRDHUP|EPOLLONESHOT|0x1a0fd820, {u32=32711, u64=4295000007}}) = -1 ENOENT (No such file or directory)
epoll_ctl(9, EPOLL_CTL_DEL, 27, {EPOLLRDNORM|EPOLLRDBAND|EPOLLWRBAND|EPOLLERR|EPOLLRDHUP|EPOLLONESHOT|0x1a0fd820, {u32=32711, u64=4295000007}}) = -1 ENOENT (No such file or directory)
epoll_ctl(9, EPOLL_CTL_DEL, 11, {EPOLLRDNORM|EPOLLRDBAND|EPOLLWRBAND|EPOLLERR|EPOLLRDHUP|EPOLLONESHOT|0x1a0fd820, {u32=32711, u64=4295000007}}) = -1 ENOENT (No such file or directory)
epoll_ctl(9, EPOLL_CTL_DEL, 15, {EPOLLRDNORM|EPOLLRDBAND|EPOLLWRBAND|EPOLLERR|EPOLLRDHUP|EPOLLONESHOT|0x1a0fd820, {u32=32711, u64=4295000007}}) = -1 ENOENT (No such file or directory)"><pre class="notranslate"><code class="notranslate">epoll_ctl(9, EPOLL_CTL_DEL, 43, {EPOLLRDNORM|EPOLLRDBAND|EPOLLWRNORM|EPOLLERR|0x36f80000, {u32=32711, u64=4295000007}}) = -1 ENOENT (No such file or directory)
epoll_ctl(9, EPOLL_CTL_DEL, 46, {EPOLLERR|0x36ca1800, {u32=32711, u64=4295000007}}) = -1 EBADF (Bad file descriptor)
epoll_ctl(9, EPOLL_CTL_DEL, 44, {EPOLLERR|0x36ca1800, {u32=32711, u64=4295000007}}) = -1 EBADF (Bad file descriptor)
epoll_ctl(9, EPOLL_CTL_DEL, 19, {EPOLLRDNORM|EPOLLRDBAND|EPOLLWRBAND|EPOLLERR|EPOLLRDHUP|EPOLLONESHOT|0x1a0fd820, {u32=32711, u64=4295000007}}) = -1 ENOENT (No such file or directory)
epoll_ctl(9, EPOLL_CTL_DEL, 23, {EPOLLRDNORM|EPOLLRDBAND|EPOLLWRBAND|EPOLLERR|EPOLLRDHUP|EPOLLONESHOT|0x1a0fd820, {u32=32711, u64=4295000007}}) = -1 ENOENT (No such file or directory)
epoll_ctl(9, EPOLL_CTL_DEL, 27, {EPOLLRDNORM|EPOLLRDBAND|EPOLLWRBAND|EPOLLERR|EPOLLRDHUP|EPOLLONESHOT|0x1a0fd820, {u32=32711, u64=4295000007}}) = -1 ENOENT (No such file or directory)
epoll_ctl(9, EPOLL_CTL_DEL, 11, {EPOLLRDNORM|EPOLLRDBAND|EPOLLWRBAND|EPOLLERR|EPOLLRDHUP|EPOLLONESHOT|0x1a0fd820, {u32=32711, u64=4295000007}}) = -1 ENOENT (No such file or directory)
epoll_ctl(9, EPOLL_CTL_DEL, 15, {EPOLLRDNORM|EPOLLRDBAND|EPOLLWRBAND|EPOLLERR|EPOLLRDHUP|EPOLLONESHOT|0x1a0fd820, {u32=32711, u64=4295000007}}) = -1 ENOENT (No such file or directory)
</code></pre></div>
<p dir="auto">There have been similar issues around, but they most of the time were related to <code class="notranslate">redis</code>. Also what's different to former issues, is that the worker still distributes / consumes new tasks. So basically, everything works, but the CPU is not idling, but hammering the loop instead. I can consistently reproduce this with my SQS worker (concurrency 5 btw), so please let me know, what else information i can gather to hunt down this problem. THX!</p>