<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> This has already been asked to the <a href="https://groups.google.com/forum/#!forum/celery-users" rel="nofollow">discussion group</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="23412453" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/1700" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/1700/hovercard" href="https://github.com/celery/celery/issues/1700">#1700</a></li>
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="97109603" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/2725" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/2725/hovercard" href="https://github.com/celery/celery/issues/2725">#2725</a></li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>: 4.4.2/master (<a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/celery/celery/commit/20b7a808398b7b865651a5cf0fd7aa2b74be3a4b/hovercard" href="https://github.com/celery/celery/commit/20b7a808398b7b865651a5cf0fd7aa2b74be3a4b"><tt>20b7a80</tt></a>)</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="developer@b7825f7f6305:~/celery$ celery -A tasks report

software -&gt; celery:4.4.2 (cliffs) kombu:4.6.8 py:3.8.0
            billiard:3.6.3.0 py-amqp:2.5.2
platform -&gt; system:Linux arch:64bit, ELF
            kernel version:3.10.0-957.el7.x86_64 imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:amqp results:db+sqlite:///results.sqlite

broker_url: 'amqp://guest:********@rabbit:5672//'
result_backend: 'db+sqlite:///results.sqlite'"><pre class="notranslate"><code class="notranslate">developer@b7825f7f6305:~/celery$ celery -A tasks report

software -&gt; celery:4.4.2 (cliffs) kombu:4.6.8 py:3.8.0
            billiard:3.6.3.0 py-amqp:2.5.2
platform -&gt; system:Linux arch:64bit, ELF
            kernel version:3.10.0-957.el7.x86_64 imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:amqp results:db+sqlite:///results.sqlite

broker_url: 'amqp://guest:********@rabbit:5672//'
result_backend: 'db+sqlite:///results.sqlite'
</code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Celery Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Kombu Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Version</strong>: N/A or Unknown</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Client Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="alabaster==0.7.12
amqp==2.5.2
appdirs==1.4.3
attrs==19.3.0
aws-xray-sdk==0.95
azure-common==1.1.5
azure-nspkg==3.0.2
azure-storage==0.36.0
azure-storage-common==1.1.0
azure-storage-nspkg==3.1.0
Babel==2.8.0
basho-erlastic==2.1.1
billiard==3.6.3.0
boto==2.49.0
boto3==1.12.37
botocore==1.15.37
bumpversion==0.5.3
case==1.5.3
cassandra-driver==3.23.0
-e git+https://github.com/celery/celery.git@20b7a808398b7b865651a5cf0fd7aa2b74be3a4b#egg=celery
certifi==2020.4.5.1
cffi==1.14.0
cfgv==3.1.0
chardet==3.0.4
click==7.1.1
codecov==2.0.22
couchbase==2.5.11
coverage==5.0.4
cryptography==2.9
DateTime==4.3
distlib==0.3.0
dnspython==1.16.0
docker==4.2.0
docutils==0.16
ecdsa==0.15
elasticsearch==7.6.0
entrypoints==0.3
ephem==3.7.7.1
eventlet==0.25.1
filelock==3.0.12
flake8==3.7.9
flakeplus==1.1.0
future==0.18.2
geomet==0.1.2
gevent==1.4.0
greenlet==0.4.15
identify==1.4.14
idna==2.9
imagesize==1.2.0
Jinja2==2.11.1
jmespath==0.9.5
jsondiff==1.1.1
jsonpickle==1.3
kombu==4.6.8
linecache2==1.0.0
MarkupSafe==1.1.1
mccabe==0.6.1
mock==4.0.2
monotonic==1.5
more-itertools==8.2.0
moto==1.3.7
msgpack==1.0.0
nodeenv==1.3.5
nose==1.3.7
packaging==20.3
pluggy==0.13.1
pre-commit==2.2.0
py==1.8.1
pyaml==20.4.0
pyArango==1.3.4
pycodestyle==2.5.0
pycouchdb==1.14.1
pycparser==2.20
pycryptodome==3.9.7
pycurl==7.43.0.5
pydocstyle==1.1.1
pydocumentdb==2.3.2
pyflakes==2.1.1
Pygments==2.6.1
pylibmc==1.6.1
pymongo==3.10.1
pyparsing==2.4.7
pytest==5.3.4
pytest-cov==2.8.1
pytest-rerunfailures==9.0
pytest-sugar==0.9.2
pytest-travis-fold==1.3.0
python-consul==1.1.0
python-dateutil==2.8.1
python-jose==2.0.2
python-memcached==1.59
python3-protobuf==2.5.0
pytz==2019.3
PyYAML==5.3.1
redis==3.4.1
requests==2.23.0
responses==0.10.12
riak==2.7.0
s3transfer==0.3.3
simplejson==3.17.0
six==1.14.0
snowballstemmer==2.0.0
softlayer-messaging==1.0.3
Sphinx==1.8.5
sphinx-celery==1.4.6
sphinx-testing==0.7.2
sphinx2rst==1.1.0
sphinxcontrib-websupport==1.2.1
SQLAlchemy==1.3.15
tblib==1.6.0
termcolor==1.1.0
toml==0.10.0
tox==3.14.6
traceback2==1.4.0
Unipath==1.1
unittest2==1.1.0
urllib3==1.25.8
vine==1.3.0
virtualenv==20.0.16
wcwidth==0.1.9
websocket-client==0.57.0
Werkzeug==1.0.1
wrapt==1.12.1
xmltodict==0.12.0
zope.interface==5.0.2"><pre class="notranslate"><code class="notranslate">alabaster==0.7.12
amqp==2.5.2
appdirs==1.4.3
attrs==19.3.0
aws-xray-sdk==0.95
azure-common==1.1.5
azure-nspkg==3.0.2
azure-storage==0.36.0
azure-storage-common==1.1.0
azure-storage-nspkg==3.1.0
Babel==2.8.0
basho-erlastic==2.1.1
billiard==3.6.3.0
boto==2.49.0
boto3==1.12.37
botocore==1.15.37
bumpversion==0.5.3
case==1.5.3
cassandra-driver==3.23.0
-e git+https://github.com/celery/celery.git@20b7a808398b7b865651a5cf0fd7aa2b74be3a4b#egg=celery
certifi==2020.4.5.1
cffi==1.14.0
cfgv==3.1.0
chardet==3.0.4
click==7.1.1
codecov==2.0.22
couchbase==2.5.11
coverage==5.0.4
cryptography==2.9
DateTime==4.3
distlib==0.3.0
dnspython==1.16.0
docker==4.2.0
docutils==0.16
ecdsa==0.15
elasticsearch==7.6.0
entrypoints==0.3
ephem==3.7.7.1
eventlet==0.25.1
filelock==3.0.12
flake8==3.7.9
flakeplus==1.1.0
future==0.18.2
geomet==0.1.2
gevent==1.4.0
greenlet==0.4.15
identify==1.4.14
idna==2.9
imagesize==1.2.0
Jinja2==2.11.1
jmespath==0.9.5
jsondiff==1.1.1
jsonpickle==1.3
kombu==4.6.8
linecache2==1.0.0
MarkupSafe==1.1.1
mccabe==0.6.1
mock==4.0.2
monotonic==1.5
more-itertools==8.2.0
moto==1.3.7
msgpack==1.0.0
nodeenv==1.3.5
nose==1.3.7
packaging==20.3
pluggy==0.13.1
pre-commit==2.2.0
py==1.8.1
pyaml==20.4.0
pyArango==1.3.4
pycodestyle==2.5.0
pycouchdb==1.14.1
pycparser==2.20
pycryptodome==3.9.7
pycurl==7.43.0.5
pydocstyle==1.1.1
pydocumentdb==2.3.2
pyflakes==2.1.1
Pygments==2.6.1
pylibmc==1.6.1
pymongo==3.10.1
pyparsing==2.4.7
pytest==5.3.4
pytest-cov==2.8.1
pytest-rerunfailures==9.0
pytest-sugar==0.9.2
pytest-travis-fold==1.3.0
python-consul==1.1.0
python-dateutil==2.8.1
python-jose==2.0.2
python-memcached==1.59
python3-protobuf==2.5.0
pytz==2019.3
PyYAML==5.3.1
redis==3.4.1
requests==2.23.0
responses==0.10.12
riak==2.7.0
s3transfer==0.3.3
simplejson==3.17.0
six==1.14.0
snowballstemmer==2.0.0
softlayer-messaging==1.0.3
Sphinx==1.8.5
sphinx-celery==1.4.6
sphinx-testing==0.7.2
sphinx2rst==1.1.0
sphinxcontrib-websupport==1.2.1
SQLAlchemy==1.3.15
tblib==1.6.0
termcolor==1.1.0
toml==0.10.0
tox==3.14.6
traceback2==1.4.0
Unipath==1.1
unittest2==1.1.0
urllib3==1.25.8
vine==1.3.0
virtualenv==20.0.16
wcwidth==0.1.9
websocket-client==0.57.0
Werkzeug==1.0.1
wrapt==1.12.1
xmltodict==0.12.0
zope.interface==5.0.2
</code></pre></div>
<p dir="auto"></p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
N/A
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>

<details>
<p dir="auto">
</p><div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from celery import Celery, chain, group
app = Celery('tasks',
          backend='db+sqlite:///results.sqlite',
          broker='amqp://guest:guest@rabbit:5672//')

@app.task()
def foo():
    print(&quot;this is from foo!&quot;)

@app.task()
def fail():
    raise ValueError(&quot;this is from fail!&quot;)

if __name__ == '__main__':
    chain(group(chain(foo.si(), fail.si(), foo.si())), foo.si())()"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">celery</span> <span class="pl-k">import</span> <span class="pl-v">Celery</span>, <span class="pl-s1">chain</span>, <span class="pl-s1">group</span>
<span class="pl-s1">app</span> <span class="pl-c1">=</span> <span class="pl-v">Celery</span>(<span class="pl-s">'tasks'</span>,
          <span class="pl-s1">backend</span><span class="pl-c1">=</span><span class="pl-s">'db+sqlite:///results.sqlite'</span>,
          <span class="pl-s1">broker</span><span class="pl-c1">=</span><span class="pl-s">'amqp://guest:guest@rabbit:5672//'</span>)

<span class="pl-en">@<span class="pl-s1">app</span>.<span class="pl-en">task</span>()</span>
<span class="pl-k">def</span> <span class="pl-en">foo</span>():
    <span class="pl-en">print</span>(<span class="pl-s">"this is from foo!"</span>)

<span class="pl-en">@<span class="pl-s1">app</span>.<span class="pl-en">task</span>()</span>
<span class="pl-k">def</span> <span class="pl-en">fail</span>():
    <span class="pl-k">raise</span> <span class="pl-v">ValueError</span>(<span class="pl-s">"this is from fail!"</span>)

<span class="pl-k">if</span> <span class="pl-s1">__name__</span> <span class="pl-c1">==</span> <span class="pl-s">'__main__'</span>:
    <span class="pl-en">chain</span>(<span class="pl-en">group</span>(<span class="pl-en">chain</span>(<span class="pl-s1">foo</span>.<span class="pl-en">si</span>(), <span class="pl-s1">fail</span>.<span class="pl-en">si</span>(), <span class="pl-s1">foo</span>.<span class="pl-en">si</span>())), <span class="pl-s1">foo</span>.<span class="pl-en">si</span>())()</pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>
<p dir="auto">I'm not sure what the intended behavior of <code class="notranslate">chord_unlock</code> is here but I feel that its current behavior atleast isn't consistent. The test case I've provided above results in what I've included below in <em>Actual Behavior</em> where it gets stuck in an infinite loop. If I instead remove the call to <code class="notranslate">foo</code> that is directly after <code class="notranslate">fail</code>, then the chord_unlock doesn't get stuck in a loop and instead simply stops after the <code class="notranslate">fail</code> task has executed.</p>
<p dir="auto">If I extend <code class="notranslate">fail</code> to, for example, the following</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="@app.task()
def fail():
    if not os.path.exists('foo.txt'):
       raise ValueError(&quot;this is from fail!&quot;)
   print(&quot;file does exist!&quot;)"><pre class="notranslate"><span class="pl-en">@<span class="pl-s1">app</span>.<span class="pl-en">task</span>()</span>
<span class="pl-k">def</span> <span class="pl-en">fail</span>():
    <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s1">os</span>.<span class="pl-s1">path</span>.<span class="pl-en">exists</span>(<span class="pl-s">'foo.txt'</span>):
       <span class="pl-k">raise</span> <span class="pl-v">ValueError</span>(<span class="pl-s">"this is from fail!"</span>)
   <span class="pl-en">print</span>(<span class="pl-s">"file does exist!"</span>)</pre></div>
<p dir="auto">then I can also do the test below to see what happens if I retry the failed task:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="chain(group(chain(foo.si(), fail.si().set(task_id=&quot;123&quot;), foo.si())), foo.si().set(task_id=&quot;456&quot;))()
sleep(10)
open('foo.txt', 'a').close()
fail.apply_async(task_id=&quot;123&quot;)"><pre class="notranslate"><span class="pl-en">chain</span>(<span class="pl-en">group</span>(<span class="pl-en">chain</span>(<span class="pl-s1">foo</span>.<span class="pl-en">si</span>(), <span class="pl-s1">fail</span>.<span class="pl-en">si</span>().<span class="pl-en">set</span>(<span class="pl-s1">task_id</span><span class="pl-c1">=</span><span class="pl-s">"123"</span>), <span class="pl-s1">foo</span>.<span class="pl-en">si</span>())), <span class="pl-s1">foo</span>.<span class="pl-en">si</span>().<span class="pl-en">set</span>(<span class="pl-s1">task_id</span><span class="pl-c1">=</span><span class="pl-s">"456"</span>))()
<span class="pl-en">sleep</span>(<span class="pl-c1">10</span>)
<span class="pl-en">open</span>(<span class="pl-s">'foo.txt'</span>, <span class="pl-s">'a'</span>).<span class="pl-en">close</span>()
<span class="pl-s1">fail</span>.<span class="pl-en">apply_async</span>(<span class="pl-s1">task_id</span><span class="pl-c1">=</span><span class="pl-s">"123"</span>)</pre></div>
<p dir="auto">Here I would atleast expect the infinite loop to end, but instead I see that the failed task now succeeds successfully (which is great) but the loop continues. I'm not sure if I should also expect the tasks after the retried task to now also run (they currently do not).</p>
<h1 dir="auto">Actual Behavior</h1>

<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2020-04-08 10:06:34,804: INFO/MainProcess] Connected to amqp://guest:**@rabbit:5672//
[2020-04-08 10:06:34,816: INFO/MainProcess] mingle: searching for neighbors
[2020-04-08 10:06:35,861: INFO/MainProcess] mingle: all alone
[2020-04-08 10:06:35,876: INFO/MainProcess] celery@b7825f7f6305 ready.
[2020-04-08 10:06:56,992: INFO/MainProcess] Received task: celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b]  ETA:[2020-04-08 10:06:57.952640+00:00]
[2020-04-08 10:06:57,004: INFO/MainProcess] Received task: tasks.foo[a2d3f985-68c1-40d8-9a7e-e3d323824901]
[2020-04-08 10:06:57,005: WARNING/ForkPoolWorker-2] this is from foo!
[2020-04-08 10:06:57,031: INFO/MainProcess] Received task: tasks.fail[5a61ee80-e695-4cf7-b182-68de7d848bc7]
[2020-04-08 10:06:57,057: INFO/ForkPoolWorker-2] Task tasks.foo[a2d3f985-68c1-40d8-9a7e-e3d323824901] succeeded in 0.05272179003804922s: None
[2020-04-08 10:06:57,081: ERROR/ForkPoolWorker-3] Task tasks.fail[5a61ee80-e695-4cf7-b182-68de7d848bc7] raised unexpected: ValueError('this is from fail!')
Traceback (most recent call last):
  File &quot;/home/developer/celery/celery/app/trace.py&quot;, line 411, in trace_task
    R = retval = fun(*args, **kwargs)
  File &quot;/home/developer/celery/celery/app/trace.py&quot;, line 680, in __protected_call__
    return self.run(*args, **kwargs)
  File &quot;/home/developer/celery/tasks.py&quot;, line 13, in fail
    raise ValueError(&quot;this is from fail!&quot;)
ValueError: this is from fail!
[2020-04-08 10:06:59,908: INFO/MainProcess] Received task: celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b]  ETA:[2020-04-08 10:07:00.906229+00:00]
[2020-04-08 10:06:59,908: INFO/ForkPoolWorker-2] Task celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b] retry: Retry in 1s
[2020-04-08 10:07:01,914: INFO/ForkPoolWorker-2] Task celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b] retry: Retry in 1s
[2020-04-08 10:07:01,920: INFO/MainProcess] Received task: celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b]  ETA:[2020-04-08 10:07:02.912826+00:00]
[2020-04-08 10:07:03,914: INFO/ForkPoolWorker-2] Task celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b] retry: Retry in 1s
[2020-04-08 10:07:03,915: INFO/MainProcess] Received task: celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b]  ETA:[2020-04-08 10:07:04.913285+00:00]
[2020-04-08 10:07:05,876: INFO/ForkPoolWorker-2] Task celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b] retry: Retry in 1s
[2020-04-08 10:07:05,877: INFO/MainProcess] Received task: celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b]  ETA:[2020-04-08 10:07:06.875506+00:00]"><pre class="notranslate"><code class="notranslate">[2020-04-08 10:06:34,804: INFO/MainProcess] Connected to amqp://guest:**@rabbit:5672//
[2020-04-08 10:06:34,816: INFO/MainProcess] mingle: searching for neighbors
[2020-04-08 10:06:35,861: INFO/MainProcess] mingle: all alone
[2020-04-08 10:06:35,876: INFO/MainProcess] celery@b7825f7f6305 ready.
[2020-04-08 10:06:56,992: INFO/MainProcess] Received task: celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b]  ETA:[2020-04-08 10:06:57.952640+00:00]
[2020-04-08 10:06:57,004: INFO/MainProcess] Received task: tasks.foo[a2d3f985-68c1-40d8-9a7e-e3d323824901]
[2020-04-08 10:06:57,005: WARNING/ForkPoolWorker-2] this is from foo!
[2020-04-08 10:06:57,031: INFO/MainProcess] Received task: tasks.fail[5a61ee80-e695-4cf7-b182-68de7d848bc7]
[2020-04-08 10:06:57,057: INFO/ForkPoolWorker-2] Task tasks.foo[a2d3f985-68c1-40d8-9a7e-e3d323824901] succeeded in 0.05272179003804922s: None
[2020-04-08 10:06:57,081: ERROR/ForkPoolWorker-3] Task tasks.fail[5a61ee80-e695-4cf7-b182-68de7d848bc7] raised unexpected: ValueError('this is from fail!')
Traceback (most recent call last):
  File "/home/developer/celery/celery/app/trace.py", line 411, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/developer/celery/celery/app/trace.py", line 680, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/developer/celery/tasks.py", line 13, in fail
    raise ValueError("this is from fail!")
ValueError: this is from fail!
[2020-04-08 10:06:59,908: INFO/MainProcess] Received task: celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b]  ETA:[2020-04-08 10:07:00.906229+00:00]
[2020-04-08 10:06:59,908: INFO/ForkPoolWorker-2] Task celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b] retry: Retry in 1s
[2020-04-08 10:07:01,914: INFO/ForkPoolWorker-2] Task celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b] retry: Retry in 1s
[2020-04-08 10:07:01,920: INFO/MainProcess] Received task: celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b]  ETA:[2020-04-08 10:07:02.912826+00:00]
[2020-04-08 10:07:03,914: INFO/ForkPoolWorker-2] Task celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b] retry: Retry in 1s
[2020-04-08 10:07:03,915: INFO/MainProcess] Received task: celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b]  ETA:[2020-04-08 10:07:04.913285+00:00]
[2020-04-08 10:07:05,876: INFO/ForkPoolWorker-2] Task celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b] retry: Retry in 1s
[2020-04-08 10:07:05,877: INFO/MainProcess] Received task: celery.chord_unlock[1cceb33c-3e4f-401d-bc29-cc758b98222b]  ETA:[2020-04-08 10:07:06.875506+00:00]
</code></pre></div>
<p dir="auto">Have an issue using celery with SQS: when the connection is being lost and recovered, messages are not being served by celery.<br>
Only after restarting celery (using supervisor), messages are being processed.</p>
<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the<br>
<a href="https://github.com/celery/celery/issues/3932/" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/3932/hovercard">celery issues 3932</a><br>
<a href="https://github.com/celery/kombu/issues/931" data-hovercard-type="issue" data-hovercard-url="/celery/kombu/issues/931/hovercard">kombu issue 931</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="216299501" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/3932" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/3932/hovercard" href="https://github.com/celery/celery/issues/3932">#3932</a></li>
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="370842185" data-permission-text="Title is private" data-url="https://github.com/celery/kombu/issues/931" data-hovercard-type="issue" data-hovercard-url="/celery/kombu/issues/931/hovercard" href="https://github.com/celery/kombu/issues/931">celery/kombu#931</a></li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>:</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="software -&gt; celery:4.4.2 (cliffs) kombu:4.6.11 py:3.7.6
            billiard:3.6.3.0 py-amqp:2.6.0
platform -&gt; system:Linux arch:64bit
            kernel version:4.15.0-1065-aws imp:CPython
loader   -&gt; celery.loaders.default.Loader
settings -&gt; transport:amqp results:disabled"><pre class="notranslate"><code class="notranslate">software -&gt; celery:4.4.2 (cliffs) kombu:4.6.11 py:3.7.6
            billiard:3.6.3.0 py-amqp:2.6.0
platform -&gt; system:Linux arch:64bit
            kernel version:4.15.0-1065-aws imp:CPython
loader   -&gt; celery.loaders.default.Loader
settings -&gt; transport:amqp results:disabled
</code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: 3.7.6 imp: CPython</li>
<li><strong>Minimal Celery Version</strong>: 4.4.2</li>
<li><strong>Minimal Kombu Version</strong>: 4.6.11</li>
<li><strong>Minimal Broker Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Version</strong>: N/A or Unknown</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: GNU Linux Ubuntu x86_64 18.04</li>
<li><strong>Minimal Broker Client Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="aiohttp==3.6.2
amqp==2.6.0
argh==0.26.2
asgiref==3.2.10
async-timeout==3.0.1
attrs==19.3.0
billiard==3.6.3.0
boto3==1.12.39
botocore==1.15.49
celery==4.4.2
certifi==2020.6.20
chardet==3.0.4
ddtrace==0.37.0
Django==3.0.4
docutils==0.15.2
future==0.18.2
gunicorn==20.0.4
idna==2.10
jmespath==0.10.0
kombu==4.6.11
msgpack==1.0.0
multidict==4.7.6
pathtools==0.1.2
psycopg2-binary==2.8.4
python-dateutil==2.8.1
python-dotenv==0.12.0
pytz==2020.1
PyYAML==5.3.1
requests==2.23.0
s3transfer==0.3.3
setproctitle==1.1.10
six==1.15.0
slackclient==2.5.0
sqlparse==0.3.1
urllib3==1.25.9
vine==1.3.0
yarl==1.4.2"><pre class="notranslate"><code class="notranslate">aiohttp==3.6.2
amqp==2.6.0
argh==0.26.2
asgiref==3.2.10
async-timeout==3.0.1
attrs==19.3.0
billiard==3.6.3.0
boto3==1.12.39
botocore==1.15.49
celery==4.4.2
certifi==2020.6.20
chardet==3.0.4
ddtrace==0.37.0
Django==3.0.4
docutils==0.15.2
future==0.18.2
gunicorn==20.0.4
idna==2.10
jmespath==0.10.0
kombu==4.6.11
msgpack==1.0.0
multidict==4.7.6
pathtools==0.1.2
psycopg2-binary==2.8.4
python-dateutil==2.8.1
python-dotenv==0.12.0
pytz==2020.1
PyYAML==5.3.1
requests==2.23.0
s3transfer==0.3.3
setproctitle==1.1.10
six==1.15.0
slackclient==2.5.0
sqlparse==0.3.1
urllib3==1.25.9
vine==1.3.0
yarl==1.4.2
</code></pre></div>
<p dir="auto"></p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
N/A
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>

<details>
<p dir="auto">
It depends. The issue is occurring when we will get a connection lost error.
```python
```
</p>
</details>
<h1 dir="auto">Expected Behavior</h1>

<h1 dir="auto">Actual Behavior</h1>
<p dir="auto">Got this in logs.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2020-06-03 01:36:35,499: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connection.py&quot;, line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/util/connection.py&quot;, line 61, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File &quot;/opt/pyenv/versions/3.7.6/lib/python3.7/socket.py&quot;, line 752, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/httpsession.py&quot;, line 263, in send
    chunked=self._chunked(request.headers),
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connectionpool.py&quot;, line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/util/retry.py&quot;, line 379, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/packages/six.py&quot;, line 735, in reraise
    raise value
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connectionpool.py&quot;, line 677, in urlopen
    chunked=chunked,
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connectionpool.py&quot;, line 381, in _make_request
    self._validate_conn(conn)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connectionpool.py&quot;, line 976, in _validate_conn
    conn.connect()
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connection.py&quot;, line 308, in connect
    conn = self._new_conn()
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connection.py&quot;, line 172, in _new_conn
    self, &quot;Failed to establish a new connection: %s&quot; % e
urllib3.exceptions.NewConnectionError: &lt;botocore.awsrequest.AWSHTTPSConnection object at 0x7f910d7373d0&gt;: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/celery/worker/consumer/consumer.py&quot;, line 318, in start
    blueprint.start(self)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/celery/bootsteps.py&quot;, line 119, in start
    step.start(parent)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/celery/worker/consumer/consumer.py&quot;, line 599, in start
    c.loop(*c.loop_args())
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/celery/worker/loops.py&quot;, line 83, in asynloop
    next(loop)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/asynchronous/hub.py&quot;, line 301, in create_loop
    item()
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/vine/promises.py&quot;, line 170, in __call__
    return self.throw()
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/vine/promises.py&quot;, line 167, in __call__
    retval = fun(*final_args, **final_kwargs)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/transport/SQS.py&quot;, line 390, in _schedule_queue
    queue, callback=promise(self._loop1, (queue,)),
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/transport/SQS.py&quot;, line 406, in _get_bulk_async
    return self._get_async(queue, maxcount, callback=callback)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/transport/SQS.py&quot;, line 417, in _get_async
    callback=transform(self._on_messages_ready, callback, q, queue),
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/transport/SQS.py&quot;, line 442, in _get_from_sqs
    queue_url = connection.get_queue_url(queue)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/asynchronous/aws/sqs/connection.py&quot;, line 45, in get_queue_url
    res = self.sqs_connection.get_queue_url(QueueName=queue)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/client.py&quot;, line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/ddtrace/contrib/botocore/patch.py&quot;, line 70, in patched_api_call
    result = original_func(*args, **kwargs)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/client.py&quot;, line 613, in _make_api_call
    operation_model, request_dict, request_context)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/client.py&quot;, line 632, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/endpoint.py&quot;, line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/endpoint.py&quot;, line 137, in _send_request
    success_response, exception):
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/endpoint.py&quot;, line 231, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/hooks.py&quot;, line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/hooks.py&quot;, line 228, in emit
    return self._emit(event_name, kwargs)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/hooks.py&quot;, line 211, in _emit
    response = handler(**kwargs)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py&quot;, line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py&quot;, line 251, in __call__
    caught_exception)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py&quot;, line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py&quot;, line 317, in __call__
    caught_exception)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py&quot;, line 223, in __call__
    attempt_number, caught_exception)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py&quot;, line 359, in _check_caught_exception
    raise caught_exception
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/endpoint.py&quot;, line 200, in _do_get_response
    http_response = self._send(request)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/endpoint.py&quot;, line 244, in _send
    return self.http_session.send(request)
  File &quot;/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/httpsession.py&quot;, line 283, in send
    raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: &quot;https://queue.amazonaws.com/&quot;
[2020-06-03 01:36:47,080: ERROR/MainProcess] consumer: Cannot connect to sqs://localhost//: Could not connect to the endpoint URL: &quot;https://queue.amazonaws.com/&quot;.
Trying again in 2.00 seconds... (1/100)

[2020-06-03 01:36:57,674: INFO/MainProcess] Connected to sqs://localhost//
"><pre class="notranslate"><code class="notranslate">[2020-06-03 01:36:35,499: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/util/connection.py", line 61, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/opt/pyenv/versions/3.7.6/lib/python3.7/socket.py", line 752, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/httpsession.py", line 263, in send
    chunked=self._chunked(request.headers),
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/util/retry.py", line 379, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 976, in _validate_conn
    conn.connect()
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connection.py", line 308, in connect
    conn = self._new_conn()
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/urllib3/connection.py", line 172, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: &lt;botocore.awsrequest.AWSHTTPSConnection object at 0x7f910d7373d0&gt;: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 599, in start
    c.loop(*c.loop_args())
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/celery/worker/loops.py", line 83, in asynloop
    next(loop)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 301, in create_loop
    item()
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/vine/promises.py", line 170, in __call__
    return self.throw()
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/vine/promises.py", line 167, in __call__
    retval = fun(*final_args, **final_kwargs)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/transport/SQS.py", line 390, in _schedule_queue
    queue, callback=promise(self._loop1, (queue,)),
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/transport/SQS.py", line 406, in _get_bulk_async
    return self._get_async(queue, maxcount, callback=callback)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/transport/SQS.py", line 417, in _get_async
    callback=transform(self._on_messages_ready, callback, q, queue),
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/transport/SQS.py", line 442, in _get_from_sqs
    queue_url = connection.get_queue_url(queue)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/kombu/asynchronous/aws/sqs/connection.py", line 45, in get_queue_url
    res = self.sqs_connection.get_queue_url(QueueName=queue)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/ddtrace/contrib/botocore/patch.py", line 70, in patched_api_call
    result = original_func(*args, **kwargs)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/client.py", line 613, in _make_api_call
    operation_model, request_dict, request_context)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/client.py", line 632, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/endpoint.py", line 137, in _send_request
    success_response, exception):
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/endpoint.py", line 231, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py", line 251, in __call__
    caught_exception)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py", line 317, in __call__
    caught_exception)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/endpoint.py", line 200, in _do_get_response
    http_response = self._send(request)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/endpoint.py", line 244, in _send
    return self.http_session.send(request)
  File "/home/$USER/.local/share/virtualenvs/$PROJECT_NAME-zlr2NI-5/lib/python3.7/site-packages/botocore/httpsession.py", line 283, in send
    raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://queue.amazonaws.com/"
[2020-06-03 01:36:47,080: ERROR/MainProcess] consumer: Cannot connect to sqs://localhost//: Could not connect to the endpoint URL: "https://queue.amazonaws.com/".
Trying again in 2.00 seconds... (1/100)

[2020-06-03 01:36:57,674: INFO/MainProcess] Connected to sqs://localhost//

</code></pre></div>
<p dir="auto">And after the last line logs are empty because celery stops handling any message.</p>