<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>
<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="413720289" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/5358" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/5358/hovercard" href="https://github.com/celery/celery/issues/5358">#5358</a></li>
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="337946694" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/4876" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/4876/hovercard" href="https://github.com/celery/celery/issues/4876">#4876</a></li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>
<p dir="auto"><strong>Celery version</strong>: 4.3.0 (rhubarb)</p>
<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="software -&gt; celery:4.3.0 (rhubarb) kombu:4.5.0 py:2.7.13
            billiard:3.6.0.0 py-amqp:2.4.2
platform -&gt; system:Linux arch:64bit
            kernel version:4.9.0-8-amd64 imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:amqp results:rpc:///

task_queues: 
    (&lt;unbound Queue gpon_tentacle -&gt; &lt;unbound Exchange celery(direct)&gt; -&gt; &gt;,
 &lt;unbound Queue livestatus -&gt; &lt;unbound Exchange celery(direct)&gt; -&gt; &gt;,
 &lt;unbound Queue kraken_host_events -&gt; &lt;unbound Exchange celery(direct)&gt; -&gt; &gt;)
broker_heartbeat: 5
broker_connection_timeout: 10
task_default_exchange_type: 'direct'
broker_pool_limit: 1
task_routes: {
    'gpon_tentacle.*': {   'queue': 'gpon_tentacle'},
    'kraken_host_events.*': {   'queue': 'kraken_host_events'},
    'livestatus.*': {   'queue': 'livestatus'}}
broker_connection_max_retries: 0
result_expires: 3600
broker_url: u'amqp://guest:********@localhost:5672//'
result_backend: u'rpc:///'
task_default_exchange: 'celery'"><pre class="notranslate"><code class="notranslate">software -&gt; celery:4.3.0 (rhubarb) kombu:4.5.0 py:2.7.13
            billiard:3.6.0.0 py-amqp:2.4.2
platform -&gt; system:Linux arch:64bit
            kernel version:4.9.0-8-amd64 imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:amqp results:rpc:///

task_queues: 
    (&lt;unbound Queue gpon_tentacle -&gt; &lt;unbound Exchange celery(direct)&gt; -&gt; &gt;,
 &lt;unbound Queue livestatus -&gt; &lt;unbound Exchange celery(direct)&gt; -&gt; &gt;,
 &lt;unbound Queue kraken_host_events -&gt; &lt;unbound Exchange celery(direct)&gt; -&gt; &gt;)
broker_heartbeat: 5
broker_connection_timeout: 10
task_default_exchange_type: 'direct'
broker_pool_limit: 1
task_routes: {
    'gpon_tentacle.*': {   'queue': 'gpon_tentacle'},
    'kraken_host_events.*': {   'queue': 'kraken_host_events'},
    'livestatus.*': {   'queue': 'livestatus'}}
broker_connection_max_retries: 0
result_expires: 3600
broker_url: u'amqp://guest:********@localhost:5672//'
result_backend: u'rpc:///'
task_default_exchange: 'celery'
</code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<p dir="auto">See this gist: <a href="https://gist.github.com/palvarezcordoba/a88e17e79f3efa8f15c0094afb4a2ec2">https://gist.github.com/palvarezcordoba/a88e17e79f3efa8f15c0094afb4a2ec2</a></p>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Celery Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Kombu Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Version</strong>: N/A or Unknown</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Client Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="amqp==2.4.2
astroid==1.6.5
asyncio==3.4.3
atomicwrites==1.3.0
attrs==19.1.0
backports.functools-lru-cache==1.5
backports.shutil-get-terminal-size==1.0.0
billiard==3.6.0.0
bitstring==3.1.5
caniusepython3==7.0.0
celery==4.3.0
celery-slack==0.3.0
certifi==2019.3.9
chardet==3.0.4
Click==7.0
configparser==3.7.1
decorator==4.3.2
dill==0.2.5
distlib==0.2.8
Django==1.11.20
django-appconf==1.0.3
django-betterforms==1.2
django-filter==1.1.0
django-multiselectfield==0.1.8
django-rq==1.3.1
Django-Select2==5.11.1
django-slack==5.13.0
djangorestframework==3.9.2
enum34==1.1.6
ephem==3.7.6.0
filelock==3.0.10
funcsigs==1.0.2
future==0.17.1
futures==3.2.0
googlemaps==3.0.2
gpon-tentacle==0.0.0
idna==2.8
ipython==5.0.0
ipython-genutils==0.2.0
isort==4.3.4
kombu==4.5.0
kraken==0.0.2
krill-gpon==0.5.0
lazy-object-proxy==1.3.1
libkrill==1.1.6
mccabe==0.6.1
mock==2.0.0
more-itertools==5.0.0
MySQL-python==1.2.5
mysqlclient==1.4.2.post1
netaddr==0.7.19
numpy==1.16.1
packaging==19.0
pathlib2==2.3.3
pbr==5.1.3
pexpect==4.6.0
pickleshare==0.7.5
pika==0.13.0
pluggy==0.9.0
ply==3.11
prompt-toolkit==1.0.15
psutil==5.5.0
ptyprocess==0.6.0
py==1.8.0
pyasn1==0.1.9
pycrypto==2.6.1
pycurl==7.43.0.2
pydot==1.4.1
PyFunctional==1.2.0
Pygments==2.3.1
pylint==1.9.4
pyparsing==2.3.1
pysmi==0.3.3
pysnmp==4.3.2
pytest==4.4.0
pytz==2019.1
PyYAML==5.1
recordclass==0.9
recordtype==1.3
redis==3.2.1
redlock==1.2.0
requests==2.21.0
rq==0.13.0
scandir==1.9.0
Shinken==2.4.3
simplegeneric==0.8.1
singledispatch==3.4.0.3
singleton-decorator==1.0.0
six==1.12.0
tabulate==0.8.3
termcolor==1.1.0
toml==0.10.0
tox==3.8.4
traitlets==4.3.2
typing==3.6.6
urllib3==1.24.1
urlparse2==1.1.1
vine==1.3.0
virtualenv==16.4.3
wcwidth==0.1.7
wrapt==1.11.1
"><pre class="notranslate"><code class="notranslate">amqp==2.4.2
astroid==1.6.5
asyncio==3.4.3
atomicwrites==1.3.0
attrs==19.1.0
backports.functools-lru-cache==1.5
backports.shutil-get-terminal-size==1.0.0
billiard==3.6.0.0
bitstring==3.1.5
caniusepython3==7.0.0
celery==4.3.0
celery-slack==0.3.0
certifi==2019.3.9
chardet==3.0.4
Click==7.0
configparser==3.7.1
decorator==4.3.2
dill==0.2.5
distlib==0.2.8
Django==1.11.20
django-appconf==1.0.3
django-betterforms==1.2
django-filter==1.1.0
django-multiselectfield==0.1.8
django-rq==1.3.1
Django-Select2==5.11.1
django-slack==5.13.0
djangorestframework==3.9.2
enum34==1.1.6
ephem==3.7.6.0
filelock==3.0.10
funcsigs==1.0.2
future==0.17.1
futures==3.2.0
googlemaps==3.0.2
gpon-tentacle==0.0.0
idna==2.8
ipython==5.0.0
ipython-genutils==0.2.0
isort==4.3.4
kombu==4.5.0
kraken==0.0.2
krill-gpon==0.5.0
lazy-object-proxy==1.3.1
libkrill==1.1.6
mccabe==0.6.1
mock==2.0.0
more-itertools==5.0.0
MySQL-python==1.2.5
mysqlclient==1.4.2.post1
netaddr==0.7.19
numpy==1.16.1
packaging==19.0
pathlib2==2.3.3
pbr==5.1.3
pexpect==4.6.0
pickleshare==0.7.5
pika==0.13.0
pluggy==0.9.0
ply==3.11
prompt-toolkit==1.0.15
psutil==5.5.0
ptyprocess==0.6.0
py==1.8.0
pyasn1==0.1.9
pycrypto==2.6.1
pycurl==7.43.0.2
pydot==1.4.1
PyFunctional==1.2.0
Pygments==2.3.1
pylint==1.9.4
pyparsing==2.3.1
pysmi==0.3.3
pysnmp==4.3.2
pytest==4.4.0
pytz==2019.1
PyYAML==5.1
recordclass==0.9
recordtype==1.3
redis==3.2.1
redlock==1.2.0
requests==2.21.0
rq==0.13.0
scandir==1.9.0
Shinken==2.4.3
simplegeneric==0.8.1
singledispatch==3.4.0.3
singleton-decorator==1.0.0
six==1.12.0
tabulate==0.8.3
termcolor==1.1.0
toml==0.10.0
tox==3.8.4
traitlets==4.3.2
typing==3.6.6
urllib3==1.24.1
urlparse2==1.1.1
vine==1.3.0
virtualenv==16.4.3
wcwidth==0.1.7
wrapt==1.11.1

</code></pre></div>
<p dir="auto"></p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
N/A
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>
<p dir="auto"><a href="https://gist.github.com/palvarezcordoba/a88e17e79f3efa8f15c0094afb4a2ec2">https://gist.github.com/palvarezcordoba/a88e17e79f3efa8f15c0094afb4a2ec2</a></p>
<details>
<p dir="auto">
</p><div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>
<p dir="auto">Celery client should send tasks, and after &gt;60 seconds of inactivity, it should be able to send one more task and get the result.</p>
<h1 dir="auto">Actual Behavior</h1>
<p dir="auto">When I send a task by first time, Celery connects to the broker, with a hertbeat conf of 60s.<br>
It declares a random result quue, say <code class="notranslate">QueueX</code>, then consumes it and sends a task, saiyng "eh, reply to <code class="notranslate">QueueX</code>". The worker replies to <code class="notranslate">QueueX</code> and RabbitMQ does a <code class="notranslate">Basic.Deliver</code> as the client is consumer of <code class="notranslate">QueueX</code>.<br>
Celery does not send heartbeats, and after 60 seconds (or configured heartbeat) RabbitMQ closes the connection, and as <code class="notranslate">QueueX</code> has auto-delete active, RabbitMQ deletes it. At this point, if I send other task, - I do not why but... - celery sends the task, the worker receives it, but celery raises <code class="notranslate">error: [Errno 104] Connection reset by peer</code>. If I send another task, celery redeclares <code class="notranslate">QueueX</code>, sends a task which the reply-to being <code class="notranslate">QueueX</code> <em>BUT</em> now celery is not a consumer of <code class="notranslate">QueueX</code> so RabbitMQ does not do a deliver, so celery waits until a timeout.</p>
<p dir="auto">RabbitMQ send heartbeat frames to celery, it ignores (I guess) those packets, but at TCP level they are replied with an ACK, I do not know if it is correct.<br>
A solution (which I do not like) is setting heartbeat at 0.<br>
In an ideal world, celery would send heartbeats, or if broker closes the connection, it would:</p>
<ul dir="auto">
<li>Not send a task whose result can not be received</li>
<li>Detect the network failure</li>
<li>(again) Not send a task whose result can not be received</li>
<li>Reconnect to the broker</li>
<li>Declare <code class="notranslate">QueueX</code> and consume it</li>
<li>Now it can send the task :)</li>
</ul>
<p dir="auto">Currently, the Redis and SQS transports provide a <code class="notranslate">visibility_timeout</code> option in the config.  This has an effect on how apply_async() works with either eta or countdown kwargs.  The caveats regarding this are buried in:</p>
<p dir="auto"><a href="http://docs.celeryproject.org/en/latest/getting-started/brokers/redis.html#redis-caveats" rel="nofollow">http://docs.celeryproject.org/en/latest/getting-started/brokers/redis.html#redis-caveats</a></p>
<p dir="auto"><a href="http://docs.celeryproject.org/en/latest/getting-started/brokers/sqs.html" rel="nofollow">http://docs.celeryproject.org/en/latest/getting-started/brokers/sqs.html</a></p>
<p dir="auto">Using etas or countdowns greater than the visibility_timeout value will result in tasks being duplicated. There should be some mention of this and/or links to these in:</p>
<p dir="auto"><a href="http://docs.celeryproject.org/en/latest/userguide/calling.html#eta-and-countdown" rel="nofollow">http://docs.celeryproject.org/en/latest/userguide/calling.html#eta-and-countdown</a></p>