<p dir="auto">I got weird problem with Celery. Shortly after start each pool worker dies with one of these two exception traces:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2012-09-03 16:11:42,326: WARNING/MainProcess] Process PoolWorker-6:
[2012-09-03 16:11:42,326: WARNING/MainProcess] Traceback (most recent call last):
[2012-09-03 16:11:42,327: WARNING/MainProcess] File &quot;/mnt/nimble/eggs/billiard-2.7.3.12-py2.7-linux-x86_64.egg/billiard/process.py&quot;, line 273, in _bootstrap
[2012-09-03 16:11:42,327: WARNING/MainProcess] self.run()
[2012-09-03 16:11:42,327: WARNING/MainProcess] File &quot;/mnt/nimble/eggs/billiard-2.7.3.12-py2.7-linux-x86_64.egg/billiard/process.py&quot;, line 122, in run
[2012-09-03 16:11:42,327: WARNING/MainProcess] self._target(*self._args, **self._kwargs)
[2012-09-03 16:11:42,327: WARNING/MainProcess] File &quot;/mnt/nimble/eggs/billiard-2.7.3.12-py2.7-linux-x86_64.egg/billiard/pool.py&quot;, line 302, in worker
[2012-09-03 16:11:42,328: WARNING/MainProcess] put((ACK, (job, i, time.time(), pid)))
[2012-09-03 16:11:42,328: WARNING/MainProcess] File &quot;/mnt/nimble/eggs/billiard-2.7.3.12-py2.7-linux-x86_64.egg/billiard/queues.py&quot;, line 377, in put
[2012-09-03 16:11:42,329: WARNING/MainProcess] return send(obj)
[2012-09-03 16:11:42,329: WARNING/MainProcess] IOError: [Errno 32] Broken pipe"><pre class="notranslate"><code class="notranslate">[2012-09-03 16:11:42,326: WARNING/MainProcess] Process PoolWorker-6:
[2012-09-03 16:11:42,326: WARNING/MainProcess] Traceback (most recent call last):
[2012-09-03 16:11:42,327: WARNING/MainProcess] File "/mnt/nimble/eggs/billiard-2.7.3.12-py2.7-linux-x86_64.egg/billiard/process.py", line 273, in _bootstrap
[2012-09-03 16:11:42,327: WARNING/MainProcess] self.run()
[2012-09-03 16:11:42,327: WARNING/MainProcess] File "/mnt/nimble/eggs/billiard-2.7.3.12-py2.7-linux-x86_64.egg/billiard/process.py", line 122, in run
[2012-09-03 16:11:42,327: WARNING/MainProcess] self._target(*self._args, **self._kwargs)
[2012-09-03 16:11:42,327: WARNING/MainProcess] File "/mnt/nimble/eggs/billiard-2.7.3.12-py2.7-linux-x86_64.egg/billiard/pool.py", line 302, in worker
[2012-09-03 16:11:42,328: WARNING/MainProcess] put((ACK, (job, i, time.time(), pid)))
[2012-09-03 16:11:42,328: WARNING/MainProcess] File "/mnt/nimble/eggs/billiard-2.7.3.12-py2.7-linux-x86_64.egg/billiard/queues.py", line 377, in put
[2012-09-03 16:11:42,329: WARNING/MainProcess] return send(obj)
[2012-09-03 16:11:42,329: WARNING/MainProcess] IOError: [Errno 32] Broken pipe
</code></pre></div>
<p dir="auto">or</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2012-09-03 15:21:45,071: WARNING/MainProcess] Process PoolWorker-11:
[2012-09-03 15:21:45,072: WARNING/MainProcess] Traceback (most recent call last):
[2012-09-03 15:21:45,072: WARNING/MainProcess] File &quot;/mnt/nimble/eggs/billiard-2.7.3.12-py2.7-linux-x86_64.egg/billiard/process.py&quot;, line 273, in _bootstrap
[2012-09-03 15:21:45,072: WARNING/MainProcess] self.run()
[2012-09-03 15:21:45,072: WARNING/MainProcess] File &quot;/mnt/nimble/eggs/billiard-2.7.3.12-py2.7-linux-x86_64.egg/billiard/process.py&quot;, line 122, in run
[2012-09-03 15:21:45,072: WARNING/MainProcess] self._target(*self._args, **self._kwargs)
[2012-09-03 15:21:45,073: WARNING/MainProcess] File &quot;/mnt/nimble/eggs/billiard-2.7.3.12-py2.7-linux-x86_64.egg/billiard/pool.py&quot;, line 314, in worker
[2012-09-03 15:21:45,073: WARNING/MainProcess] put((READY, (job, i, (False, einfo))))
[2012-09-03 15:21:45,073: WARNING/MainProcess] File &quot;/mnt/nimble/eggs/billiard-2.7.3.12-py2.7-linux-x86_64.egg/billiard/queues.py&quot;, line 377, in put
[2012-09-03 15:21:45,073: WARNING/MainProcess] return send(obj)
[2012-09-03 15:21:45,073: WARNING/MainProcess] IOError: [Errno 32] Broken pipe"><pre class="notranslate"><code class="notranslate">[2012-09-03 15:21:45,071: WARNING/MainProcess] Process PoolWorker-11:
[2012-09-03 15:21:45,072: WARNING/MainProcess] Traceback (most recent call last):
[2012-09-03 15:21:45,072: WARNING/MainProcess] File "/mnt/nimble/eggs/billiard-2.7.3.12-py2.7-linux-x86_64.egg/billiard/process.py", line 273, in _bootstrap
[2012-09-03 15:21:45,072: WARNING/MainProcess] self.run()
[2012-09-03 15:21:45,072: WARNING/MainProcess] File "/mnt/nimble/eggs/billiard-2.7.3.12-py2.7-linux-x86_64.egg/billiard/process.py", line 122, in run
[2012-09-03 15:21:45,072: WARNING/MainProcess] self._target(*self._args, **self._kwargs)
[2012-09-03 15:21:45,073: WARNING/MainProcess] File "/mnt/nimble/eggs/billiard-2.7.3.12-py2.7-linux-x86_64.egg/billiard/pool.py", line 314, in worker
[2012-09-03 15:21:45,073: WARNING/MainProcess] put((READY, (job, i, (False, einfo))))
[2012-09-03 15:21:45,073: WARNING/MainProcess] File "/mnt/nimble/eggs/billiard-2.7.3.12-py2.7-linux-x86_64.egg/billiard/queues.py", line 377, in put
[2012-09-03 15:21:45,073: WARNING/MainProcess] return send(obj)
[2012-09-03 15:21:45,073: WARNING/MainProcess] IOError: [Errno 32] Broken pipe
</code></pre></div>
<p dir="auto">This problem disappears when launching Celery with --concurrency=1, but that's kind of poor workaround.<br>
Celery report for my machine looks like this:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="software -&gt; celery:3.0.9 (Chiastic Slide) kombu:2.4.5 py:2.7.3
            billiard:2.7.3.12 pymongo:2.1.1
platform -&gt; system:Linux arch:64bit, ELF imp:CPython
loader   -&gt; celery.loaders.default.Loader
settings -&gt; transport:mongodb results:mongodb"><pre class="notranslate"><code class="notranslate">software -&gt; celery:3.0.9 (Chiastic Slide) kombu:2.4.5 py:2.7.3
            billiard:2.7.3.12 pymongo:2.1.1
platform -&gt; system:Linux arch:64bit, ELF imp:CPython
loader   -&gt; celery.loaders.default.Loader
settings -&gt; transport:mongodb results:mongodb
</code></pre></div>
<p dir="auto">I was able to track the problem down to CELERYD_MAX_TASKS_PER_CHILD option in configuration file. After removing it Celery started working as expected. Looks like some multiprocessing issue with main process loosing track of rotating workers.</p>
<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> This has already been asked to the <a href="https://groups.google.com/forum/#!forum/celery-users" rel="nofollow">discussion group</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="109864323" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/2839" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/2839/hovercard" href="https://github.com/celery/celery/issues/2839">#2839</a></li>
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="298982784" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/4550" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/4550/hovercard" href="https://github.com/celery/celery/issues/4550">#4550</a></li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>:</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="software -&gt; celery:4.4.6 (cliffs) kombu:4.6.11 py:3.7.6
            billiard:3.6.3.0 redis:3.5.3
platform -&gt; system:Darwin arch:64bit
            kernel version:19.6.0 imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:redis results:disabled

broker_url: 'redis://h:********@ec2-23-21-96-121.compute-1.amazonaws.com:24849//'
celery_backend_result: 'redis://h:********@ec2-23-21-96-121.compute-1.amazonaws.com:24849/'
task_acks_late: True
task_reject_on_worker_lost: True
task_time_limit: 300
broker_transport_options: {
 'visibility_timeout': 3600}"><pre class="notranslate"><code class="notranslate">software -&gt; celery:4.4.6 (cliffs) kombu:4.6.11 py:3.7.6
            billiard:3.6.3.0 redis:3.5.3
platform -&gt; system:Darwin arch:64bit
            kernel version:19.6.0 imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:redis results:disabled

broker_url: 'redis://h:********@ec2-23-21-96-121.compute-1.amazonaws.com:24849//'
celery_backend_result: 'redis://h:********@ec2-23-21-96-121.compute-1.amazonaws.com:24849/'
task_acks_late: True
task_reject_on_worker_lost: True
task_time_limit: 300
broker_transport_options: {
 'visibility_timeout': 3600}
</code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Celery Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Kombu Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Version</strong>: N/A or Unknown</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Client Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="amqp==2.6.0
appdirs==1.4.4
attrs==19.3.0
billiard==3.6.3.0
black==19.10b0
celery==4.4.6
click==7.1.2
flake8==3.8.3
future==0.18.2
importlib-metadata==1.7.0
kombu==4.6.11
mccabe==0.6.1
mypy==0.782
mypy-extensions==0.4.3
pathspec==0.8.0
pycodestyle==2.6.0
pyflakes==2.2.0
pytz==2020.1
redis==3.5.3
regex==2020.7.14
toml==0.10.1
typed-ast==1.4.1
typing-extensions==3.7.4.2
vine==1.3.0
zipp==3.1.0"><pre class="notranslate"><code class="notranslate">amqp==2.6.0
appdirs==1.4.4
attrs==19.3.0
billiard==3.6.3.0
black==19.10b0
celery==4.4.6
click==7.1.2
flake8==3.8.3
future==0.18.2
importlib-metadata==1.7.0
kombu==4.6.11
mccabe==0.6.1
mypy==0.782
mypy-extensions==0.4.3
pathspec==0.8.0
pycodestyle==2.6.0
pyflakes==2.2.0
pytz==2020.1
redis==3.5.3
regex==2020.7.14
toml==0.10.1
typed-ast==1.4.1
typing-extensions==3.7.4.2
vine==1.3.0
zipp==3.1.0
</code></pre></div>
<p dir="auto"></p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
N/A
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>

<details>
<p dir="auto">
</p><p dir="auto">I have reproduced it here: <a href="https://github.com/sayanarijit/remap-sigterm">https://github.com/sayanarijit/remap-sigterm</a></p>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>

<p dir="auto"><a href="https://github.com/sayanarijit/remap-sigterm#what-should-happen">https://github.com/sayanarijit/remap-sigterm#what-should-happen</a></p>
<ul dir="auto">
<li>The task is queued by redis.</li>
<li>Heroku worker starts executing the task which goes to a 5 min. sleep.</li>
<li>When shutdown, Heroku worker sends SIGQUIT (because REMAP_SIGTERM="SIGQUIT") to the worker.</li>
<li>The worker exists immediately never acknowledging the task (the task stays in queue).</li>
<li>After the worker comes back, it fetches the task from queue and starts running again.</li>
</ul>
<h1 dir="auto">Actual Behavior</h1>

<p dir="auto"><a href="https://github.com/sayanarijit/remap-sigterm#what-happens">https://github.com/sayanarijit/remap-sigterm#what-happens</a></p>
<ul dir="auto">
<li>The task is queued by redis.</li>
<li>Heroku worker starts executing the task which goes to a 5 min. sleep.</li>
<li>When shutdown, Heroku worker sends SIGTERM to the worker.</li>
<li>The worker waits for the process to exit.</li>
<li>After 30 seconds it kills the worker.</li>
<li>After the worker comes back, the queue is empty. Hence task is lost.</li>
</ul>