<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> This has already been asked to the <a href="https://groups.google.com/forum/#!forum/celery-users" rel="nofollow">discussion group</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>: 4.1.0</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"><code class="notranslate"></code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: 2.7.5</li>
<li><strong>Minimal Celery Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Kombu Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Version</strong>: N/A or Unknown</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Client Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"><code class="notranslate"></code></pre></div>
<p dir="auto"></p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
N/A
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>

<details>
<p dir="auto">
</p><div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>

<h1 dir="auto">Actual Behavior</h1>
<p dir="auto">When I run Celery worker with this command</p>
<p dir="auto"><code class="notranslate">celery -A sx_workers worker -c1 -l info</code></p>
<p dir="auto">it shows me the next error stack trace:</p>
<p dir="auto"><code class="notranslate">Traceback (most recent call last): File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/celery", line 11, in &lt;module&gt; sys.exit(main()) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/__main__.py", line 14, in main _main() File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/bin/celery.py", line 326, in main cmd.execute_from_commandline(argv) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/bin/celery.py", line 488, in execute_from_commandline super(CeleryCommand, self).execute_from_commandline(argv))) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/bin/base.py", line 281, in execute_from_commandline return self.handle_argv(self.prog_name, argv[1:]) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/bin/celery.py", line 480, in handle_argv return self.execute(command, argv) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/bin/celery.py", line 412, in execute ).run_from_argv(self.prog_name, argv[1:], command=argv[0]) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/bin/worker.py", line 219, in run_from_argv *self.parse_options(prog_name, argv, command)) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/bin/base.py", line 403, in parse_options self.parser = self.create_parser(prog_name, command) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/bin/base.py", line 419, in create_parser self.add_arguments(parser) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/bin/worker.py", line 275, in add_arguments default=conf.worker_state_db, File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 431, in __getitem__ return getitem(k) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 280, in __getitem__ return mapping[_key] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] ... /Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/collections.py", line 130, in __getattr__ return self[k] File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/bin/../lib/python2.7/UserDict.py", line 36, in __getitem__ if key in self.data: File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/kombu/utils/objects.py", line 44, in __get__ value = obj.__dict__[self.__name__] = self.__get(obj) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/app/base.py", line 148, in data return self.callback() File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/app/base.py", line 911, in _finalize_pending_conf conf = self._conf = self._load_config() File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/app/base.py", line 921, in _load_config self.loader.config_from_object(self._config_source) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/loaders/base.py", line 128, in config_from_object obj = self._smart_import(obj, imp=self.import_from_cwd) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/loaders/base.py", line 146, in _smart_import return imp(path) File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/loaders/base.py", line 106, in import_from_cwd package=package, File "/Users/amerzildzic/swipestox/sx_distributed_processing/sxdpenv/lib/python2.7/site-packages/celery/utils/imports.py", line 100, in import_from_cwd with cwd_in_path(): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 84, in helper return GeneratorContextManager(func(*args, **kwds)) RuntimeError: maximum recursion depth exceeded </code></p>
<p dir="auto">I dig through the code a bit and as I see there is recutsion between</p>
<p dir="auto">collections.py : self[k]<br>
and<br>
UserDyct.py: <strong>getitem</strong> (which should be same as indexing??)</p>
<p dir="auto">since both are used on the stack trace.</p>
<p dir="auto">I'm not so familiar with Python nor project, pretty new to this.<br>
Thanks in advance,</p>
<p dir="auto">Amer</p>
<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?utf8=%E2%9C%93&amp;q=is%3Aissue+label%3A%22Category%3A+Documentation%22+">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/main">commit log</a><br>
to find out if the bug was already fixed in the main branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues in this issue<br>
(If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="335069356" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/4843" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/4843/hovercard" href="https://github.com/celery/celery/issues/4843">#4843</a> - inappropriate usage of <code class="notranslate">eta</code> and <code class="notranslate">countdown</code> can lead to increasing worker RAM usage which may be mistakenly recognized as a memory leak</li>
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="326620950" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/4766" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/4766/hovercard" href="https://github.com/celery/celery/issues/4766">#4766</a></li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h1 dir="auto">Description</h1>

<p dir="auto"><code class="notranslate">eta</code> and <code class="notranslate">countdown</code> parameters can be easily overused, especially when using Redis as the broker.</p>
<p dir="auto">When a task with those parameters is sent to the queue, the worker grabs that task immediately but doesn't actually execute it until <code class="notranslate">eta</code>/<code class="notranslate">countdown</code> conditions pass. Until that, the task is stored in memory. Obviously, one such task won't cause a problem, but in our production environment, we ended up in a situation where hundreds of thousands were accumulated in the worker, causing RAM usage to increase by 30GB over a period of a few days. This behavior is documented very deeply in the <a href="https://docs.celeryq.dev/en/stable/internals/worker.html" rel="nofollow">documentation</a>, so I guess not many people have reached that page.</p>
<p dir="auto">The situation can get even worse because of the <code class="notranslate">visibility_timeout</code> setting (1 hour by default), which causes those tasks to be redelivered. If the <code class="notranslate">eta</code>/<code class="notranslate">countdown</code> points to a distant future, tasks will be redelivered every <code class="notranslate">visibility_timeout</code> period.</p>
<p dir="auto">The <a href="https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/redis.html#id1" rel="nofollow">documentation</a> suggests increasing the <code class="notranslate">visibility_timeout</code> to match the longest possible ETA in the application:</p>
<blockquote>
<p dir="auto">So you have to increase the visibility timeout to match the time of the longest ETA you’re planning to use</p>
</blockquote>
<p dir="auto">IMO, that's not a good idea as it can have a negative effect on reliability. On worker's failure, dropped tasks won't get redelivered until <code class="notranslate">visibility_timeout</code> passes. If set to some high value, it may impact user experience (e.g. email is sent after x days when its content is no longer relevant, some data important for the user is processed with x days of delay, etc.).</p>
<h1 dir="auto">Suggestions</h1>
<p dir="auto">I'd suggest strongly discouraging using <code class="notranslate">eta</code>/<code class="notranslate">countdown</code> parameters. In my opinion, they should be only used with very low values (seconds, minutes). Instead, alternatives could be suggested, e.g. database-backed celery beat.</p>
<p dir="auto">In <a href="https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/redis.html#id1" rel="nofollow">Caveats</a> I'd not recommend increasing <code class="notranslate">visibility_timeout</code> as a feasible solution, but if the task needs a longer ETA, I'd point to the alternatives mentioned above.</p>
<p dir="auto">Optionally, it would be nice to add a warning section with all the implications I've mentioned above.</p>
<p dir="auto">Lots of credit to the author of that article: <a href="https://engineering.instawork.com/celery-eta-tasks-demystified-424b836e4e94" rel="nofollow">https://engineering.instawork.com/celery-eta-tasks-demystified-424b836e4e94</a> who explained the problem perfectly</p>