<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Enhancement%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical enhancement to an existing feature.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22Issue+Type%3A+Enhancement%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed enhancements.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the if the same enhancement was already implemented in the<br>
master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues in this issue<br>
(If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="797478166" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/6622" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/6622/hovercard" href="https://github.com/celery/celery/issues/6622">#6622</a></li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h1 dir="auto">Brief Summary</h1>
<p dir="auto"><a href="https://github.com/celery/celery/blob/v5.2.7/celery/backends/dynamodb.py#L204">https://github.com/celery/celery/blob/v5.2.7/celery/backends/dynamodb.py#L204</a><br>
This always calls CreatTable even the table existed already.<br>
Each call will have an AWS CloudTrail management event for CreateTable API call with ResourceInUseException error.<br>
Each CouldTrail management event cost 0.00002 USD in AWS US West (Oregon) region.<br>
I have 10M+ tasks everyday and they produce 10M+ such events, and AWS charges me for 200+ USD.</p>
<h1 dir="auto">Design</h1>
<h2 dir="auto">Architectural Considerations</h2>

<p dir="auto">None</p>
<h2 dir="auto">Proposed Behavior</h2>
<p dir="auto">Just like what proposed in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="797478166" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/6622" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/6622/hovercard" href="https://github.com/celery/celery/issues/6622">#6622</a>, check existence of table with another boto3 call first instead of just calling create table</p>
<h2 dir="auto">Proposed UI/UX</h2>

<h2 dir="auto">Diagrams</h2>

<p dir="auto">N/A</p>
<h2 dir="auto">Alternatives</h2>

<p dir="auto">None</p>
<p dir="auto">For some reason after running for about a day our worker loses it's connection to rabbitmq and is unable to reconnect.</p>
<p dir="auto">I initially encountered this issue with 4.2, so I upgraded to the latest version (4.3.0) and am still encountering the issue. I've also found several issues related to this, but since they are all closed I am opening this new one (and referencing them below).</p>
<p dir="auto">The application I am encountering this with is open source, so you can view it in the <a href="https://github.com/tedivm/nebula">Nebula</a> repository.</p>

<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>
<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="282804088" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/4457" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/4457/hovercard" href="https://github.com/celery/celery/issues/4457">#4457</a></li>
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="439877207" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/5499" data-hovercard-type="pull_request" data-hovercard-url="/celery/celery/pull/5499/hovercard" href="https://github.com/celery/celery/pull/5499">#5499</a></li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="282804088" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/4457" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/4457/hovercard" href="https://github.com/celery/celery/issues/4457">#4457</a></li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>: 4.3.0</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"><code class="notranslate"></code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: 3.7</li>
<li><strong>Minimal Celery Version</strong>: 4.2.0</li>
<li><strong>Minimal Kombu Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Version</strong>: N/A or Unknown</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: Docker</li>
<li><strong>Minimal Broker Client Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
</p><p dir="auto"><a href="https://raw.githubusercontent.com/tedivm/nebula/master/requirements.txt" rel="nofollow">https://raw.githubusercontent.com/tedivm/nebula/master/requirements.txt</a></p>
<p dir="auto"></p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
N/A
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>
<p dir="auto">I can't reproduce on demand unfortunately.</p>
<h1 dir="auto">Expected Behavior</h1>
<p dir="auto">The worker should stay connected to the broker, or it should reconnect successfully.</p>
<h1 dir="auto">Actual Behavior</h1>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File &quot;/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py&quot;, line 318, in start
    blueprint.start(self)
  File &quot;/usr/local/lib/python3.6/site-packages/celery/bootsteps.py&quot;, line 119, in start
    step.start(parent)
  File &quot;/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py&quot;, line 596, in start
    c.loop(*c.loop_args())
  File &quot;/usr/local/lib/python3.6/site-packages/celery/worker/loops.py&quot;, line 50, in asynloop
    obj.controller.register_with_event_loop(hub)
  File &quot;/usr/local/lib/python3.6/site-packages/celery/worker/worker.py&quot;, line 219, in register_with_event_loop
    description='hub.register',
  File &quot;/usr/local/lib/python3.6/site-packages/celery/bootsteps.py&quot;, line 151, in send_all
    fun(parent, *args)
  File &quot;/usr/local/lib/python3.6/site-packages/celery/worker/components.py&quot;, line 179, in register_with_event_loop
    w.pool.register_with_event_loop(hub)
  File &quot;/usr/local/lib/python3.6/site-packages/celery/concurrency/prefork.py&quot;, line 134, in register_with_event_loop
    return reg(loop)
  File &quot;/usr/local/lib/python3.6/site-packages/celery/concurrency/asynpool.py&quot;, line 482, in register_with_event_loop
    for fd in self._fileno_to_outq]
  File &quot;/usr/local/lib/python3.6/site-packages/celery/concurrency/asynpool.py&quot;, line 482, in &lt;listcomp&gt;
    for fd in self._fileno_to_outq]
  File &quot;/usr/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py&quot;, line 216, in add_reader
    return self.add(fds, callback, READ | ERR, args)
  File &quot;/usr/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py&quot;, line 167, in add
    self.poller.register(fd, flags)
  File &quot;/usr/local/lib/python3.6/site-packages/kombu/utils/eventio.py&quot;, line 67, in register
    self._epoll.register(fd, events)
OSError: [Errno 9] Bad file descriptor"><pre class="notranslate"><code class="notranslate">consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 596, in start
    c.loop(*c.loop_args())
  File "/usr/local/lib/python3.6/site-packages/celery/worker/loops.py", line 50, in asynloop
    obj.controller.register_with_event_loop(hub)
  File "/usr/local/lib/python3.6/site-packages/celery/worker/worker.py", line 219, in register_with_event_loop
    description='hub.register',
  File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line 151, in send_all
    fun(parent, *args)
  File "/usr/local/lib/python3.6/site-packages/celery/worker/components.py", line 179, in register_with_event_loop
    w.pool.register_with_event_loop(hub)
  File "/usr/local/lib/python3.6/site-packages/celery/concurrency/prefork.py", line 134, in register_with_event_loop
    return reg(loop)
  File "/usr/local/lib/python3.6/site-packages/celery/concurrency/asynpool.py", line 482, in register_with_event_loop
    for fd in self._fileno_to_outq]
  File "/usr/local/lib/python3.6/site-packages/celery/concurrency/asynpool.py", line 482, in &lt;listcomp&gt;
    for fd in self._fileno_to_outq]
  File "/usr/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 216, in add_reader
    return self.add(fds, callback, READ | ERR, args)
  File "/usr/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 167, in add
    self.poller.register(fd, flags)
  File "/usr/local/lib/python3.6/site-packages/kombu/utils/eventio.py", line 67, in register
    self._epoll.register(fd, events)
OSError: [Errno 9] Bad file descriptor
</code></pre></div>