<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> This has already been asked to the <a href="https://groups.google.com/forum/#!forum/celery-users" rel="nofollow">discussion group</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>: 4.3.0 (rhubarb)</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="software -&gt; celery:4.3.0 (rhubarb) kombu:4.6.11 py:3.6.12
            billiard:3.6.3.0 redis:3.3.11
platform -&gt; system:Linux arch:64bit, ELF
            kernel version:4.14.106-97.85.amzn2.x86_64 imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:redis results:redis://query-runner-redis-master:6379/0

BROKER_URL: 'redis://query-runner-redis-master:6379/0'
CELERY_RESULT_BACKEND: 'redis://query-runner-redis-master:6379/0'
CELERY_INCLUDE: ['tasks.query']
CELERY_TIMEZONE: 'UTC'
CELERY_TASK_SERIALIZER: 'json'
CELERY_IGNORE_RESULT: True
CELERY_TASK_ACKS_LATE: True
CELERYD_PREFETCH_MULTIPLIER: 1
CELERY_ROUTES: {
    'tasks.query.kill_query': {'queue': 'meta'},
    'tasks.query.run_query': {'queue': 'default'}}
CELERY_ALWAYS_EAGER: False
CELERYBEAT_SCHEDULE: {
 }
CELERY_TASK_CREATE_MISSING_QUEUES: True

"><pre class="notranslate"><code class="notranslate">software -&gt; celery:4.3.0 (rhubarb) kombu:4.6.11 py:3.6.12
            billiard:3.6.3.0 redis:3.3.11
platform -&gt; system:Linux arch:64bit, ELF
            kernel version:4.14.106-97.85.amzn2.x86_64 imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:redis results:redis://query-runner-redis-master:6379/0

BROKER_URL: 'redis://query-runner-redis-master:6379/0'
CELERY_RESULT_BACKEND: 'redis://query-runner-redis-master:6379/0'
CELERY_INCLUDE: ['tasks.query']
CELERY_TIMEZONE: 'UTC'
CELERY_TASK_SERIALIZER: 'json'
CELERY_IGNORE_RESULT: True
CELERY_TASK_ACKS_LATE: True
CELERYD_PREFETCH_MULTIPLIER: 1
CELERY_ROUTES: {
    'tasks.query.kill_query': {'queue': 'meta'},
    'tasks.query.run_query': {'queue': 'default'}}
CELERY_ALWAYS_EAGER: False
CELERYBEAT_SCHEDULE: {
 }
CELERY_TASK_CREATE_MISSING_QUEUES: True


</code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Celery Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Kombu Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Version</strong>: N/A or Unknown</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Client Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="amqp==2.6.1
billiard==3.6.3.0
blinker==1.4
boto==2.49.0
boto3==1.16.21
botocore==1.19.21
celery==4.3.0
celery-prometheus-exporter==1.7.0
certifi==2020.11.8
chardet==3.0.4
cheroot==8.4.7
CherryPy==18.6.0
click==7.1.2
datadog==0.39.0
decorator==4.4.2
env-excavator==1.5.0
Flask==1.1.2
Flask-Cors==3.0.9
flask-swagger==0.2.14
flask-swagger-ui==3.36.0
flower==0.9.5
future==0.18.2
gunicorn==20.0.4
humanize==3.1.0
idna==2.10
importlib-metadata==2.0.0
importlib-resources==3.3.0
inflection==0.5.1
influxdb==5.3.1
itsdangerous==1.1.0
jaraco.classes==3.1.0
jaraco.collections==3.0.0
jaraco.functools==3.0.1
jaraco.text==3.2.0
JayDeBeApi==1.1.1
Jinja2==2.11.2
jmespath==0.10.0
JPype1==0.6.3
kombu==4.6.11
MarkupSafe==1.1.1
more-itertools==8.6.0
msgpack==1.0.0
numpy==1.17.0
portend==2.7.0
prometheus-client==0.9.0
prometheus-flask-exporter==0.18.1
psycopg2==2.8.6
pycrypto==2.6.1
PyHive==0.6.3
PyMySQL==0.10.1
python-dateutil==2.8.1
pytz==2020.4
PyYAML==5.3.1
qds-sdk==1.16.1
raven==6.10.0
redis==3.3.11
redlock==1.2.0
requests==2.25.0
s3transfer==0.3.3
six==1.15.0
tempora==4.0.1
thrift==0.13.0
tornado==6.1
urllib3==1.26.2
vine==1.3.0
Werkzeug==1.0.1
zc.lockfile==2.0
zipp==3.4.0"><pre class="notranslate"><code class="notranslate">amqp==2.6.1
billiard==3.6.3.0
blinker==1.4
boto==2.49.0
boto3==1.16.21
botocore==1.19.21
celery==4.3.0
celery-prometheus-exporter==1.7.0
certifi==2020.11.8
chardet==3.0.4
cheroot==8.4.7
CherryPy==18.6.0
click==7.1.2
datadog==0.39.0
decorator==4.4.2
env-excavator==1.5.0
Flask==1.1.2
Flask-Cors==3.0.9
flask-swagger==0.2.14
flask-swagger-ui==3.36.0
flower==0.9.5
future==0.18.2
gunicorn==20.0.4
humanize==3.1.0
idna==2.10
importlib-metadata==2.0.0
importlib-resources==3.3.0
inflection==0.5.1
influxdb==5.3.1
itsdangerous==1.1.0
jaraco.classes==3.1.0
jaraco.collections==3.0.0
jaraco.functools==3.0.1
jaraco.text==3.2.0
JayDeBeApi==1.1.1
Jinja2==2.11.2
jmespath==0.10.0
JPype1==0.6.3
kombu==4.6.11
MarkupSafe==1.1.1
more-itertools==8.6.0
msgpack==1.0.0
numpy==1.17.0
portend==2.7.0
prometheus-client==0.9.0
prometheus-flask-exporter==0.18.1
psycopg2==2.8.6
pycrypto==2.6.1
PyHive==0.6.3
PyMySQL==0.10.1
python-dateutil==2.8.1
pytz==2020.4
PyYAML==5.3.1
qds-sdk==1.16.1
raven==6.10.0
redis==3.3.11
redlock==1.2.0
requests==2.25.0
s3transfer==0.3.3
six==1.15.0
tempora==4.0.1
thrift==0.13.0
tornado==6.1
urllib3==1.26.2
vine==1.3.0
Werkzeug==1.0.1
zc.lockfile==2.0
zipp==3.4.0
</code></pre></div>
<p dir="auto"></p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
N/A
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>

<details>
<p dir="auto">
</p><div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>

<p dir="auto">I was hoping by specifying <code class="notranslate">-Ofair</code> that tasks would not be reserved by workers as stated here <a href="https://docs.celeryproject.org/en/v4.3.0/userguide/optimizing.html?highlight=optimization#prefork-pool-prefetch-settings" rel="nofollow">https://docs.celeryproject.org/en/v4.3.0/userguide/optimizing.html?highlight=optimization#prefork-pool-prefetch-settings</a></p>
<h1 dir="auto">Actual Behavior</h1>

<p dir="auto">Workers have reserved tasks while other workers are free to do work.</p>
<p dir="auto">I am going to paste the state we are seeing in prod when a lot of tasks come in.  Notice how <code class="notranslate">query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj</code> has a reserved task while <code class="notranslate">celery@query-runner-celery-datalake-interval-refresh-784bb858c8-dgmf9</code> is empty and could run that task.  Please note the name of the worker is the name of the queue they are reading from.</p>
<details>
<summary><b><code class="notranslate">celery inspect active and celery inspect reserved</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="
root@query-runner-celery-datalake-687847cbb6-mxrnc:/opt/workdir# celery -A tasks inspect active
2020-11-24 21:03:59,312 [WARNING] datadog.api: No agent or invalid configuration file found
2020-11-24 21:03:59,316 [INFO] tasks.datadog_helper: initialized with
2020-11-24 21:03:59,316 [INFO] tasks.datadog_helper: {'statsd_host': '172.29.156.91', 'statsd_port': 8125, 'statsd_namespace': 'query_runner_celery_tasks'}
-&gt; celery@query-runner-celery-default-6c9bd65fd7-4cdz6: OK
    - empty -
-&gt; celery@query-runner-celery-default-6c9bd65fd7-v8w22: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-68xqn: OK
    - empty -
-&gt; celery@query-runner-celery-ga-55cdb77d4-lsk2z: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-mxrnc: OK
    - empty -
-&gt; celery@query-runner-celery-default-6c9bd65fd7-9f8r2: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-hnrhw: OK
    - empty -
-&gt; celery@query-runner-celery-ga-55cdb77d4-9jp2n: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-4dx2b: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-q82hx: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk: OK
    * {'id': '93a32a04-7a6d-4a32-a2ee-8b88ca4f9049', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_aa42f1c1970bc1ac2778a65c60b2e08b', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3107', 'caller_json': {'queryId': 3107, 'interval': 'HOURLY', 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk', 'time_start': 1606251603.8548772, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
    * {'id': '747a1dd7-209a-4870-a636-c18176587723', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_373fafa1388d047c83cae9d40f249655', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3149 on dash 616', 'caller_json': {'queryId': 3149, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk', 'time_start': 1606251609.6809852, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
    * {'id': '28ec904f-197f-4aef-99e2-0887f6f85ec7', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_829e5787764013fbf5eb0958eec86a8f', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2400 on dash 609', 'caller_json': {'queryId': 2400, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk', 'time_start': 1606251604.9346218, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': '705d4f46-662d-4930-9011-265e1065346d', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_04cd73d13695b2a7d9503148018c2650', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3179 on dash 616', 'caller_json': {'queryId': 3179, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk', 'time_start': 1606240820.285725, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
-&gt; celery@query-runner-celery-default-6c9bd65fd7-4s4lz: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc: OK
    * {'id': '7afd7a99-368e-4995-8e5a-a1802dff7476', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_9fe008b693e6dc12999bf58212430afa', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2403 on dash 609', 'caller_json': {'queryId': 2403, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc', 'time_start': 1606251605.1041203, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
    * {'id': '4d59fc3f-40b9-4d51-ad41-e241518c629f', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_9a6f5ab9f25ab21bddcc128aee8677d6', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2396 on dash 609', 'caller_json': {'queryId': 2396, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc', 'time_start': 1606251604.2693179, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': '05bc6789-2183-47d2-abe0-7093bad200a5', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_a937a6def4c1010c15dd3c5e1b488d2d', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3116 on dash 616', 'caller_json': {'queryId': 3116, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc', 'time_start': 1606251609.2829497, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc: OK
    * {'id': 'd6ddf309-e58a-4f2d-be71-1824e618716e', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_b7253439941188da3776ffdc8bab2b07', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2401 on dash 609', 'caller_json': {'queryId': 2401, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc', 'time_start': 1606251605.0180476, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': '62bb08c5-d6a6-4966-8de1-0d872faba09f', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_b75d10ba197e1bb1ab787217ca5daa96', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2395 on dash 609', 'caller_json': {'queryId': 2395, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc', 'time_start': 1606251604.1448753, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
    * {'id': '82240c16-ec84-4279-9cd4-694a73a145eb', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_6435b6004863fd5a10de1abc8bf294cd', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3188 on dash 624', 'caller_json': {'queryId': 3188, 'interval': 'HOURLY', 'dashboard_id': 624, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc', 'time_start': 1606251609.9815636, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
    * {'id': '9c062ae6-bc8d-44d1-b861-068cd1032c38', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_6912fef8aee81d8aa0a68f1326e5bbf6', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3117 on dash 616', 'caller_json': {'queryId': 3117, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc', 'time_start': 1606251609.2058136, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-94qs4: OK
    * {'id': '87228aca-e02f-48a3-9096-5540c7b4fb22', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_e97a5e28903b3652a4a2c386fc287cc6', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3115 on dash 616', 'caller_json': {'queryId': 3115, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-94qs4', 'time_start': 1606251609.4260392, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-6xldj: OK
    * {'id': '1a8982d5-32cd-4fcf-a134-72a79d7e5da7', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_02758ef185f77a7397ead9e8e90e9743', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2397 on dash 609', 'caller_json': {'queryId': 2397, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-6xldj', 'time_start': 1606251604.4064894, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': 'b2804139-6d51-4ffe-817b-8b9efc7a9cd4', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_1920e1f67f84ac24fb3d2b8e8bb4bd22', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3114 on dash 616', 'caller_json': {'queryId': 3114, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-6xldj', 'time_start': 1606251609.3467886, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
-&gt; celery@query-runner-celery-ga-55cdb77d4-j6994: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-m8wfp: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-dgmf9: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-drdl2: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-lgstj: OK
    * {'id': '741f2751-ca24-4bcd-b78c-1d92e679952d', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_e526e7f0e7a7c28271dc8f08ee19504b', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3021 on dash 595', 'caller_json': {'queryId': 3021, 'interval': 'HOURLY', 'dashboard_id': 595, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-lgstj', 'time_start': 1606251616.69369, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
-&gt; celery@query-runner-celery-ga-55cdb77d4-6jxzz: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh: OK
    * {'id': '650009cc-4d6b-42c4-96b1-9a3112a1e5c8', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_711f6cdfe13c22710887f00f5678baeb', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2399 on dash 609', 'caller_json': {'queryId': 2399, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh', 'time_start': 1606251604.6513581, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
    * {'id': 'd74e3272-45f0-44ee-bcab-7b67de60b407', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_5ca350ca8b4b185ad42f94c6dd3b6fc9', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3189 on dash 624', 'caller_json': {'queryId': 3189, 'interval': 'HOURLY', 'dashboard_id': 624, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh', 'time_start': 1606251611.4904687, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
    * {'id': '6f61b237-fa9e-4b30-8131-8d8ee75306c5', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_9bc92aabee38e8d6d607bbb24785d814', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2137 on dash 557', 'caller_json': {'queryId': 2137, 'interval': 'HOURLY', 'dashboard_id': 557, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh', 'time_start': 1606248030.1126661, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
-&gt; celery@query-runner-celery-datalake-687847cbb6-dvdpl: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-pdxsk: OK
    - empty -
-&gt; celery@query-runner-celery-ga-55cdb77d4-vl2t6: OK
    - empty -
-&gt; celery@query-runner-celery-default-6c9bd65fd7-g55gl: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-7spd4: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj: OK
    * {'id': 'ab45784f-8d85-452c-b690-43ead1a010c6', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_bb8a680bab37f6bc86eb9f5c7435d656', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2816 on dash 609', 'caller_json': {'queryId': 2816, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': 1606251605.576281, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
    * {'id': '4e64157e-9356-4362-88af-318013c0c4de', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_4859064fc4b96df99eec68e94f733157', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3143 on dash 616', 'caller_json': {'queryId': 3143, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': 1606251609.5426545, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
    * {'id': 'ac0c403b-ade6-4377-9a41-6d305268204b', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_b04a82ce9376c9c6618c3a300abe4182', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3109', 'caller_json': {'queryId': 3109, 'interval': 'HOURLY', 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': 1606240815.8285584, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': 'e9c368de-d47f-4a84-b05d-81cbda00a394', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_4d03e965dbd57a05d6071c0b5cbf699d', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2398 on dash 609', 'caller_json': {'queryId': 2398, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': 1606251604.517987, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
-&gt; celery@query-runner-celery-default-6c9bd65fd7-8zdvm: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-2zxhc: OK
    * {'id': '03e786df-44b1-412a-8354-3acbc762528f', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_0687aa16794011f4d1a0ee1023173bf8', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3105 on dash 557', 'caller_json': {'queryId': 3105, 'interval': 'HOURLY', 'dashboard_id': 557, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-2zxhc', 'time_start': 1606251618.9102547, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': '646617ce-9835-4b60-b5fd-c4a4a5631749', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_ed09cf8c1166115a4ed1a5aa33b032c7', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3144 on dash 616', 'caller_json': {'queryId': 3144, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-2zxhc', 'time_start': 1606251609.6082683, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
root@query-runner-celery-datalake-687847cbb6-mxrnc:/opt/workdir# celery -A tasks inspect reserved
2020-11-24 21:04:03,851 [WARNING] datadog.api: No agent or invalid configuration file found
2020-11-24 21:04:03,854 [INFO] tasks.datadog_helper: initialized with
2020-11-24 21:04:03,855 [INFO] tasks.datadog_helper: {'statsd_host': '172.29.156.91', 'statsd_port': 8125, 'statsd_namespace': 'query_runner_celery_tasks'}
-&gt; celery@query-runner-celery-default-6c9bd65fd7-4cdz6: OK
    - empty -
-&gt; celery@query-runner-celery-default-6c9bd65fd7-v8w22: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-mxrnc: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-68xqn: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-q82hx: OK
    - empty -
-&gt; celery@query-runner-celery-ga-55cdb77d4-lsk2z: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-4dx2b: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-hnrhw: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-lgstj: OK
    - empty -
-&gt; celery@query-runner-celery-default-6c9bd65fd7-9f8r2: OK
    - empty -
-&gt; celery@query-runner-celery-ga-55cdb77d4-j6994: OK
    - empty -
-&gt; celery@query-runner-celery-ga-55cdb77d4-9jp2n: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk: OK
    - empty -
-&gt; celery@query-runner-celery-default-6c9bd65fd7-4s4lz: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-94qs4: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-m8wfp: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-6xldj: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-dgmf9: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh: OK
    - empty -
-&gt; celery@query-runner-celery-ga-55cdb77d4-6jxzz: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-dvdpl: OK
    - empty -
-&gt; celery@query-runner-celery-default-6c9bd65fd7-g55gl: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-drdl2: OK
    - empty -
-&gt; celery@query-runner-celery-ga-55cdb77d4-vl2t6: OK
    - empty -
-&gt; celery@query-runner-celery-default-6c9bd65fd7-8zdvm: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj: OK
    * {'id': '3f501bbe-6c56-48e2-ba64-9b1adac30695', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': &quot;{'query_id': 'query_6eb04958c42a71ebd014f9be9e116f73', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2823 on dash 557', 'caller_json': {'queryId': 2823, 'interval': 'HOURLY', 'dashboard_id': 557, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}&quot;, 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': None, 'acknowledged': False, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': None}
-&gt; celery@query-runner-celery-datalake-687847cbb6-7spd4: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-2zxhc: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-pdxsk: OK
    - empty -
"><pre class="notranslate"><code class="notranslate">
root@query-runner-celery-datalake-687847cbb6-mxrnc:/opt/workdir# celery -A tasks inspect active
2020-11-24 21:03:59,312 [WARNING] datadog.api: No agent or invalid configuration file found
2020-11-24 21:03:59,316 [INFO] tasks.datadog_helper: initialized with
2020-11-24 21:03:59,316 [INFO] tasks.datadog_helper: {'statsd_host': '172.29.156.91', 'statsd_port': 8125, 'statsd_namespace': 'query_runner_celery_tasks'}
-&gt; celery@query-runner-celery-default-6c9bd65fd7-4cdz6: OK
    - empty -
-&gt; celery@query-runner-celery-default-6c9bd65fd7-v8w22: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-68xqn: OK
    - empty -
-&gt; celery@query-runner-celery-ga-55cdb77d4-lsk2z: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-mxrnc: OK
    - empty -
-&gt; celery@query-runner-celery-default-6c9bd65fd7-9f8r2: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-hnrhw: OK
    - empty -
-&gt; celery@query-runner-celery-ga-55cdb77d4-9jp2n: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-4dx2b: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-q82hx: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk: OK
    * {'id': '93a32a04-7a6d-4a32-a2ee-8b88ca4f9049', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_aa42f1c1970bc1ac2778a65c60b2e08b', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3107', 'caller_json': {'queryId': 3107, 'interval': 'HOURLY', 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk', 'time_start': 1606251603.8548772, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
    * {'id': '747a1dd7-209a-4870-a636-c18176587723', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_373fafa1388d047c83cae9d40f249655', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3149 on dash 616', 'caller_json': {'queryId': 3149, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk', 'time_start': 1606251609.6809852, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
    * {'id': '28ec904f-197f-4aef-99e2-0887f6f85ec7', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_829e5787764013fbf5eb0958eec86a8f', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2400 on dash 609', 'caller_json': {'queryId': 2400, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk', 'time_start': 1606251604.9346218, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': '705d4f46-662d-4930-9011-265e1065346d', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_04cd73d13695b2a7d9503148018c2650', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3179 on dash 616', 'caller_json': {'queryId': 3179, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk', 'time_start': 1606240820.285725, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
-&gt; celery@query-runner-celery-default-6c9bd65fd7-4s4lz: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc: OK
    * {'id': '7afd7a99-368e-4995-8e5a-a1802dff7476', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_9fe008b693e6dc12999bf58212430afa', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2403 on dash 609', 'caller_json': {'queryId': 2403, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc', 'time_start': 1606251605.1041203, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
    * {'id': '4d59fc3f-40b9-4d51-ad41-e241518c629f', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_9a6f5ab9f25ab21bddcc128aee8677d6', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2396 on dash 609', 'caller_json': {'queryId': 2396, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc', 'time_start': 1606251604.2693179, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': '05bc6789-2183-47d2-abe0-7093bad200a5', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_a937a6def4c1010c15dd3c5e1b488d2d', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3116 on dash 616', 'caller_json': {'queryId': 3116, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc', 'time_start': 1606251609.2829497, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc: OK
    * {'id': 'd6ddf309-e58a-4f2d-be71-1824e618716e', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_b7253439941188da3776ffdc8bab2b07', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2401 on dash 609', 'caller_json': {'queryId': 2401, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc', 'time_start': 1606251605.0180476, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': '62bb08c5-d6a6-4966-8de1-0d872faba09f', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_b75d10ba197e1bb1ab787217ca5daa96', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2395 on dash 609', 'caller_json': {'queryId': 2395, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc', 'time_start': 1606251604.1448753, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
    * {'id': '82240c16-ec84-4279-9cd4-694a73a145eb', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_6435b6004863fd5a10de1abc8bf294cd', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3188 on dash 624', 'caller_json': {'queryId': 3188, 'interval': 'HOURLY', 'dashboard_id': 624, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc', 'time_start': 1606251609.9815636, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
    * {'id': '9c062ae6-bc8d-44d1-b861-068cd1032c38', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_6912fef8aee81d8aa0a68f1326e5bbf6', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3117 on dash 616', 'caller_json': {'queryId': 3117, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc', 'time_start': 1606251609.2058136, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-94qs4: OK
    * {'id': '87228aca-e02f-48a3-9096-5540c7b4fb22', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_e97a5e28903b3652a4a2c386fc287cc6', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3115 on dash 616', 'caller_json': {'queryId': 3115, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-94qs4', 'time_start': 1606251609.4260392, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-6xldj: OK
    * {'id': '1a8982d5-32cd-4fcf-a134-72a79d7e5da7', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_02758ef185f77a7397ead9e8e90e9743', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2397 on dash 609', 'caller_json': {'queryId': 2397, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-6xldj', 'time_start': 1606251604.4064894, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': 'b2804139-6d51-4ffe-817b-8b9efc7a9cd4', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_1920e1f67f84ac24fb3d2b8e8bb4bd22', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3114 on dash 616', 'caller_json': {'queryId': 3114, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-6xldj', 'time_start': 1606251609.3467886, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
-&gt; celery@query-runner-celery-ga-55cdb77d4-j6994: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-m8wfp: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-dgmf9: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-drdl2: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-lgstj: OK
    * {'id': '741f2751-ca24-4bcd-b78c-1d92e679952d', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_e526e7f0e7a7c28271dc8f08ee19504b', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3021 on dash 595', 'caller_json': {'queryId': 3021, 'interval': 'HOURLY', 'dashboard_id': 595, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-lgstj', 'time_start': 1606251616.69369, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
-&gt; celery@query-runner-celery-ga-55cdb77d4-6jxzz: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh: OK
    * {'id': '650009cc-4d6b-42c4-96b1-9a3112a1e5c8', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_711f6cdfe13c22710887f00f5678baeb', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2399 on dash 609', 'caller_json': {'queryId': 2399, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh', 'time_start': 1606251604.6513581, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
    * {'id': 'd74e3272-45f0-44ee-bcab-7b67de60b407', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_5ca350ca8b4b185ad42f94c6dd3b6fc9', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3189 on dash 624', 'caller_json': {'queryId': 3189, 'interval': 'HOURLY', 'dashboard_id': 624, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh', 'time_start': 1606251611.4904687, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
    * {'id': '6f61b237-fa9e-4b30-8131-8d8ee75306c5', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_9bc92aabee38e8d6d607bbb24785d814', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2137 on dash 557', 'caller_json': {'queryId': 2137, 'interval': 'HOURLY', 'dashboard_id': 557, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh', 'time_start': 1606248030.1126661, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
-&gt; celery@query-runner-celery-datalake-687847cbb6-dvdpl: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-pdxsk: OK
    - empty -
-&gt; celery@query-runner-celery-ga-55cdb77d4-vl2t6: OK
    - empty -
-&gt; celery@query-runner-celery-default-6c9bd65fd7-g55gl: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-7spd4: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj: OK
    * {'id': 'ab45784f-8d85-452c-b690-43ead1a010c6', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_bb8a680bab37f6bc86eb9f5c7435d656', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2816 on dash 609', 'caller_json': {'queryId': 2816, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': 1606251605.576281, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
    * {'id': '4e64157e-9356-4362-88af-318013c0c4de', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_4859064fc4b96df99eec68e94f733157', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3143 on dash 616', 'caller_json': {'queryId': 3143, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': 1606251609.5426545, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
    * {'id': 'ac0c403b-ade6-4377-9a41-6d305268204b', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_b04a82ce9376c9c6618c3a300abe4182', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3109', 'caller_json': {'queryId': 3109, 'interval': 'HOURLY', 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': 1606240815.8285584, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': 'e9c368de-d47f-4a84-b05d-81cbda00a394', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_4d03e965dbd57a05d6071c0b5cbf699d', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2398 on dash 609', 'caller_json': {'queryId': 2398, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': 1606251604.517987, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
-&gt; celery@query-runner-celery-default-6c9bd65fd7-8zdvm: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-2zxhc: OK
    * {'id': '03e786df-44b1-412a-8354-3acbc762528f', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_0687aa16794011f4d1a0ee1023173bf8', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3105 on dash 557', 'caller_json': {'queryId': 3105, 'interval': 'HOURLY', 'dashboard_id': 557, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-2zxhc', 'time_start': 1606251618.9102547, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
    * {'id': '646617ce-9835-4b60-b5fd-c4a4a5631749', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_ed09cf8c1166115a4ed1a5aa33b032c7', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3144 on dash 616', 'caller_json': {'queryId': 3144, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-2zxhc', 'time_start': 1606251609.6082683, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
root@query-runner-celery-datalake-687847cbb6-mxrnc:/opt/workdir# celery -A tasks inspect reserved
2020-11-24 21:04:03,851 [WARNING] datadog.api: No agent or invalid configuration file found
2020-11-24 21:04:03,854 [INFO] tasks.datadog_helper: initialized with
2020-11-24 21:04:03,855 [INFO] tasks.datadog_helper: {'statsd_host': '172.29.156.91', 'statsd_port': 8125, 'statsd_namespace': 'query_runner_celery_tasks'}
-&gt; celery@query-runner-celery-default-6c9bd65fd7-4cdz6: OK
    - empty -
-&gt; celery@query-runner-celery-default-6c9bd65fd7-v8w22: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-mxrnc: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-68xqn: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-q82hx: OK
    - empty -
-&gt; celery@query-runner-celery-ga-55cdb77d4-lsk2z: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-4dx2b: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-hnrhw: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-lgstj: OK
    - empty -
-&gt; celery@query-runner-celery-default-6c9bd65fd7-9f8r2: OK
    - empty -
-&gt; celery@query-runner-celery-ga-55cdb77d4-j6994: OK
    - empty -
-&gt; celery@query-runner-celery-ga-55cdb77d4-9jp2n: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk: OK
    - empty -
-&gt; celery@query-runner-celery-default-6c9bd65fd7-4s4lz: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-94qs4: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-m8wfp: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-6xldj: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-dgmf9: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh: OK
    - empty -
-&gt; celery@query-runner-celery-ga-55cdb77d4-6jxzz: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-dvdpl: OK
    - empty -
-&gt; celery@query-runner-celery-default-6c9bd65fd7-g55gl: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-drdl2: OK
    - empty -
-&gt; celery@query-runner-celery-ga-55cdb77d4-vl2t6: OK
    - empty -
-&gt; celery@query-runner-celery-default-6c9bd65fd7-8zdvm: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj: OK
    * {'id': '3f501bbe-6c56-48e2-ba64-9b1adac30695', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_6eb04958c42a71ebd014f9be9e116f73', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2823 on dash 557', 'caller_json': {'queryId': 2823, 'interval': 'HOURLY', 'dashboard_id': 557, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': None, 'acknowledged': False, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': None}
-&gt; celery@query-runner-celery-datalake-687847cbb6-7spd4: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-2zxhc: OK
    - empty -
-&gt; celery@query-runner-celery-datalake-687847cbb6-pdxsk: OK
    - empty -

</code></pre></div>
<p dir="auto"></p>
</details>
<p dir="auto">Also, here is the output of <code class="notranslate">celery inspect stats</code> where you can see <code class="notranslate">fair</code> is getting in properly</p>
<details>
<summary><b><code class="notranslate">celery inspect stats for worker query-runner-celery-datalake-interval-refresh-784bb858c8-dgmf9 </code> Output:</b></summary>
<p dir="auto">
```
-&gt; celery@query-runner-celery-datalake-interval-refresh-784bb858c8-dgmf9: OK
    {
        "broker": {
            "alternates": [],
            "connect_timeout": 4,
            "failover_strategy": "round-robin",
            "heartbeat": 120.0,
            "hostname": "query-runner-redis-master",
            "insist": false,
            "login_method": null,
            "port": 6379,
            "ssl": false,
            "transport": "redis",
            "transport_options": {},
            "uri_prefix": null,
            "userid": null,
            "virtual_host": "0"
        },
        "clock": "8600271",
        "pid": 8,
        "pool": {
            "max-concurrency": 4,
            "max-tasks-per-child": "N/A",
            "processes": [
                13,
                14,
                15,
                16
            ],
            "put-guarded-by-semaphore": false,
            "timeouts": [
                0,
                0
            ],
            "writes": {
                "all": "42.86%, 42.86%, 14.29%",
                "avg": "33.33%",
                "inqueues": {
                    "active": 0,
                    "total": 4
                },
                "raw": "3, 3, 1",
                "strategy": "fair",
                "total": 7
            }
        },
        "prefetch_count": 4,
        "rusage": {
            "idrss": 0,
            "inblock": 72400,
            "isrss": 0,
            "ixrss": 0,
            "majflt": 158,
            "maxrss": 68800,
            "minflt": 69557,
            "msgrcv": 0,
            "msgsnd": 0,
            "nivcsw": 2942,
            "nsignals": 0,
            "nswap": 0,
            "nvcsw": 334278,
            "oublock": 1584,
            "stime": 7.775837,
            "utime": 152.889586
        },
        "total": {
            "tasks.query.run_query": 7
        }
    }
```
</p>
</details>
<p dir="auto">Generally, we have a lot of tasks long running tasks.  This issue is making our tasks take double the amount of time because they are unnecessarily waiting behind other tasks without utilizing the other workers.</p>
<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li>[x ] I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li>[ x] This has already been asked to the <a href="https://groups.google.com/forum/#!forum/celery-users" rel="nofollow">discussion group</a> first.</li>
<li>[ x] I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li>[ x] I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li>[x ] I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li>[ x] I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="497429847" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/5748" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/5748/hovercard" href="https://github.com/celery/celery/issues/5748">#5748</a></li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>: 4.4.0</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"><code class="notranslate"></code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<p dir="auto">I first start Celery with Beat option (-B).<br>
It add_periodic_task after config as specified in the doc (<a href="https://docs.celeryproject.org/en/stable/userguide/periodic-tasks.html#starting-the-scheduler" rel="nofollow">https://docs.celeryproject.org/en/stable/userguide/periodic-tasks.html#starting-the-scheduler</a>)<br>
Then, I try to add or remove a task though API Flask app, by either:</p>
<ul dir="auto">
<li>modifying the app.conf.beat_schedule (remove or add task by name</li>
<li>modifying the celerybeat-schedule shelve</li>
</ul>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: 3.7.4</li>
<li><strong>Minimal Celery Version</strong>: 4.4.°</li>
<li><strong>Minimal Kombu Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Version</strong>: RabboitMQ 3.8.0</li>
<li><strong>Minimal Result Backend Version</strong>: N/A or Unknown</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: Redhat 7</li>
<li><strong>Minimal Broker Client Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
amqp==2.5.2
astroid==2.3.3
attrs==19.3.0
autopep8==1.4.4
Babel==2.7.0
billiard==3.6.3.0
celery==4.4.0
certifi==2019.9.11
chardet==3.0.4
Click==7.0
coverage==4.5.4
fixtures-mongoengine==1.3.1
Flask==1.1.1
Flask-Babel==0.12.2
Flask-Session==0.3.1
Flask-WTF==0.14.2
idna==2.8
importlib-metadata==1.5.0
isodate==0.6.0
isort==4.3.21
itsdangerous==1.1.0
Jinja2==2.11.1
jsonschema==3.2.0
kombu==4.6.8
lazy-object-proxy==1.4.3
MarkupSafe==1.1.1
mccabe==0.6.1
mongoengine==0.18.2
paho-mqtt==1.4.0
pycodestyle==2.5.0
pylint==2.4.2
pymongo==3.9.0
pyrsistent==0.15.7
pytz==2019.3
PyYAML==5.3
requests==2.22.0
semver==2.9.0
six==1.12.0
typed-ast==1.4.1
urllib3==1.25.6
vine==1.3.0
Werkzeug==0.16.0
wrapt==1.11.2
WTForms==2.2.1
zipp==2.1.0
</p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
N/A
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>

<details>
<p dir="auto">
</p><div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>

<p dir="auto">I would like to be able to add or remove a task dynmically.</p>
<h1 dir="auto">Actual Behavior</h1>

<p dir="auto">In fact nothing happen, whatever I do, If I remove the task either from shelve (as suggested here: <a href="https://stackoverflow.com/questions/10194975/how-to-dynamically-add-remove-periodic-tasks-to-celery-celerybeat" rel="nofollow">https://stackoverflow.com/questions/10194975/how-to-dynamically-add-remove-periodic-tasks-to-celery-celerybeat</a> proposed by Tristant Brown) or from config app.conf.beat_schedule, the task continue to be schedule at the original cron rules I provided.</p>
<p dir="auto">I first though it was an issue in the sync processed which happend every 3 minutes, and after looking at the code, I discovered that in <code class="notranslate">beat.py</code> file line 421, the function sync does nothing..</p>