<h1 dir="auto">Checklist</h1>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> This has already been asked to the <a href="https://groups.google.com/forum/#!forum/celery-users" rel="nofollow">discussion group</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>
<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="417799131" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/5371" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/5371/hovercard" href="https://github.com/celery/celery/issues/5371">#5371</a></li>
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="575623570" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/5988" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/5988/hovercard" href="https://github.com/celery/celery/issues/5988">#5988</a></li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>
<ul dir="auto">
<li>Celery: 5.0.0</li>
<li>Kombu: 5.0.2</li>
<li>Redis: 3.5.3</li>
<li>vine: 5.0.0</li>
<li>billiard: 3.6.3.0</li>
</ul>
<h1 dir="auto">Steps to Reproduce</h1>
<h2 dir="auto">Required Dependencies</h2>
<ul dir="auto">
<li><strong>Minimal Python Version</strong>: Python 3.6.5 :: Anaconda, Inc.</li>
<li><strong>Minimal Celery Version</strong>: 5.0.0</li>
<li><strong>Minimal Kombu Version</strong>: 5.0.2</li>
<li><strong>Minimal Broker Version</strong>: Redis Server 6.0.8</li>
<li><strong>Minimal Result Backend Version</strong>: Memory / Redis Server 6.0.8</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: CentOS 7</li>
<li><strong>Minimal Broker Client Version</strong>: RedisPy 3.5.3</li>
<li><strong>Minimal Result Backend Client Version</strong>: Memory / RedisPy 3.5.3</li>
</ul>
<h3 dir="auto">Other Dependencies</h3>
<p dir="auto">Needs SSL certificates for Redis TLS setup - I used <a href="https://github.com/michaelklishin/tls-gen">https://github.com/michaelklishin/tls-gen</a> to create them</p>
<h2 dir="auto">Minimally Reproducible Test Case</h2>
<p dir="auto">I have the following configs:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="redis_ssl = 
broker_connection_retry = False
broker_transport_options = broker_transport_options
# result_backend_transport_options = broker_transport_options
broker_use_ssl = {
  'ssl_keyfile': '/home/centos/redis/tls-gen/basic/result/client_key.pem',
  'ssl_certfile': '/home/centos/redis/tls-gen/basic/result/client_certificate.pem',
  'ssl_ca_certs': '/home/centos/redis/tls-gen/basic/result/ca_certificate.pem',
  'ssl_cert_reqs': 'required',
}
# redis_backend_use_ssl = redis_ssl

CELERY_BROKER_URL = 'sentinel://127.0.0.1:29876'
CELERY_RESULT_BACKEND = 'cache+memory://'  # Used only for testing - even with Redis here, it doesnt work"><pre class="notranslate"><span class="pl-s1">redis_ssl</span> <span class="pl-c1">=</span> 
<span class="pl-s1">broker_connection_retry</span> <span class="pl-c1">=</span> <span class="pl-c1">False</span>
<span class="pl-s1">broker_transport_options</span> <span class="pl-c1">=</span> <span class="pl-s1">broker_transport_options</span>
<span class="pl-c"># result_backend_transport_options = broker_transport_options</span>
<span class="pl-s1">broker_use_ssl</span> <span class="pl-c1">=</span> {
  <span class="pl-s">'ssl_keyfile'</span>: <span class="pl-s">'/home/centos/redis/tls-gen/basic/result/client_key.pem'</span>,
  <span class="pl-s">'ssl_certfile'</span>: <span class="pl-s">'/home/centos/redis/tls-gen/basic/result/client_certificate.pem'</span>,
  <span class="pl-s">'ssl_ca_certs'</span>: <span class="pl-s">'/home/centos/redis/tls-gen/basic/result/ca_certificate.pem'</span>,
  <span class="pl-s">'ssl_cert_reqs'</span>: <span class="pl-s">'required'</span>,
}
<span class="pl-c"># redis_backend_use_ssl = redis_ssl</span>

<span class="pl-v">CELERY_BROKER_URL</span> <span class="pl-c1">=</span> <span class="pl-s">'sentinel://127.0.0.1:29876'</span>
<span class="pl-v">CELERY_RESULT_BACKEND</span> <span class="pl-c1">=</span> <span class="pl-s">'cache+memory://'</span>  <span class="pl-c"># Used only for testing - even with Redis here, it doesnt work</span></pre></div>
<p dir="auto">With redis.conf:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="bind 0.0.0.0
protected-mode yes
port 0

tls-port 9875
tls-cert-file /home/centos/redis/tls-gen/basic/result/server_certificate.pem
tls-key-file /home/centos/redis/tls-gen/basic/result/server_key.pem
tls-ca-cert-file /home/centos/redis/tls-gen/basic/result/ca_certificate.pem
tls-replication yes

daemonize yes
pidfile /home/centos/redis/redis.pid
logfile /home/centos/redis/redis.log
dir /home/centos/redis"><pre class="notranslate"><code class="notranslate">bind 0.0.0.0
protected-mode yes
port 0

tls-port 9875
tls-cert-file /home/centos/redis/tls-gen/basic/result/server_certificate.pem
tls-key-file /home/centos/redis/tls-gen/basic/result/server_key.pem
tls-ca-cert-file /home/centos/redis/tls-gen/basic/result/ca_certificate.pem
tls-replication yes

daemonize yes
pidfile /home/centos/redis/redis.pid
logfile /home/centos/redis/redis.log
dir /home/centos/redis
</code></pre></div>
<p dir="auto">With redis-sentinel.conf:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="port 29876
tls-port 29875
tls-cert-file &quot;/home/centos/redis/tls-gen/basic/result/server_certificate.pem&quot;
tls-key-file &quot;/home/centos/redis/tls-gen/basic/result/server_key.pem&quot;
tls-ca-cert-file &quot;/home/centos/redis/tls-gen/basic/result/ca_certificate.pem&quot;
tls-replication yes

daemonize yes
pidfile /home/centos/redis/sentinel.pid
logfile /home/centos/redis/sentinel.log
dir /home/centos/redis

sentinel monitor mymaster 127.0.0.1 9875 1
sentinel deny-scripts-reconfig yes
sentinel config-epoch mymaster 6"><pre class="notranslate"><code class="notranslate">port 29876
tls-port 29875
tls-cert-file "/home/centos/redis/tls-gen/basic/result/server_certificate.pem"
tls-key-file "/home/centos/redis/tls-gen/basic/result/server_key.pem"
tls-ca-cert-file "/home/centos/redis/tls-gen/basic/result/ca_certificate.pem"
tls-replication yes

daemonize yes
pidfile /home/centos/redis/sentinel.pid
logfile /home/centos/redis/sentinel.log
dir /home/centos/redis

sentinel monitor mymaster 127.0.0.1 9875 1
sentinel deny-scripts-reconfig yes
sentinel config-epoch mymaster 6
</code></pre></div>
<h1 dir="auto">Expected Behavior</h1>
<p dir="auto">I was expecting Celery to be able to connect to Sentinel, fetch the host and port of the Master.<br>
And then use TLS to connect to the Redis Master.</p>
<h1 dir="auto">Actual Behavior</h1>
<p dir="auto">It gives an error <code class="notranslate">TypeError: __init__() got an unexpected keyword argument 'connection_pool'</code><br>
Traceback:</p>
<details>
<pre class="notranslate"> -------------- celery@ip v5.0.0 (singularity)
--- ***** -----
-- ******* ---- Linux-3.10.0-862.3.2.el7.x86_64-x86_64-with-centos-7.5.1804-Core 2020-11-02 14:00:50
- *** --- * ---
- ** ---------- [config]
- ** ---------- .&gt; app:         corridor_api:0x7fce57b40860
- ** ---------- .&gt; transport:   sentinel://IP:29876//
- ** ---------- .&gt; results:     memory:///
- *** --- * --- .&gt; concurrency: 1 (prefork)
-- ******* ---- .&gt; task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
 -------------- [queues]
                .&gt; apilocal         exchange=apilocal(direct) key=apilocal
                .&gt; celery           exchange=celery(direct) key=celery
<p dir="auto">[2020-11-02 14:00:50,595: CRITICAL/MainProcess] Unrecoverable error: TypeError("<strong>init</strong>() got an unexpected keyword argument 'connection_pool'",)<br>
Traceback (most recent call last):<br>
File "/opt/app/venv/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 918, in create_channel<br>
return self._avail_channels.pop()<br>
IndexError: pop from empty list</p>
<p dir="auto">During handling of the above exception, another exception occurred:</p>
<p dir="auto">Traceback (most recent call last):<br>
File "/opt/app/venv/lib/python3.6/site-packages/redis/connection.py", line 1185, in get_connection<br>
connection = self._available_connections.pop()<br>
IndexError: pop from empty list</p>
<p dir="auto">During handling of the above exception, another exception occurred:</p>
<p dir="auto">Traceback (most recent call last):<br>
File "/opt/app/venv/lib/python3.6/site-packages/celery/worker/worker.py", line 203, in start<br>
self.blueprint.start(self)<br>
File "/opt/app/venv/lib/python3.6/site-packages/celery/bootsteps.py", line 116, in start<br>
step.start(parent)<br>
File "/opt/app/venv/lib/python3.6/site-packages/celery/bootsteps.py", line 365, in start<br>
return self.obj.start()<br>
File "/opt/app/venv/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 311, in start<br>
blueprint.start(self)<br>
File "/opt/app/venv/lib/python3.6/site-packages/celery/bootsteps.py", line 116, in start<br>
step.start(parent)<br>
File "/opt/app/venv/lib/python3.6/site-packages/celery/worker/consumer/connection.py", line 21, in start<br>
c.connection = c.connect()<br>
File "/opt/app/venv/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 398, in connect<br>
conn = self.connection_for_read(heartbeat=self.amqheartbeat)<br>
File "/opt/app/venv/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 405, in connection_for_read<br>
self.app.connection_for_read(heartbeat=heartbeat))<br>
File "/opt/app/venv/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 427, in ensure_connected<br>
conn.connect()<br>
File "/opt/app/venv/lib/python3.6/site-packages/kombu/connection.py", line 277, in connect<br>
max_retries=1, reraise_as_library_errors=False<br>
File "/opt/app/venv/lib/python3.6/site-packages/kombu/connection.py", line 439, in _ensure_connection<br>
callback, timeout=timeout<br>
File "/opt/app/venv/lib/python3.6/site-packages/kombu/utils/functional.py", line 325, in retry_over_time<br>
return fun(*args, **kwargs)<br>
File "/opt/app/venv/lib/python3.6/site-packages/kombu/connection.py", line 866, in _connection_factory<br>
self._connection = self._establish_connection()<br>
File "/opt/app/venv/lib/python3.6/site-packages/kombu/connection.py", line 801, in _establish_connection<br>
conn = self.transport.establish_connection()<br>
File "/opt/app/venv/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 938, in establish_connection<br>
self._avail_channels.append(self.create_channel(self))<br>
File "/opt/app/venv/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 920, in create_channel<br>
channel = self.Channel(connection)<br>
File "/opt/app/venv/lib/python3.6/site-packages/kombu/transport/redis.py", line 526, in <strong>init</strong><br>
self.client.ping()<br>
File "/opt/app/venv/lib/python3.6/site-packages/redis/client.py", line 1378, in ping<br>
return self.execute_command('PING')<br>
File "/opt/app/venv/lib/python3.6/site-packages/redis/client.py", line 898, in execute_command<br>
conn = self.connection or pool.get_connection(command_name, **options)<br>
File "/opt/app/venv/lib/python3.6/site-packages/redis/connection.py", line 1187, in get_connection<br>
connection = self.make_connection()<br>
File "/opt/app/venv/lib/python3.6/site-packages/redis/connection.py", line 1227, in make_connection<br>
return self.connection_class(**self.connection_kwargs)<br>
File "/opt/app/venv/lib/python3.6/site-packages/redis/connection.py", line 828, in <strong>init</strong><br>
super(SSLConnection, self).<strong>init</strong>(**kwargs)<br>
TypeError: <strong>init</strong>() got an unexpected keyword argument 'connection_pool'<br>
</p></pre><p dir="auto"></p>
</details>
<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> This has already been asked to the <a href="https://groups.google.com/forum/#!forum/celery-users" rel="nofollow">discussion group</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>: celery==4.4.2</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2020-05-31 23:28:34,434: INFO/MainProcess] Connected to amqp://remote_worker:**@127.0.0.1:5672//
[2020-05-31 23:28:34,453: INFO/MainProcess] mingle: searching for neighbors
[2020-05-31 23:28:35,487: INFO/MainProcess] mingle: all alone
[2020-05-31 23:28:35,528: WARNING/MainProcess] /home/ubuntu/.local/lib/python3.7/site-packages/celery/fixups/django.py:203: UserWarning: Using settings.DEBUG leads to a memory
            leak, never use this setting in production environments!
  leak, never use this setting in production environments!''')
[2020-05-31 23:28:35,529: INFO/MainProcess] celery@testroom ready.
[2020-05-31 23:28:47,351: INFO/MainProcess] Received task: api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906]  
[2020-05-31 23:28:47,689: WARNING/ForkPoolWorker-1] started
[2020-05-31 23:28:47,690: WARNING/ForkPoolWorker-1] retry
[2020-05-31 23:28:47,721: INFO/MainProcess] Received task: api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906]  ETA:[2020-05-31 23:28:57.692348+00:00] 
[2020-05-31 23:28:47,722: INFO/MainProcess] Received task: api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906]  ETA:[2020-05-31 23:28:57.716321+00:00] 
[2020-05-31 23:28:47,777: INFO/ForkPoolWorker-1] Task api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906] retry: Retry in 10s: Retry(Retry(...), Exception('i have filled now'), 10)
[2020-05-31 23:28:57,999: WARNING/ForkPoolWorker-1] started
[2020-05-31 23:28:58,000: WARNING/ForkPoolWorker-1] ended
[2020-05-31 23:28:58,062: INFO/ForkPoolWorker-1] Task api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906] succeeded in 0.34440315900428686s: None
[2020-05-31 23:28:58,301: WARNING/ForkPoolWorker-1] started
[2020-05-31 23:28:58,302: WARNING/ForkPoolWorker-1] retry
[2020-05-31 23:28:58,304: INFO/MainProcess] Received task: api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906]  ETA:[2020-05-31 23:29:08.303091+00:00] 
[2020-05-31 23:28:58,307: INFO/MainProcess] Received task: api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906]  ETA:[2020-05-31 23:29:08.306141+00:00] 
[2020-05-31 23:28:58,368: INFO/ForkPoolWorker-1] Task api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906] retry: Retry in 10s: Retry(Retry(...), Exception('i have filled now'), 10)
[2020-05-31 23:29:08,572: WARNING/ForkPoolWorker-1] started
[2020-05-31 23:29:08,573: WARNING/ForkPoolWorker-1] ended
[2020-05-31 23:29:08,633: INFO/ForkPoolWorker-1] Task api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906] succeeded in 0.3256059319974156s: None
[2020-05-31 23:29:08,872: WARNING/ForkPoolWorker-1] started
[2020-05-31 23:29:08,873: WARNING/ForkPoolWorker-1] retry
[2020-05-31 23:29:08,875: INFO/MainProcess] Received task: api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906]  ETA:[2020-05-31 23:29:18.873799+00:00] 
[2020-05-31 23:29:08,880: INFO/MainProcess] Received task: api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906]  ETA:[2020-05-31 23:29:18.877550+00:00] 
[2020-05-31 23:29:08,940: INFO/ForkPoolWorker-1] Task api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906] retry: Retry in 10s: Retry(Retry(...), Exception('i have filled now'), 10)
[2020-05-31 23:29:19,144: WARNING/ForkPoolWorker-1] started
[2020-05-31 23:29:19,145: WARNING/ForkPoolWorker-1] ended
[2020-05-31 23:29:19,205: INFO/ForkPoolWorker-1] Task api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906] succeeded in 0.326258520995907s: None
[2020-05-31 23:29:19,444: WARNING/ForkPoolWorker-1] started
[2020-05-31 23:29:19,445: WARNING/ForkPoolWorker-1] retry
[2020-05-31 23:29:19,505: ERROR/ForkPoolWorker-1] Task api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906] raised unexpected: Exception('i have filled now')
Traceback (most recent call last):
  File &quot;/home/ubuntu/.local/lib/python3.7/site-packages/celery/app/trace.py&quot;, line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File &quot;/home/ubuntu/.local/lib/python3.7/site-packages/celery/app/trace.py&quot;, line 650, in __protected_call__
    return self.run(*args, **kwargs)
  File &quot;/home/ubuntu/.local/lib/python3.7/site-packages/celery/app/base.py&quot;, line 500, in run
    raise task.retry(exc=exc, **retry_kwargs)
  File &quot;/home/ubuntu/.local/lib/python3.7/site-packages/celery/app/task.py&quot;, line 704, in retry
    raise_with_context(exc)
  File &quot;/home/ubuntu/.local/lib/python3.7/site-packages/celery/app/base.py&quot;, line 487, in run
    return task._orig_run(*args, **kwargs)
  File &quot;/var/www/django_projects/earthalytics-api/api_v3/tests.py&quot;, line 26, in execute
    self.retry(exc=Exception(&quot;i have filled now&quot;), args=[param_a, param_b], kwargs=kwargs)
  File &quot;/home/ubuntu/.local/lib/python3.7/site-packages/celery/app/task.py&quot;, line 704, in retry
    raise_with_context(exc)
  File &quot;/home/ubuntu/.local/lib/python3.7/site-packages/celery/utils/serialization.py&quot;, line 288, in raise_with_context
    _raise_with_context(exc, exc_info[1])
  File &quot;&lt;string&gt;&quot;, line 1, in _raise_with_context
Exception: i have filled now
"><pre class="notranslate"><code class="notranslate">[2020-05-31 23:28:34,434: INFO/MainProcess] Connected to amqp://remote_worker:**@127.0.0.1:5672//
[2020-05-31 23:28:34,453: INFO/MainProcess] mingle: searching for neighbors
[2020-05-31 23:28:35,487: INFO/MainProcess] mingle: all alone
[2020-05-31 23:28:35,528: WARNING/MainProcess] /home/ubuntu/.local/lib/python3.7/site-packages/celery/fixups/django.py:203: UserWarning: Using settings.DEBUG leads to a memory
            leak, never use this setting in production environments!
  leak, never use this setting in production environments!''')
[2020-05-31 23:28:35,529: INFO/MainProcess] celery@testroom ready.
[2020-05-31 23:28:47,351: INFO/MainProcess] Received task: api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906]  
[2020-05-31 23:28:47,689: WARNING/ForkPoolWorker-1] started
[2020-05-31 23:28:47,690: WARNING/ForkPoolWorker-1] retry
[2020-05-31 23:28:47,721: INFO/MainProcess] Received task: api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906]  ETA:[2020-05-31 23:28:57.692348+00:00] 
[2020-05-31 23:28:47,722: INFO/MainProcess] Received task: api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906]  ETA:[2020-05-31 23:28:57.716321+00:00] 
[2020-05-31 23:28:47,777: INFO/ForkPoolWorker-1] Task api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906] retry: Retry in 10s: Retry(Retry(...), Exception('i have filled now'), 10)
[2020-05-31 23:28:57,999: WARNING/ForkPoolWorker-1] started
[2020-05-31 23:28:58,000: WARNING/ForkPoolWorker-1] ended
[2020-05-31 23:28:58,062: INFO/ForkPoolWorker-1] Task api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906] succeeded in 0.34440315900428686s: None
[2020-05-31 23:28:58,301: WARNING/ForkPoolWorker-1] started
[2020-05-31 23:28:58,302: WARNING/ForkPoolWorker-1] retry
[2020-05-31 23:28:58,304: INFO/MainProcess] Received task: api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906]  ETA:[2020-05-31 23:29:08.303091+00:00] 
[2020-05-31 23:28:58,307: INFO/MainProcess] Received task: api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906]  ETA:[2020-05-31 23:29:08.306141+00:00] 
[2020-05-31 23:28:58,368: INFO/ForkPoolWorker-1] Task api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906] retry: Retry in 10s: Retry(Retry(...), Exception('i have filled now'), 10)
[2020-05-31 23:29:08,572: WARNING/ForkPoolWorker-1] started
[2020-05-31 23:29:08,573: WARNING/ForkPoolWorker-1] ended
[2020-05-31 23:29:08,633: INFO/ForkPoolWorker-1] Task api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906] succeeded in 0.3256059319974156s: None
[2020-05-31 23:29:08,872: WARNING/ForkPoolWorker-1] started
[2020-05-31 23:29:08,873: WARNING/ForkPoolWorker-1] retry
[2020-05-31 23:29:08,875: INFO/MainProcess] Received task: api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906]  ETA:[2020-05-31 23:29:18.873799+00:00] 
[2020-05-31 23:29:08,880: INFO/MainProcess] Received task: api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906]  ETA:[2020-05-31 23:29:18.877550+00:00] 
[2020-05-31 23:29:08,940: INFO/ForkPoolWorker-1] Task api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906] retry: Retry in 10s: Retry(Retry(...), Exception('i have filled now'), 10)
[2020-05-31 23:29:19,144: WARNING/ForkPoolWorker-1] started
[2020-05-31 23:29:19,145: WARNING/ForkPoolWorker-1] ended
[2020-05-31 23:29:19,205: INFO/ForkPoolWorker-1] Task api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906] succeeded in 0.326258520995907s: None
[2020-05-31 23:29:19,444: WARNING/ForkPoolWorker-1] started
[2020-05-31 23:29:19,445: WARNING/ForkPoolWorker-1] retry
[2020-05-31 23:29:19,505: ERROR/ForkPoolWorker-1] Task api_v3.tests.execute[e97d93b5-b0e5-4b87-96ab-1aab66119906] raised unexpected: Exception('i have filled now')
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.7/site-packages/celery/app/trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/celery/app/trace.py", line 650, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/celery/app/base.py", line 500, in run
    raise task.retry(exc=exc, **retry_kwargs)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/celery/app/task.py", line 704, in retry
    raise_with_context(exc)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/celery/app/base.py", line 487, in run
    return task._orig_run(*args, **kwargs)
  File "/var/www/django_projects/earthalytics-api/api_v3/tests.py", line 26, in execute
    self.retry(exc=Exception("i have filled now"), args=[param_a, param_b], kwargs=kwargs)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/celery/app/task.py", line 704, in retry
    raise_with_context(exc)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/celery/utils/serialization.py", line 288, in raise_with_context
    _raise_with_context(exc, exc_info[1])
  File "&lt;string&gt;", line 1, in _raise_with_context
Exception: i have filled now

</code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<p dir="auto">Make a celery task with a retry changing one parameters.<br>
Set the max_retries and countdown.</p>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Celery Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Kombu Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Version</strong>: N/A or Unknown</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Client Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"><code class="notranslate"></code></pre></div>
<p dir="auto"></p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
N/A
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>

<details>
<p dir="auto">
</p><div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="
@app_cluster.task(bind=True, autoretry_for=(Exception,), max_retries=3,
                  default_retry_delay=10)
def execute(self, param_a, param_b=None, **kwargs):
    print(&quot;started&quot;)
    if param_b is None:
        param_b = &quot;filled&quot;
        print(&quot;retry&quot;)
        self.retry(exc=Exception(&quot;i have filled now&quot;), args=[param_a, param_b], kwargs=kwargs)
    print(&quot;ended&quot;)

def test_celery(self):
    sig = execute.si(&quot;something&quot;)
    t = sig.delay()
    t = 0"><pre class="notranslate"><span class="pl-en">@<span class="pl-s1">app_cluster</span>.<span class="pl-en">task</span>(<span class="pl-s1">bind</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">autoretry_for</span><span class="pl-c1">=</span>(<span class="pl-v">Exception</span>,), <span class="pl-s1">max_retries</span><span class="pl-c1">=</span><span class="pl-c1">3</span>,</span>
<span class="pl-en">                  <span class="pl-s1">default_retry_delay</span><span class="pl-c1">=</span><span class="pl-c1">10</span>)</span>
<span class="pl-k">def</span> <span class="pl-en">execute</span>(<span class="pl-s1">self</span>, <span class="pl-s1">param_a</span>, <span class="pl-s1">param_b</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
    <span class="pl-en">print</span>(<span class="pl-s">"started"</span>)
    <span class="pl-k">if</span> <span class="pl-s1">param_b</span> <span class="pl-c1">is</span> <span class="pl-c1">None</span>:
        <span class="pl-s1">param_b</span> <span class="pl-c1">=</span> <span class="pl-s">"filled"</span>
        <span class="pl-en">print</span>(<span class="pl-s">"retry"</span>)
        <span class="pl-s1">self</span>.<span class="pl-en">retry</span>(<span class="pl-s1">exc</span><span class="pl-c1">=</span><span class="pl-v">Exception</span>(<span class="pl-s">"i have filled now"</span>), <span class="pl-s1">args</span><span class="pl-c1">=</span>[<span class="pl-s1">param_a</span>, <span class="pl-s1">param_b</span>], <span class="pl-s1">kwargs</span><span class="pl-c1">=</span><span class="pl-s1">kwargs</span>)
    <span class="pl-en">print</span>(<span class="pl-s">"ended"</span>)

<span class="pl-k">def</span> <span class="pl-en">test_celery</span>(<span class="pl-s1">self</span>):
    <span class="pl-s1">sig</span> <span class="pl-c1">=</span> <span class="pl-s1">execute</span>.<span class="pl-en">si</span>(<span class="pl-s">"something"</span>)
    <span class="pl-s1">t</span> <span class="pl-c1">=</span> <span class="pl-s1">sig</span>.<span class="pl-en">delay</span>()
    <span class="pl-s1">t</span> <span class="pl-c1">=</span> <span class="pl-c1">0</span></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>

<p dir="auto">I expect the task get overrided with updated parameters</p>
<h1 dir="auto">Actual Behavior</h1>
 The task goes in retry with origianl parameters.
<p dir="auto">A new task is made with new parameters and get scheduled.<br>
The "old" task get executed.<br>
The "new" task get executed.<br>
The "old" task goes in retry making a "new" "new task" with updated parameters and the "old" goes scheduled again with original parameters.</p>