<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> This has already been asked to the <a href="https://groups.google.com/forum/#!forum/celery-users" rel="nofollow">discussion group</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>:</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"><code class="notranslate"></code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="~/celery $ docker build -f docker/Dockerfile ."><pre class="notranslate"><code class="notranslate">~/celery $ docker build -f docker/Dockerfile .
</code></pre></div>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Celery Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Kombu Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Version</strong>: N/A or Unknown</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Client Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Not applicable"><pre class="notranslate"><code class="notranslate">Not applicable
</code></pre></div>
<p dir="auto"></p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
N/A
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>

<details>
<p dir="auto">
</p><div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>

<p dir="auto">The docker image should build without error.</p>
<h1 dir="auto">Actual Behavior</h1>

<p dir="auto">The image fails to build due to a dependency versioning issue with couchbase.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ podman build -t celery -f docker/Dockerfile .
STEP 1: FROM ubuntu:focal
STEP 2: ENV PYTHONIOENCODING UTF-8
--&gt; Using cache acaba7be84835036fb85e90b837986d94871823c68921aca8b9bc6858f7eede6
STEP 3: ARG DEBIAN_FRONTEND=noninteractive
--&gt; Using cache f53b7bd5d6f45cd40239c7d083c268db8b62e826a8018a47c32d96c8e439952f
STEP 4: RUN apt-get update &amp;&amp; apt-get install -y build-essential     libcurl4-openssl-dev     libffi-dev     tk-dev     xz-utils     curl     lsb-release     git     libmemcached-dev     make     liblzma-dev     libreadline-dev     libbz2-dev     llvm     libncurses5-dev     libsqlite3-dev     wget     pypy     python-openssl     libncursesw5-dev     zlib1g-dev     pkg-config     libssl-dev     sudo
--&gt; Using cache 3528dd3a7bde42f48e3f792b717ce95ad3bcc1d169b2d676abf20faaa2eaaacc
STEP 5: ENV PROVISIONING /provisioning
--&gt; Using cache 5cd403e630187ed510ed7aa5c1a7279b2e169e77a35a17dacaf82a49bbc8ce7c
STEP 6: ARG CELERY_USER=developer
--&gt; Using cache faa9faee390dded0bab0e78ebca4b9075b19f05a50bc055bb77f88efbd64682f
STEP 7: RUN : &quot;${CELERY_USER:?CELERY_USER build argument needs to be set and non-empty.}&quot;
--&gt; Using cache 43389d0aac47abf3b16a5af3689c5371180a6b3950841c179a2569affffed4eb
STEP 8: ENV HOME /home/$CELERY_USER
--&gt; Using cache ec0be7e288094b41313bd80d5ea9df3fbb44b99cd8a7299f5a4b21a4d7932b39
STEP 9: ENV PATH=&quot;$HOME/.pyenv/bin:$PATH&quot;
--&gt; Using cache e7239a2d2c12e00bfabaec790d0c1937690095b55c3473b80a13c3afbd9a0333
STEP 10: WORKDIR $PROVISIONING
--&gt; Using cache f0f43f30b813e023ba7a9edb2fdef8761187b529deb69f4e131d207665a8016b
STEP 11: COPY docker/scripts/install-couchbase.sh .
--&gt; Using cache 86faecc53eba42812d4ac16e7498707e855e93030de83e14230f5da13391ca4f
STEP 12: RUN sh install-couchbase.sh
sudo: setrlimit(RLIMIT_CORE): Operation not permitted
sudo: setrlimit(RLIMIT_CORE): Operation not permitted
--2020-07-14 01:15:18--  http://packages.couchbase.com/ubuntu/couchbase.key
Resolving packages.couchbase.com (packages.couchbase.com)... 143.204.83.79, 143.204.83.24, 143.204.83.78, ...
Connecting to packages.couchbase.com (packages.couchbase.com)|143.204.83.79|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4815 (4.7K) [application/pgp-keys]
Saving to: 'STDOUT'

-                                 100%[=============================================================&gt;]   4.70K  --.-KB/s    in 0s

2020-07-14 01:15:19 (57.3 MB/s) - written to stdout [4815/4815]

OK
sudo: setrlimit(RLIMIT_CORE): Operation not permitted
deb http://packages.couchbase.com/ubuntu bionic bionic/main
Get:1 http://packages.couchbase.com/ubuntu bionic InRelease [2992 B]
Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease
Get:4 http://packages.couchbase.com/ubuntu bionic/bionic/main amd64 Packages [1419 B]
Hit:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Fetched 4411 B in 2s (2570 B/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
build-essential is already the newest version (12.8ubuntu1).
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 libcouchbase2-bin : Depends: libcouchbase2-libevent (= 2.10.6-1) but it is not going to be installed
                     Depends: libevent-core-2.1-6 (&gt;= 2.1.8-stable) but it is not installable
E: Unable to correct problems, you have held broken packages.
Error: error building at STEP &quot;RUN sh install-couchbase.sh&quot;: error while running runtime: exit status 100"><pre class="notranslate"><code class="notranslate">$ podman build -t celery -f docker/Dockerfile .
STEP 1: FROM ubuntu:focal
STEP 2: ENV PYTHONIOENCODING UTF-8
--&gt; Using cache acaba7be84835036fb85e90b837986d94871823c68921aca8b9bc6858f7eede6
STEP 3: ARG DEBIAN_FRONTEND=noninteractive
--&gt; Using cache f53b7bd5d6f45cd40239c7d083c268db8b62e826a8018a47c32d96c8e439952f
STEP 4: RUN apt-get update &amp;&amp; apt-get install -y build-essential     libcurl4-openssl-dev     libffi-dev     tk-dev     xz-utils     curl     lsb-release     git     libmemcached-dev     make     liblzma-dev     libreadline-dev     libbz2-dev     llvm     libncurses5-dev     libsqlite3-dev     wget     pypy     python-openssl     libncursesw5-dev     zlib1g-dev     pkg-config     libssl-dev     sudo
--&gt; Using cache 3528dd3a7bde42f48e3f792b717ce95ad3bcc1d169b2d676abf20faaa2eaaacc
STEP 5: ENV PROVISIONING /provisioning
--&gt; Using cache 5cd403e630187ed510ed7aa5c1a7279b2e169e77a35a17dacaf82a49bbc8ce7c
STEP 6: ARG CELERY_USER=developer
--&gt; Using cache faa9faee390dded0bab0e78ebca4b9075b19f05a50bc055bb77f88efbd64682f
STEP 7: RUN : "${CELERY_USER:?CELERY_USER build argument needs to be set and non-empty.}"
--&gt; Using cache 43389d0aac47abf3b16a5af3689c5371180a6b3950841c179a2569affffed4eb
STEP 8: ENV HOME /home/$CELERY_USER
--&gt; Using cache ec0be7e288094b41313bd80d5ea9df3fbb44b99cd8a7299f5a4b21a4d7932b39
STEP 9: ENV PATH="$HOME/.pyenv/bin:$PATH"
--&gt; Using cache e7239a2d2c12e00bfabaec790d0c1937690095b55c3473b80a13c3afbd9a0333
STEP 10: WORKDIR $PROVISIONING
--&gt; Using cache f0f43f30b813e023ba7a9edb2fdef8761187b529deb69f4e131d207665a8016b
STEP 11: COPY docker/scripts/install-couchbase.sh .
--&gt; Using cache 86faecc53eba42812d4ac16e7498707e855e93030de83e14230f5da13391ca4f
STEP 12: RUN sh install-couchbase.sh
sudo: setrlimit(RLIMIT_CORE): Operation not permitted
sudo: setrlimit(RLIMIT_CORE): Operation not permitted
--2020-07-14 01:15:18--  http://packages.couchbase.com/ubuntu/couchbase.key
Resolving packages.couchbase.com (packages.couchbase.com)... 143.204.83.79, 143.204.83.24, 143.204.83.78, ...
Connecting to packages.couchbase.com (packages.couchbase.com)|143.204.83.79|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4815 (4.7K) [application/pgp-keys]
Saving to: 'STDOUT'

-                                 100%[=============================================================&gt;]   4.70K  --.-KB/s    in 0s

2020-07-14 01:15:19 (57.3 MB/s) - written to stdout [4815/4815]

OK
sudo: setrlimit(RLIMIT_CORE): Operation not permitted
deb http://packages.couchbase.com/ubuntu bionic bionic/main
Get:1 http://packages.couchbase.com/ubuntu bionic InRelease [2992 B]
Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease
Get:4 http://packages.couchbase.com/ubuntu bionic/bionic/main amd64 Packages [1419 B]
Hit:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Fetched 4411 B in 2s (2570 B/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
build-essential is already the newest version (12.8ubuntu1).
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 libcouchbase2-bin : Depends: libcouchbase2-libevent (= 2.10.6-1) but it is not going to be installed
                     Depends: libevent-core-2.1-6 (&gt;= 2.1.8-stable) but it is not installable
E: Unable to correct problems, you have held broken packages.
Error: error building at STEP "RUN sh install-couchbase.sh": error while running runtime: exit status 100
</code></pre></div>
<p dir="auto">The dockerfile was originally changed to be based on focal in <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/celery/celery/commit/8246f3a914376fe060668731f37b0f2e4ecb822d/hovercard" href="https://github.com/celery/celery/commit/8246f3a914376fe060668731f37b0f2e4ecb822d"><tt>8246f3a</tt></a>, but it's likely that a more recent change in the Couchbase bionic package repo or the Ubuntu focal repos is what causes the breakage. Changing the docker image to be based on bionic again allows the image to build again.</p>
<p dir="auto">The Task.retry function sets a default "countdown" option even if "eta" is set, and passes on these options to Task.apply_async. Apply_async ignores "eta" if "countdown" is set. Hence, "retry" always ends up ignoring "eta". Bug exists in both 2.1 &amp; 2.2</p>