<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> This has already been asked to the <a href="https://groups.google.com/forum/#!forum/celery-users" rel="nofollow">discussion group</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li>[ x ] I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="501265323" data-permission-text="Title is private" data-url="https://github.com/celery/kombu/issues/1102" data-hovercard-type="issue" data-hovercard-url="/celery/kombu/issues/1102/hovercard" href="https://github.com/celery/kombu/issues/1102">celery/kombu#1102</a></li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="501265323" data-permission-text="Title is private" data-url="https://github.com/celery/kombu/issues/1102" data-hovercard-type="issue" data-hovercard-url="/celery/kombu/issues/1102/hovercard" href="https://github.com/celery/kombu/issues/1102">celery/kombu#1102</a></li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>:<br>
5.0.0rc3 (singularity)</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[ec2-user@ip-172-31-50-49 ~]$ celery -A celery_worker report

software -&gt; celery:5.0.0rc3 (singularity) kombu:5.0.2 py:3.7.9
            billiard:3.6.3.0 sqs:N/A
platform -&gt; system:Linux arch:64bit, ELF
            kernel version:4.14.193-149.317.amzn2.x86_64 imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:sqs results:disabled

broker_url: 'sqs://localhost//'"><pre class="notranslate"><code class="notranslate">[ec2-user@ip-172-31-50-49 ~]$ celery -A celery_worker report

software -&gt; celery:5.0.0rc3 (singularity) kombu:5.0.2 py:3.7.9
            billiard:3.6.3.0 sqs:N/A
platform -&gt; system:Linux arch:64bit, ELF
            kernel version:4.14.193-149.317.amzn2.x86_64 imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:sqs results:disabled

broker_url: 'sqs://localhost//'
</code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Celery Version</strong>: 4.4.3</li>
<li><strong>Minimal Kombu Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Version</strong>: N/A or Unknown</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Client Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="amqp==5.0.1
awscli==1.18.139
backcall==0.2.0
billiard==3.6.3.0
boto3==1.14.62
botocore==1.17.62
celery==5.0.0rc3
click==7.1.2
click-didyoumean==0.0.3
click-repl==0.1.6
colorama==0.4.3
decorator==4.4.2
docutils==0.15.2
importlib-metadata==1.7.0
ipython==7.18.1
ipython-genutils==0.2.0
jedi==0.17.2
jmespath==0.10.0
kombu==5.0.2
parso==0.7.1
pexpect==4.8.0
pickleshare==0.7.5
prompt-toolkit==3.0.7
ptyprocess==0.6.0
pyasn1==0.4.8
pycurl==7.43.0.6
Pygments==2.7.0
python-dateutil==2.8.1
pytz==2020.1
PyYAML==5.3.1
rsa==4.5
s3transfer==0.3.3
six==1.15.0
traitlets==5.0.4
urllib3==1.25.10
vine==5.0.0
wcwidth==0.2.5
zipp==3.1.0"><pre class="notranslate"><code class="notranslate">amqp==5.0.1
awscli==1.18.139
backcall==0.2.0
billiard==3.6.3.0
boto3==1.14.62
botocore==1.17.62
celery==5.0.0rc3
click==7.1.2
click-didyoumean==0.0.3
click-repl==0.1.6
colorama==0.4.3
decorator==4.4.2
docutils==0.15.2
importlib-metadata==1.7.0
ipython==7.18.1
ipython-genutils==0.2.0
jedi==0.17.2
jmespath==0.10.0
kombu==5.0.2
parso==0.7.1
pexpect==4.8.0
pickleshare==0.7.5
prompt-toolkit==3.0.7
ptyprocess==0.6.0
pyasn1==0.4.8
pycurl==7.43.0.6
Pygments==2.7.0
python-dateutil==2.8.1
pytz==2020.1
PyYAML==5.3.1
rsa==4.5
s3transfer==0.3.3
six==1.15.0
traitlets==5.0.4
urllib3==1.25.10
vine==5.0.0
wcwidth==0.2.5
zipp==3.1.0
</code></pre></div>
<p dir="auto"></p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
N/A
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>

<details>
<p dir="auto">
</p><p dir="auto">Schedule 1000 simple <code class="notranslate">4+4</code> tasks from howto guide, launch worker.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="In [33]: %time for x in range(1000):add.apply_async(args=(4, 4))
CPU times: user 3.4 s, sys: 61.7 ms, total: 3.46 s
Wall time: 12.7 s

from celery import Celery

app = Celery('tasks', broker='sqs://')

@app.task
def add(x, y):
    return x + y"><pre class="notranslate"><span class="pl-v">In</span> [<span class="pl-c1">33</span>]: <span class="pl-c1">%</span><span class="pl-s1">time</span> <span class="pl-s1">for</span> <span class="pl-s1">x</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">1000</span>):<span class="pl-s1">add</span>.<span class="pl-en">apply_async</span>(<span class="pl-s1">args</span><span class="pl-c1">=</span>(<span class="pl-c1">4</span>, <span class="pl-c1">4</span>))
<span class="pl-v">CPU</span> <span class="pl-s1">times</span>: <span class="pl-s1">user</span> <span class="pl-c1">3.4</span> <span class="pl-s1">s</span>, <span class="pl-s1">sys</span>: <span class="pl-c1">61.7</span> <span class="pl-s1">ms</span>, <span class="pl-s1">total</span>: <span class="pl-c1">3.46</span> <span class="pl-s1">s</span>
<span class="pl-v">Wall</span> <span class="pl-s1">time</span>: <span class="pl-c1">12.7</span> <span class="pl-s1">s</span>

<span class="pl-k">from</span> <span class="pl-s1">celery</span> <span class="pl-k">import</span> <span class="pl-v">Celery</span>

<span class="pl-s1">app</span> <span class="pl-c1">=</span> <span class="pl-v">Celery</span>(<span class="pl-s">'tasks'</span>, <span class="pl-s1">broker</span><span class="pl-c1">=</span><span class="pl-s">'sqs://'</span>)

<span class="pl-en">@<span class="pl-s1">app</span>.<span class="pl-s1">task</span></span>
<span class="pl-k">def</span> <span class="pl-en">add</span>(<span class="pl-s1">x</span>, <span class="pl-s1">y</span>):
    <span class="pl-k">return</span> <span class="pl-s1">x</span> <span class="pl-c1">+</span> <span class="pl-s1">y</span></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>

<p dir="auto">Higher throughput with higher CPU utilisation</p>
<h1 dir="auto">Actual Behavior</h1>

<p dir="auto">Worker processes ~10 messages per second. CPU utilisation is low, looks like celery waits most of the time.<br>
Celery and all test code runs was tested on fresh EC2 instance in <code class="notranslate">us-east-1</code>, with SQS in the same region.<br>
At the same time following code allows to consume all 1000 messages from queue in ~10 seconds, event without using bulk deletes or any async APIs:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="In [30]: %time for x in range(1000):add.apply_async(args=(4, 4))
CPU times: user 3.44 s, sys: 80.7 ms, total: 3.52 s
Wall time: 12.9 s

In [31]: def get_all():
    ...:     while True:
    ...:         messages = client.receive_message(QueueUrl='https://queue.amazonaws.com/385755224306/celery', MaxNumberOfMessages=10)
    ...:         if 'Messages' not in messages:
    ...:             break
    ...:         for message in messages['Messages']:
    ...:             client.delete_message(QueueUrl='https://queue.amazonaws.com/385755224306/celery', ReceiptHandle=message['ReceiptHandle'])
    ...: 
In [35]: %time get_all()
CPU times: user 2.9 s, sys: 84.9 ms, total: 2.98 s
Wall time: 9.72 s"><pre class="notranslate"><code class="notranslate">In [30]: %time for x in range(1000):add.apply_async(args=(4, 4))
CPU times: user 3.44 s, sys: 80.7 ms, total: 3.52 s
Wall time: 12.9 s

In [31]: def get_all():
    ...:     while True:
    ...:         messages = client.receive_message(QueueUrl='https://queue.amazonaws.com/385755224306/celery', MaxNumberOfMessages=10)
    ...:         if 'Messages' not in messages:
    ...:             break
    ...:         for message in messages['Messages']:
    ...:             client.delete_message(QueueUrl='https://queue.amazonaws.com/385755224306/celery', ReceiptHandle=message['ReceiptHandle'])
    ...: 
In [35]: %time get_all()
CPU times: user 2.9 s, sys: 84.9 ms, total: 2.98 s
Wall time: 9.72 s
</code></pre></div>
<p dir="auto">There are visible delays in celery worker log output, for example:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2020-09-16 14:02:14,573: INFO/ForkPoolWorker-1] Task celery_worker.add[88d2fce8-0d0b-49fe-b0a7-2713b8ab6248] succeeded in 4.8579999202047475e-05s: 8
[2020-09-16 14:02:15,489: INFO/MainProcess] Received task: celery_worker.add[76ec9b77-bd42-4042-9933-35554e678506]"><pre class="notranslate"><code class="notranslate">[2020-09-16 14:02:14,573: INFO/ForkPoolWorker-1] Task celery_worker.add[88d2fce8-0d0b-49fe-b0a7-2713b8ab6248] succeeded in 4.8579999202047475e-05s: 8
[2020-09-16 14:02:15,489: INFO/MainProcess] Received task: celery_worker.add[76ec9b77-bd42-4042-9933-35554e678506]
</code></pre></div>
<p dir="auto">Looks like delay always happens once tasks have finished execution, but before new tasks were received from queue.<br>
Tests and timings:</p>
<ul dir="auto">
<li>-c1, default prefetch -&gt; 9m30s</li>
<li>-c5, default prefetch -&gt; 3m30s</li>
<li>-c10, default prefetch -&gt; 1m45s</li>
<li>-c20, default prefetch -&gt; 1m44s</li>
<li>-c1, prefetch 100 -&gt; 1m45s<br>
Looks like celery throughput somehow gets limited by SQS <code class="notranslate">MaxNumberOfMessages=10</code> limitation, despite concurrency or prefetch settings. However, boto3 sample script shows that AWS API allows to consume messages match faster, so there should be some possible optimisation for Celery/Kombu.</li>
</ul>
<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> This has already been asked to the <a href="https://github.com/celery/celery/discussions">discussions forum</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>: 5.2.1 (dawn-chorus)</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;/tmp/test/venv/bin/celery&quot;, line 8, in &lt;module&gt;
    sys.exit(main())
  File &quot;/tmp/test/venv/lib/python3.9/site-packages/celery/__main__.py&quot;, line 15, in main
    sys.exit(_main())
  File &quot;/tmp/test/venv/lib/python3.9/site-packages/celery/bin/celery.py&quot;, line 213, in main
    return celery(auto_envvar_prefix=&quot;CELERY&quot;)
  File &quot;/tmp/test/venv/lib/python3.9/site-packages/click/core.py&quot;, line 1128, in __call__
    return self.main(*args, **kwargs)
  File &quot;/tmp/test/venv/lib/python3.9/site-packages/click/core.py&quot;, line 1053, in main
    rv = self.invoke(ctx)
  File &quot;/tmp/test/venv/lib/python3.9/site-packages/click/core.py&quot;, line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File &quot;/tmp/test/venv/lib/python3.9/site-packages/click/core.py&quot;, line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File &quot;/tmp/test/venv/lib/python3.9/site-packages/click/core.py&quot;, line 754, in invoke
    return __callback(*args, **kwargs)
  File &quot;/tmp/test/venv/lib/python3.9/site-packages/click/decorators.py&quot;, line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
TypeError: report() got an unexpected keyword argument 'ini'"><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):
  File "/tmp/test/venv/bin/celery", line 8, in &lt;module&gt;
    sys.exit(main())
  File "/tmp/test/venv/lib/python3.9/site-packages/celery/__main__.py", line 15, in main
    sys.exit(_main())
  File "/tmp/test/venv/lib/python3.9/site-packages/celery/bin/celery.py", line 213, in main
    return celery(auto_envvar_prefix="CELERY")
  File "/tmp/test/venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/tmp/test/venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/tmp/test/venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/tmp/test/venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmp/test/venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/tmp/test/venv/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
TypeError: report() got an unexpected keyword argument 'ini'
</code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<p dir="auto">Define customer user option and run <code class="notranslate">celery -A proj multi start --my-custom-option=foo</code>.</p>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Celery Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Kombu Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Version</strong>: N/A or Unknown</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Client Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"><code class="notranslate">
</code></pre></div>
<p dir="auto"></p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
N/A
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>

<details>
<p dir="auto">
</p><div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>

<p dir="auto"><code class="notranslate">Celery multi</code> should work with user options like <code class="notranslate">celery worker</code></p>
<h1 dir="auto">Actual Behavior</h1>
<p dir="auto">Starting an application that has custom options with <code class="notranslate">celery multi start</code> fails.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# celery -A proj multi start worker1 \
          --ini=config.ini \
          --concurrency=4 -Q celery
/tmp/test/venv/lib/python3.9/site-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 1.14.0-unknown is an invalid version and will not be supported in a future release
  warnings.warn(
Traceback (most recent call last):
  File &quot;/tmp/test/venv/bin/celery&quot;, line 8, in &lt;module&gt;
    sys.exit(main())
  File &quot;/tmp/test/venv/lib/python3.9/site-packages/celery/__main__.py&quot;, line 15, in main
    sys.exit(_main())
  File &quot;/tmp/test/venv/lib/python3.9/site-packages/celery/bin/celery.py&quot;, line 213, in main
    return celery(auto_envvar_prefix=&quot;CELERY&quot;)
  File &quot;/tmp/test/venv/lib/python3.9/site-packages/click/core.py&quot;, line 1128, in __call__
    return self.main(*args, **kwargs)
  File &quot;/tmp/test/venv/lib/python3.9/site-packages/click/core.py&quot;, line 1053, in main
    rv = self.invoke(ctx)
  File &quot;/tmp/test/venv/lib/python3.9/site-packages/click/core.py&quot;, line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File &quot;/tmp/test/venv/lib/python3.9/site-packages/click/core.py&quot;, line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File &quot;/tmp/test/venv/lib/python3.9/site-packages/click/core.py&quot;, line 754, in invoke
    return __callback(*args, **kwargs)
  File &quot;/tmp/test/venv/lib/python3.9/site-packages/click/decorators.py&quot;, line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File &quot;/tmp/test/venv/lib/python3.9/site-packages/celery/bin/base.py&quot;, line 134, in caller
    return f(ctx, *args, **kwargs)
TypeError: multi() got an unexpected keyword argument 'ini'"><pre class="notranslate"><code class="notranslate"># celery -A proj multi start worker1 \
          --ini=config.ini \
          --concurrency=4 -Q celery
/tmp/test/venv/lib/python3.9/site-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 1.14.0-unknown is an invalid version and will not be supported in a future release
  warnings.warn(
Traceback (most recent call last):
  File "/tmp/test/venv/bin/celery", line 8, in &lt;module&gt;
    sys.exit(main())
  File "/tmp/test/venv/lib/python3.9/site-packages/celery/__main__.py", line 15, in main
    sys.exit(_main())
  File "/tmp/test/venv/lib/python3.9/site-packages/celery/bin/celery.py", line 213, in main
    return celery(auto_envvar_prefix="CELERY")
  File "/tmp/test/venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/tmp/test/venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/tmp/test/venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/tmp/test/venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmp/test/venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/tmp/test/venv/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/tmp/test/venv/lib/python3.9/site-packages/celery/bin/base.py", line 134, in caller
    return f(ctx, *args, **kwargs)
TypeError: multi() got an unexpected keyword argument 'ini'
</code></pre></div>
<p dir="auto">The multi function in <code class="notranslate">celery/celery/bin/multi.py</code> only accepts the <code class="notranslate">ctx</code> parameter, but<br>
the <code class="notranslate">handle_preload_options</code> decorator is calling <code class="notranslate">f(ctx, *args, **kwargs)</code>.</p>