<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> This has already been asked to the <a href="https://groups.google.com/forum/#!forum/celery-users" rel="nofollow">discussion group</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li><a href="https://groups.google.com/u/1/g/celery-users/c/OnviKXOZ99U/m/Fu7YIs4SAwAJ" rel="nofollow">Google groups discussion: "Celery only handle one message a time while cocurrency = 4 is set"</a></li>
<li><a href="https://groups.google.com/u/1/g/celery-users/c/VhiLNQl5xTw/m/4OIYKEeZBgAJ" rel="nofollow">Google groups discussion: "Concurrency with Consumers running in a Celery Worker"</a></li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>:</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="software -&gt; celery:5.0.5 (singularity) kombu:5.0.2 py:3.7.7
            billiard:3.6.3.0 py-amqp:5.0.5
platform -&gt; system:Linux arch:64bit
            kernel version:4.19.121-linuxkit imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:amqp results:disabled

api_request_timeout: '********'
broker_connection_max_retries: None
broker_connection_timeout: 30
broker_heartbeat: None
broker_pool_limit: 1
broker_transport_options: {
 'confirm_publish': True}
broker_url: 'amqp://admin:********@rabbitmq:5672//'
event_queue_expires: 60
os: &lt;module 'os' from '/usr/local/lib/python3.7/os.py'&gt;
result_backend: None
task_acks_late: True
task_acks_on_failure_or_timeout: False
task_compression: 'gzip'
task_create_missing_queues: True
task_ignore_result: True
task_reject_on_worker_lost: True
task_store_errors_even_if_ignored: False
task_time_limit: 30
task_track_started: True
worker_concurrency: 4
worker_enable_remote_control: False
worker_prefetch_multiplier: 10
deprecated_settings: None"><pre class="notranslate"><code class="notranslate">software -&gt; celery:5.0.5 (singularity) kombu:5.0.2 py:3.7.7
            billiard:3.6.3.0 py-amqp:5.0.5
platform -&gt; system:Linux arch:64bit
            kernel version:4.19.121-linuxkit imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:amqp results:disabled

api_request_timeout: '********'
broker_connection_max_retries: None
broker_connection_timeout: 30
broker_heartbeat: None
broker_pool_limit: 1
broker_transport_options: {
 'confirm_publish': True}
broker_url: 'amqp://admin:********@rabbitmq:5672//'
event_queue_expires: 60
os: &lt;module 'os' from '/usr/local/lib/python3.7/os.py'&gt;
result_backend: None
task_acks_late: True
task_acks_on_failure_or_timeout: False
task_compression: 'gzip'
task_create_missing_queues: True
task_ignore_result: True
task_reject_on_worker_lost: True
task_store_errors_even_if_ignored: False
task_time_limit: 30
task_track_started: True
worker_concurrency: 4
worker_enable_remote_control: False
worker_prefetch_multiplier: 10
deprecated_settings: None
</code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: 3.7.x</li>
<li><strong>Minimal Celery Version</strong>: 5.x</li>
<li><strong>Minimal Kombu Version</strong>: 5.x</li>
<li><strong>Minimal Broker Version</strong>: RabbitMQ 3.7.x</li>
<li><strong>Minimal Result Backend Version</strong>: N/A</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: Debian 9 (ie. the <code class="notranslate">python:3.7-stretch</code> docker image)</li>
<li><strong>Minimal Broker Client Version</strong>: Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="affine==2.3.0
amqp==5.0.5
attrs==20.3.0
billiard==3.6.3.0
blinker==1.4
boto3==1.14.20
botocore==1.17.63
cached-property==1.5.1
cachetools==4.2.1
celery==5.0.5
certifi==2020.12.5
chardet==3.0.4
click==7.1.2
click-didyoumean==0.0.3
click-plugins==1.1.1
click-repl==0.1.6
cligj==0.7.1
cycler==0.10.0
ddtrace==0.31.0
decorator==4.4.2
docutils==0.15.2
fast-slic==0.4.0
Flask==1.0.2
Flask-Script==2.0.6
flatten-dict==0.3.0
gevent==21.1.2
google-api-core==1.25.1
google-auth==1.24.0
google-cloud-core==1.5.0
google-cloud-storage==1.20.0
google-resumable-media==0.4.1
googleapis-common-protos==1.52.0
greenlet==1.0.0
gunicorn==19.9.0
humanize==3.2.0
idna==2.8
imageio==2.9.0
importlib-metadata==3.4.0
itsdangerous==1.1.0
Jinja2==2.11.2
jmespath==0.10.0
kiwisolver==1.3.1
kombu==5.0.2
MarkupSafe==1.1.1
matplotlib==3.1.3
more-itertools==8.6.0
mysqlclient==1.4.4
networkx==2.5
numpy==1.17.0
opencv-python==4.1.1.26
packaging==20.8
pathlib2==2.3.5
piexif==1.1.3
Pillow==6.1.0
pluggy==0.13.1
prometheus-client==0.7.1
prompt-toolkit==3.0.14
protobuf==3.14.0
psutil==5.8.0
py==1.10.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
PyJWT==2.0.1
pyparsing==2.4.7
pytest==5.4.1
pytest-flask==1.0.0
pytest-mock==1.10.4
pytest-runner==4.4
python-dateutil==2.8.1
python-magic==0.4.15
pytz==2020.5
PyWavelets==1.1.1
PyYAML==5.1.2
rasterio==1.1.6
requests==2.22.0
rsa==4.7
s3transfer==0.3.4
scikit-image==0.16.2
scipy==1.4.1
sentry-sdk==0.16.0
Shapely==1.6.4.post2
six==1.15.0
snuggs==1.4.7
SQLAlchemy==1.3.10
stringcase==1.2.0
structlog==19.1.0
timeout-decorator==0.5.0
tornado==6.1
typing-extensions==3.7.4.3
urllib3==1.25.11
vine==5.0.0
wcwidth==0.2.5
Werkzeug==1.0.1
zipp==3.4.0
zope.event==4.5.0
zope.interface==5.3.0
"><pre class="notranslate"><code class="notranslate">affine==2.3.0
amqp==5.0.5
attrs==20.3.0
billiard==3.6.3.0
blinker==1.4
boto3==1.14.20
botocore==1.17.63
cached-property==1.5.1
cachetools==4.2.1
celery==5.0.5
certifi==2020.12.5
chardet==3.0.4
click==7.1.2
click-didyoumean==0.0.3
click-plugins==1.1.1
click-repl==0.1.6
cligj==0.7.1
cycler==0.10.0
ddtrace==0.31.0
decorator==4.4.2
docutils==0.15.2
fast-slic==0.4.0
Flask==1.0.2
Flask-Script==2.0.6
flatten-dict==0.3.0
gevent==21.1.2
google-api-core==1.25.1
google-auth==1.24.0
google-cloud-core==1.5.0
google-cloud-storage==1.20.0
google-resumable-media==0.4.1
googleapis-common-protos==1.52.0
greenlet==1.0.0
gunicorn==19.9.0
humanize==3.2.0
idna==2.8
imageio==2.9.0
importlib-metadata==3.4.0
itsdangerous==1.1.0
Jinja2==2.11.2
jmespath==0.10.0
kiwisolver==1.3.1
kombu==5.0.2
MarkupSafe==1.1.1
matplotlib==3.1.3
more-itertools==8.6.0
mysqlclient==1.4.4
networkx==2.5
numpy==1.17.0
opencv-python==4.1.1.26
packaging==20.8
pathlib2==2.3.5
piexif==1.1.3
Pillow==6.1.0
pluggy==0.13.1
prometheus-client==0.7.1
prompt-toolkit==3.0.14
protobuf==3.14.0
psutil==5.8.0
py==1.10.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
PyJWT==2.0.1
pyparsing==2.4.7
pytest==5.4.1
pytest-flask==1.0.0
pytest-mock==1.10.4
pytest-runner==4.4
python-dateutil==2.8.1
python-magic==0.4.15
pytz==2020.5
PyWavelets==1.1.1
PyYAML==5.1.2
rasterio==1.1.6
requests==2.22.0
rsa==4.7
s3transfer==0.3.4
scikit-image==0.16.2
scipy==1.4.1
sentry-sdk==0.16.0
Shapely==1.6.4.post2
six==1.15.0
snuggs==1.4.7
SQLAlchemy==1.3.10
stringcase==1.2.0
structlog==19.1.0
timeout-decorator==0.5.0
tornado==6.1
typing-extensions==3.7.4.3
urllib3==1.25.11
vine==5.0.0
wcwidth==0.2.5
Werkzeug==1.0.1
zipp==3.4.0
zope.event==4.5.0
zope.interface==5.3.0

</code></pre></div>
<p dir="auto"></p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
N/A
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>

<details>
<p dir="auto">
</p><p dir="auto">Custom consumer example from <a href="https://docs.celeryproject.org/en/latest/userguide/extending.html" rel="nofollow">the docs</a>, modified to send many messages instead of a single one</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from celery import Celery
from celery import bootsteps
from kombu import Consumer, Exchange, Queue

my_queue = Queue('custom', Exchange('custom'), 'routing_key')

app = Celery(broker='amqp://')


class MyConsumerStep(bootsteps.ConsumerStep):

    def get_consumers(self, channel):
        return [Consumer(channel,
                         queues=[my_queue],
                         callbacks=[self.handle_message],
                         accept=['json'])]

    def handle_message(self, body, message):
        print('Received message: {0!r}'.format(body))
        message.ack()
app.steps['consumer'].add(MyConsumerStep)

def send_me_a_message(who, producer=None):
    with app.producer_or_acquire(producer) as producer:
        producer.publish(
            {'hello': who},
            serializer='json',
            exchange=my_queue.exchange,
            routing_key='routing_key',
            declare=[my_queue],
            retry=True,
        )

if __name__ == '__main__':
    for i in range(1000):
        send_me_a_message('world!')
"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">celery</span> <span class="pl-k">import</span> <span class="pl-v">Celery</span>
<span class="pl-k">from</span> <span class="pl-s1">celery</span> <span class="pl-k">import</span> <span class="pl-s1">bootsteps</span>
<span class="pl-k">from</span> <span class="pl-s1">kombu</span> <span class="pl-k">import</span> <span class="pl-v">Consumer</span>, <span class="pl-v">Exchange</span>, <span class="pl-v">Queue</span>

<span class="pl-s1">my_queue</span> <span class="pl-c1">=</span> <span class="pl-v">Queue</span>(<span class="pl-s">'custom'</span>, <span class="pl-v">Exchange</span>(<span class="pl-s">'custom'</span>), <span class="pl-s">'routing_key'</span>)

<span class="pl-s1">app</span> <span class="pl-c1">=</span> <span class="pl-v">Celery</span>(<span class="pl-s1">broker</span><span class="pl-c1">=</span><span class="pl-s">'amqp://'</span>)


<span class="pl-k">class</span> <span class="pl-v">MyConsumerStep</span>(<span class="pl-s1">bootsteps</span>.<span class="pl-v">ConsumerStep</span>):

    <span class="pl-k">def</span> <span class="pl-en">get_consumers</span>(<span class="pl-s1">self</span>, <span class="pl-s1">channel</span>):
        <span class="pl-k">return</span> [<span class="pl-v">Consumer</span>(<span class="pl-s1">channel</span>,
                         <span class="pl-s1">queues</span><span class="pl-c1">=</span>[<span class="pl-s1">my_queue</span>],
                         <span class="pl-s1">callbacks</span><span class="pl-c1">=</span>[<span class="pl-s1">self</span>.<span class="pl-s1">handle_message</span>],
                         <span class="pl-s1">accept</span><span class="pl-c1">=</span>[<span class="pl-s">'json'</span>])]

    <span class="pl-k">def</span> <span class="pl-en">handle_message</span>(<span class="pl-s1">self</span>, <span class="pl-s1">body</span>, <span class="pl-s1">message</span>):
        <span class="pl-en">print</span>(<span class="pl-s">'Received message: {0!r}'</span>.<span class="pl-en">format</span>(<span class="pl-s1">body</span>))
        <span class="pl-s1">message</span>.<span class="pl-en">ack</span>()
<span class="pl-s1">app</span>.<span class="pl-s1">steps</span>[<span class="pl-s">'consumer'</span>].<span class="pl-en">add</span>(<span class="pl-v">MyConsumerStep</span>)

<span class="pl-k">def</span> <span class="pl-en">send_me_a_message</span>(<span class="pl-s1">who</span>, <span class="pl-s1">producer</span><span class="pl-c1">=</span><span class="pl-c1">None</span>):
    <span class="pl-k">with</span> <span class="pl-s1">app</span>.<span class="pl-en">producer_or_acquire</span>(<span class="pl-s1">producer</span>) <span class="pl-k">as</span> <span class="pl-s1">producer</span>:
        <span class="pl-s1">producer</span>.<span class="pl-en">publish</span>(
            {<span class="pl-s">'hello'</span>: <span class="pl-s1">who</span>},
            <span class="pl-s1">serializer</span><span class="pl-c1">=</span><span class="pl-s">'json'</span>,
            <span class="pl-s1">exchange</span><span class="pl-c1">=</span><span class="pl-s1">my_queue</span>.<span class="pl-s1">exchange</span>,
            <span class="pl-s1">routing_key</span><span class="pl-c1">=</span><span class="pl-s">'routing_key'</span>,
            <span class="pl-s1">declare</span><span class="pl-c1">=</span>[<span class="pl-s1">my_queue</span>],
            <span class="pl-s1">retry</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
        )

<span class="pl-k">if</span> <span class="pl-s1">__name__</span> <span class="pl-c1">==</span> <span class="pl-s">'__main__'</span>:
    <span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">1000</span>):
        <span class="pl-en">send_me_a_message</span>(<span class="pl-s">'world!'</span>)</pre></div>
<p dir="auto">Worker process started via:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="celery -A app.messaging.celery worker -l INFO -E \
  --without-heartbeat --without-gossip --without-mingle \
  --concurrency=4 
  --pool=prefork"><pre class="notranslate"><code class="notranslate">celery -A app.messaging.celery worker -l INFO -E \
  --without-heartbeat --without-gossip --without-mingle \
  --concurrency=4 
  --pool=prefork
</code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>

<p dir="auto">With concurrency set to the default of 4 and the default <code class="notranslate">prefork</code> execution pool, I would expect that the worker process's children all start executing the custom consumer and do work instead of staying idle.</p>
<h1 dir="auto">Actual Behavior</h1>

<p dir="auto">Worker process parent does work, while children processes remain idle. See screenshot of <code class="notranslate">watch ps auxef</code> below, and note that the child processes are idle while the parent process shows CPU usage when 1000 messages are being processed.</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/6099687/113313215-08e95500-92d9-11eb-9f33-3dfcbe03c364.png"><img width="1142" alt="Screen Shot 2021-03-31 at 9 20 41 AM" src="https://user-images.githubusercontent.com/6099687/113313215-08e95500-92d9-11eb-9f33-3dfcbe03c364.png" style="max-width: 100%;"></a></p>
<p dir="auto">Affects 3.0.23.</p>
<p dir="auto">The get_scheduler method is called as part of celery.apps.beat.Beat.startup_info (instead of accessing the beat.scheduler property).  This instantiates a scheduler object (lazy=True).</p>
<p dir="auto">Later, in celery.beat.Service.start() the beat.scheduler property is accessed.  Since this is the first call, the scheduler instance is created again (lazy=False).</p>
<p dir="auto">Is this working as designed?</p>