<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Enhancement%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical enhancement to an existing feature.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22Issue+Type%3A+Enhancement%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed enhancements.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the if the same enhancement was already implemented in the<br>
master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues in this issue<br>
(If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="572298081" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/5974" data-hovercard-type="pull_request" data-hovercard-url="/celery/celery/pull/5974/hovercard" href="https://github.com/celery/celery/pull/5974">#5974</a></li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h1 dir="auto">Brief Summary</h1>

<p dir="auto">When it's used asynchronous backend as Redis with pubsub, result of short task (0.1s length, for example) available immediate.</p>
<p dir="auto">Once i add gevent, result of the same short task became available after 1s, 10 times slower!</p>
<p dir="auto">I will clarify that in both cases I used identical separate worker runed in docker with standard perfork pool.<br>
<code class="notranslate">celery -A tasks worker --prefetch-multiplier=1 --concurrency=10</code></p>
<p dir="auto">This behavior relate with implementation features of <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="572298081" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/5974" data-hovercard-type="pull_request" data-hovercard-url="/celery/celery/pull/5974/hovercard" href="https://github.com/celery/celery/pull/5974">GH-5974</a><br>
AsyncResult.get() in a gevent context has at least 1 second latency, because</p>
<blockquote>
<p dir="auto">I've updated the code to have a specific implementation for gevent and eventlet that will cause wait_for to only return every "timeout" # of seconds, rather than returning immediately</p>
</blockquote>
<p dir="auto">I would like the task result to be available as soon as possible after the worker has submitted it. I think that <code class="notranslate">wait_for</code> have to return immediate after socket operation <code class="notranslate">result_consumer.drain_events</code> occurs, rather than every "timeout" # of seconds.</p>
<p dir="auto">It won't cause overhead, because socket operation still not block event loop, so it let other greenlets executing.</p>
<h1 dir="auto">Design</h1>
<p dir="auto">Now in <code class="notranslate">wait_for</code> it wait (with timeout 1 second) when <code class="notranslate">greenletDrainer.run</code> will finish, of course it's never finish, so it return after 1 second timeout.<br>
If in <code class="notranslate">wait_for</code> wait the end of socket operation <code class="notranslate">_pubsub.get_message</code> instead <code class="notranslate">greenletDrainer.run</code> it will less latency and still not block event loop, so it let other greenlets executing.</p>
<p dir="auto">I can prepare PR with changes in backends.asynchronous greenletDrainer, eventletDrainer, geventDrainer, that every time after <code class="notranslate">result_consumer.drain_events</code> (in greenletDrainer run loop) will send notification for wakeup all greenlets waiting results in <code class="notranslate">wait_for</code>.</p>
<h2 dir="auto">Architectural Considerations</h2>

<p dir="auto">None</p>
<h2 dir="auto">Proposed Behavior</h2>

<p dir="auto">Asynchronous backends (for example Redis) will be same efficient under gevent context as without it.</p>
<h2 dir="auto">Proposed UI/UX</h2>

<h2 dir="auto">Diagrams</h2>

<p dir="auto">N/A</p>
<h2 dir="auto">Alternatives</h2>

<p dir="auto">Another workaround may be to pass <code class="notranslate">interval</code> argument into <code class="notranslate">BaseResultConsumer._wait_for_pending</code> for pass it into <code class="notranslate">drain_events_until</code> and than into <code class="notranslate">wait_for</code>.<br>
In this case it will be possible define own timeout for <code class="notranslate">wait_for</code> interation when execute AsyncResult.get(interval=0.05).</p>
<h2 dir="auto">Howto reproduce</h2>
<ol dir="auto">
<li>run docker with redis and rabbitmq</li>
</ol>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="docker run -d -p 6379:6379 --name redis redis
docker run -d -p 5672:5672 --name rabbit rabbitmq:3"><pre class="notranslate">docker run -d -p 6379:6379 --name redis redis
docker run -d -p 5672:5672 --name rabbit rabbitmq:3</pre></div>
<ol start="2" dir="auto">
<li></li>
</ol>
<p dir="auto">create tasks.py with your ip address for broker and backend</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from celery import Celery
import time

app = Celery('tasks', broker='pyamqp://guest@192.168.1.40//', backend='redis://192.168.1.40')

@app.task(bind=True)
def sleep(self, timeout):
    print(f'task {self.request.id} sleep({timeout}) started')
    time.sleep(timeout)
    print(f'task {self.request.id} sleep({timeout}) finished')
    return f'task {self.request.id} sleep({timeout}) finished'"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">celery</span> <span class="pl-k">import</span> <span class="pl-v">Celery</span>
<span class="pl-k">import</span> <span class="pl-s1">time</span>

<span class="pl-s1">app</span> <span class="pl-c1">=</span> <span class="pl-v">Celery</span>(<span class="pl-s">'tasks'</span>, <span class="pl-s1">broker</span><span class="pl-c1">=</span><span class="pl-s">'pyamqp://guest@192.168.1.40//'</span>, <span class="pl-s1">backend</span><span class="pl-c1">=</span><span class="pl-s">'redis://192.168.1.40'</span>)

<span class="pl-en">@<span class="pl-s1">app</span>.<span class="pl-en">task</span>(<span class="pl-s1">bind</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)</span>
<span class="pl-k">def</span> <span class="pl-en">sleep</span>(<span class="pl-s1">self</span>, <span class="pl-s1">timeout</span>):
    <span class="pl-en">print</span>(<span class="pl-s">f'task <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">self</span>.<span class="pl-s1">request</span>.<span class="pl-s1">id</span><span class="pl-kos">}</span></span> sleep(<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">timeout</span><span class="pl-kos">}</span></span>) started'</span>)
    <span class="pl-s1">time</span>.<span class="pl-en">sleep</span>(<span class="pl-s1">timeout</span>)
    <span class="pl-en">print</span>(<span class="pl-s">f'task <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">self</span>.<span class="pl-s1">request</span>.<span class="pl-s1">id</span><span class="pl-kos">}</span></span> sleep(<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">timeout</span><span class="pl-kos">}</span></span>) finished'</span>)
    <span class="pl-k">return</span> <span class="pl-s">f'task <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">self</span>.<span class="pl-s1">request</span>.<span class="pl-s1">id</span><span class="pl-kos">}</span></span> sleep(<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">timeout</span><span class="pl-kos">}</span></span>) finished'</span></pre></div>
<p dir="auto">create Dockerfile</p>
<div class="highlight highlight-source-dockerfile notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="FROM python:3.8

RUN pip install celery redis

WORKDIR /root
CMD bash"><pre class="notranslate"><span class="pl-k">FROM</span> python:3.8

<span class="pl-k">RUN</span> pip install celery redis

<span class="pl-k">WORKDIR</span> /root
<span class="pl-k">CMD</span> bash</pre></div>
<p dir="auto">build and run worker</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="docker build . -t worker
docker run -it --entrypoint celery -v $(pwd):/root worker -A tasks worker --prefetch-multiplier=1 --concurrency=10 --loglevel DEBUG"><pre class="notranslate">docker build <span class="pl-c1">.</span> -t worker
docker run -it --entrypoint celery -v <span class="pl-s"><span class="pl-pds">$(</span>pwd<span class="pl-pds">)</span></span>:/root worker -A tasks worker --prefetch-multiplier=1 --concurrency=10 --loglevel DEBUG</pre></div>
<p dir="auto">create gevent_example.py</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import gevent
from gevent import monkey; monkey.patch_all()
from datetime import datetime
from tasks import sleep

st = datetime.now()


def test(timeout):
    print(f'{datetime.now() - st} sleep.delay({timeout})')
    a = sleep.delay(timeout)
    gevent.sleep(0)
    res = a.get()
    print(f'{datetime.now()-st} {res}')
    return str(res)


jobs = [gevent.spawn(test, timeout) for timeout in [0.1, 0.1, 0.1, 0.1, 0.1, 0.1]]
gevent.sleep(0)
gevent.joinall(jobs)
print(f'{datetime.now()-st} finish')"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">gevent</span>
<span class="pl-k">from</span> <span class="pl-s1">gevent</span> <span class="pl-k">import</span> <span class="pl-s1">monkey</span>; <span class="pl-s1">monkey</span>.<span class="pl-en">patch_all</span>()
<span class="pl-k">from</span> <span class="pl-s1">datetime</span> <span class="pl-k">import</span> <span class="pl-s1">datetime</span>
<span class="pl-k">from</span> <span class="pl-s1">tasks</span> <span class="pl-k">import</span> <span class="pl-s1">sleep</span>

<span class="pl-s1">st</span> <span class="pl-c1">=</span> <span class="pl-s1">datetime</span>.<span class="pl-en">now</span>()


<span class="pl-k">def</span> <span class="pl-en">test</span>(<span class="pl-s1">timeout</span>):
    <span class="pl-en">print</span>(<span class="pl-s">f'<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">datetime</span>.<span class="pl-en">now</span>() <span class="pl-c1">-</span> <span class="pl-s1">st</span><span class="pl-kos">}</span></span> sleep.delay(<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">timeout</span><span class="pl-kos">}</span></span>)'</span>)
    <span class="pl-s1">a</span> <span class="pl-c1">=</span> <span class="pl-s1">sleep</span>.<span class="pl-en">delay</span>(<span class="pl-s1">timeout</span>)
    <span class="pl-s1">gevent</span>.<span class="pl-en">sleep</span>(<span class="pl-c1">0</span>)
    <span class="pl-s1">res</span> <span class="pl-c1">=</span> <span class="pl-s1">a</span>.<span class="pl-en">get</span>()
    <span class="pl-en">print</span>(<span class="pl-s">f'<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">datetime</span>.<span class="pl-en">now</span>()<span class="pl-c1">-</span><span class="pl-s1">st</span><span class="pl-kos">}</span></span> <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">res</span><span class="pl-kos">}</span></span>'</span>)
    <span class="pl-k">return</span> <span class="pl-en">str</span>(<span class="pl-s1">res</span>)


<span class="pl-s1">jobs</span> <span class="pl-c1">=</span> [<span class="pl-s1">gevent</span>.<span class="pl-en">spawn</span>(<span class="pl-s1">test</span>, <span class="pl-s1">timeout</span>) <span class="pl-k">for</span> <span class="pl-s1">timeout</span> <span class="pl-c1">in</span> [<span class="pl-c1">0.1</span>, <span class="pl-c1">0.1</span>, <span class="pl-c1">0.1</span>, <span class="pl-c1">0.1</span>, <span class="pl-c1">0.1</span>, <span class="pl-c1">0.1</span>]]
<span class="pl-s1">gevent</span>.<span class="pl-en">sleep</span>(<span class="pl-c1">0</span>)
<span class="pl-s1">gevent</span>.<span class="pl-en">joinall</span>(<span class="pl-s1">jobs</span>)
<span class="pl-en">print</span>(<span class="pl-s">f'<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">datetime</span>.<span class="pl-en">now</span>()<span class="pl-c1">-</span><span class="pl-s1">st</span><span class="pl-kos">}</span></span> finish'</span>)</pre></div>
<p dir="auto">now run gevent_example.py</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="0:00:00.015999 sleep.delay(0.1)
0:00:00.055000 sleep.delay(0.1)
0:00:00.055000 sleep.delay(0.1)
0:00:00.055999 sleep.delay(0.1)
0:00:00.055999 sleep.delay(0.1)
0:00:00.056997 sleep.delay(0.1)
0:00:01.226389 task 3a4074c3-294b-4bfc-9139-8ed8724ef564 sleep(0.1) finished
0:00:01.226389 task e69e3232-f925-4a9e-afbc-13add65e5806 sleep(0.1) finished
0:00:01.226389 task 23c40dd4-6083-4d04-aac9-3c23e8508912 sleep(0.1) finished
0:00:01.241475 task df35ca20-8e7d-47b0-9fe8-13fe6b1a7a65 sleep(0.1) finished
0:00:01.241475 task 76bf5b84-fe4b-4791-9686-2dd3018221b0 sleep(0.1) finished
0:00:01.241475 task 2fc0ae38-2521-4371-8caf-88d36ea53eff sleep(0.1) finished
0:00:01.241475 finish"><pre class="notranslate"><code class="notranslate">0:00:00.015999 sleep.delay(0.1)
0:00:00.055000 sleep.delay(0.1)
0:00:00.055000 sleep.delay(0.1)
0:00:00.055999 sleep.delay(0.1)
0:00:00.055999 sleep.delay(0.1)
0:00:00.056997 sleep.delay(0.1)
0:00:01.226389 task 3a4074c3-294b-4bfc-9139-8ed8724ef564 sleep(0.1) finished
0:00:01.226389 task e69e3232-f925-4a9e-afbc-13add65e5806 sleep(0.1) finished
0:00:01.226389 task 23c40dd4-6083-4d04-aac9-3c23e8508912 sleep(0.1) finished
0:00:01.241475 task df35ca20-8e7d-47b0-9fe8-13fe6b1a7a65 sleep(0.1) finished
0:00:01.241475 task 76bf5b84-fe4b-4791-9686-2dd3018221b0 sleep(0.1) finished
0:00:01.241475 task 2fc0ae38-2521-4371-8caf-88d36ea53eff sleep(0.1) finished
0:00:01.241475 finish
</code></pre></div>
<p dir="auto">It is seen that first task with length 0.1s has completed only after ~1.23s (0:00:01.226389)<br>
But if comment line with monkey patching:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# from gevent import monkey; monkey.patch_all()"><pre class="notranslate"><span class="pl-c"># from gevent import monkey; monkey.patch_all()</span></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="0:00:00.015999 sleep.delay(0.1)
0:00:00.226367 sleep.delay(0.1)
0:00:00.228763 sleep.delay(0.1)
0:00:00.231746 sleep.delay(0.1)
0:00:00.233513 sleep.delay(0.1)
0:00:00.234733 sleep.delay(0.1)
0:00:00.338647 task 38f76488-d956-4671-be14-e43df9a593aa sleep(0.1) finished
0:00:00.349064 task 5222feed-6a49-4a55-8a47-f516c3642c30 sleep(0.1) finished
0:00:00.354673 task 87529c14-50a2-4ab3-8e3f-b1f9d7241c42 sleep(0.1) finished
0:00:00.361298 task af68327d-6c37-44e4-9ffa-388ef7c8a8e7 sleep(0.1) finished
0:00:00.363136 task 390980d6-e99d-40b4-afa5-241080ade7a4 sleep(0.1) finished
0:00:00.364271 task f9115418-3175-4caa-98d2-2b39be4ace41 sleep(0.1) finished
0:00:00.364271 finish"><pre class="notranslate"><code class="notranslate">0:00:00.015999 sleep.delay(0.1)
0:00:00.226367 sleep.delay(0.1)
0:00:00.228763 sleep.delay(0.1)
0:00:00.231746 sleep.delay(0.1)
0:00:00.233513 sleep.delay(0.1)
0:00:00.234733 sleep.delay(0.1)
0:00:00.338647 task 38f76488-d956-4671-be14-e43df9a593aa sleep(0.1) finished
0:00:00.349064 task 5222feed-6a49-4a55-8a47-f516c3642c30 sleep(0.1) finished
0:00:00.354673 task 87529c14-50a2-4ab3-8e3f-b1f9d7241c42 sleep(0.1) finished
0:00:00.361298 task af68327d-6c37-44e4-9ffa-388ef7c8a8e7 sleep(0.1) finished
0:00:00.363136 task 390980d6-e99d-40b4-afa5-241080ade7a4 sleep(0.1) finished
0:00:00.364271 task f9115418-3175-4caa-98d2-2b39be4ace41 sleep(0.1) finished
0:00:00.364271 finish
</code></pre></div>
<p dir="auto">It is seen that first task with length 0.1s has completed just after ~0.34s (0:00:00.338647)</p>
<p dir="auto">with proposed changes in backends.asynchronous become:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="0:00:00.016001 sleep.delay(0.1)
0:00:00.054004 sleep.delay(0.1)
0:00:00.055003 sleep.delay(0.1)
0:00:00.055003 sleep.delay(0.1)
0:00:00.056002 sleep.delay(0.1)
0:00:00.056002 sleep.delay(0.1)
0:00:00.322748 task 00a71b72-c12d-40fe-b7e1-7526120c25ad sleep(0.1) finished
0:00:00.325041 task fa31d127-35c9-42f8-a5a4-0d1e29172b59 sleep(0.1) finished
0:00:00.346190 task 3ecb8a4e-78e2-471d-b6ac-e84f9b2c44ac sleep(0.1) finished
0:00:00.349314 task eaa27232-37b1-42ed-87f5-e92c19e87aed sleep(0.1) finished
0:00:00.351648 task 48ed06c8-3683-4353-88e1-f35aeb870332 sleep(0.1) finished
0:00:00.353663 task 8e95e4b7-6988-4b00-b93d-d4240eb38645 sleep(0.1) finished
0:00:00.353663 finish"><pre class="notranslate"><code class="notranslate">0:00:00.016001 sleep.delay(0.1)
0:00:00.054004 sleep.delay(0.1)
0:00:00.055003 sleep.delay(0.1)
0:00:00.055003 sleep.delay(0.1)
0:00:00.056002 sleep.delay(0.1)
0:00:00.056002 sleep.delay(0.1)
0:00:00.322748 task 00a71b72-c12d-40fe-b7e1-7526120c25ad sleep(0.1) finished
0:00:00.325041 task fa31d127-35c9-42f8-a5a4-0d1e29172b59 sleep(0.1) finished
0:00:00.346190 task 3ecb8a4e-78e2-471d-b6ac-e84f9b2c44ac sleep(0.1) finished
0:00:00.349314 task eaa27232-37b1-42ed-87f5-e92c19e87aed sleep(0.1) finished
0:00:00.351648 task 48ed06c8-3683-4353-88e1-f35aeb870332 sleep(0.1) finished
0:00:00.353663 task 8e95e4b7-6988-4b00-b93d-d4240eb38645 sleep(0.1) finished
0:00:00.353663 finish
</code></pre></div>
<p dir="auto">It is seen that first task with length 0.1s has completed just after ~0.32s (0:00:00.322748)</p>
<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?utf8=%E2%9C%93&amp;q=is%3Aissue+label%3A%22Category%3A+Documentation%22+">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues in this issue<br>
(If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h1 dir="auto">Description</h1>

<p dir="auto">The <a href="https://docs.celeryq.dev/en/stable/userguide/calling.html#serializers" rel="nofollow">serialization docs</a> include a section on msgpack:</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/25310870/198069515-4379228e-b743-453f-b1e3-0c4ae4b60da3.png"><img src="https://user-images.githubusercontent.com/25310870/198069515-4379228e-b743-453f-b1e3-0c4ae4b60da3.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto">Specifically I'm referring to</p>
<blockquote>
<p dir="auto">It’s very young however, and support should be considered experimental at this point.</p>
</blockquote>
<p dir="auto">It seems this sentence was added 12 years back in <a href="https://github.com/celery/celery/commit/c9a0fe455593da228d80ebedb3273656408b66f9">this commit</a>, which seems to have been around version 2.2.0.</p>
<h1 dir="auto">Suggestions</h1>

<p dir="auto">I think it'd be good to revise the docs. I don't think this is accurate any longer. Do you agree? <g-emoji class="g-emoji" alias="slightly_smiling_face" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f642.png">🙂</g-emoji></p>