<p dir="auto">This is not a very common situation, but it's one that we did observe, so I'm opening this ticket to document this potential behavior.  We were relying on active_queues() to provide an accurate look at the workers, and these a the following log messages in the log concerned us.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="py.warnings:WARNING: /home/bmbouter/Documents/qpidkombu/repos/celery/celery/app/control.py:34: DuplicateNodenameWarning: Received multiple replies from node name True.
py.warnings:WARNING: Please make sure you give each node a unique nodename using the `-n` option.
py.warnings:WARNING:   W_DUPNODE.format(dup),
py.warnings:WARNING:"><pre class="notranslate"><code class="notranslate">py.warnings:WARNING: /home/bmbouter/Documents/qpidkombu/repos/celery/celery/app/control.py:34: DuplicateNodenameWarning: Received multiple replies from node name True.
py.warnings:WARNING: Please make sure you give each node a unique nodename using the `-n` option.
py.warnings:WARNING:   W_DUPNODE.format(dup),
py.warnings:WARNING:
</code></pre></div>
<p dir="auto">We saw this output exactly once from each worker some time after each restart.  In our case we had a task that uses inspect, such as:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="controller = control.Control(app=app)

@task
def my_task():
    active_queues = controller.inspect().active_queues()"><pre class="notranslate"><code class="notranslate">controller = control.Control(app=app)

@task
def my_task():
    active_queues = controller.inspect().active_queues()
</code></pre></div>
<p dir="auto">To reproduce, start 8 workers with unique names, and have them ready to run the my_task.  Then dispatch 50 my_task using:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="for i in range(50):
    my_task.delay()"><pre class="notranslate"><code class="notranslate">for i in range(50):
    my_task.delay()
</code></pre></div>
<p dir="auto">We do not have any duplicate workers.  We have exactly 8, and each one gets a different name.  We verified this manually, yet we saw the warnings about duplicate workers.</p>
<p dir="auto">When turning on debug logging, I saw a many output lines like this one:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kombu.pidbox:DEBUG: pidbox received method active_queues() [reply_to:{u'routing_key': u'37ff7c3e-8007-38c4-a836-544cfe8380e6', u'exchange': u'reply.celery.pidbox'} ticket:6f2aa08e-68bc-4750-849c-7f02b98da7b6"><pre class="notranslate"><code class="notranslate">kombu.pidbox:DEBUG: pidbox received method active_queues() [reply_to:{u'routing_key': u'37ff7c3e-8007-38c4-a836-544cfe8380e6', u'exchange': u'reply.celery.pidbox'} ticket:6f2aa08e-68bc-4750-849c-7f02b98da7b6
</code></pre></div>
<p dir="auto">Each pid showed that it received a pidbox request (the active_queues command) with a reply_to routing key of '37ff7c3e-8007-38c4-a836-544cfe8380e6'.  I believe the root cause of this issue is that this routing key does not change with pidbox command to pidbox command.  It also cannot correlate one the responses with the requests.</p>
<p dir="auto">If I issue an active_queue command with reply_to = 'foo', and I wait 1 second to timeout, and 1 worker responds later than 1 second, it's reply is going to go into my queue and not be read by my active_queue command.  Then if I issue a second active_queue() command, and use the same reply_to = 'foo', and the worker responds in time this time, as I process the replies I'll see a duplicate reply from the worker.</p>
<p dir="auto">This causes the active_queues() information to be incorrect.</p>
<p dir="auto">A correlation value could be introduced that correlates replies with requests, or the reply queue could be different with each pidbox command potentially.</p>
<p dir="auto">The fact that the logs were only printed once I is also interesting.  I verified that this actually happens a lot more than once, but the logging warning is only shown once!  That <a href="https://github.com/celery/celery/blob/master/celery/app/control.py#L33">logging occurs here</a>.</p>
<p dir="auto">The python 3.7 jobs of our CI started failing today. To reproduce the issue, <code class="notranslate">pip install celery</code> in a fresh environment and you get this</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="python -c &quot;from celery import current_app&quot;
Traceback (most recent call last):
  File &quot;&lt;string&gt;&quot;, line 1, in &lt;module&gt;
ImportError: cannot import name 'current_app' from 'celery' (/users/denolf/virtualenvs/pybox_c5ZgMc/ubuntu_20_04/py37/lib/python3.7/site-packages/celery/__init__.py)"><pre class="notranslate"><code class="notranslate">python -c "from celery import current_app"
Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
ImportError: cannot import name 'current_app' from 'celery' (/users/denolf/virtualenvs/pybox_c5ZgMc/ubuntu_20_04/py37/lib/python3.7/site-packages/celery/__init__.py)
</code></pre></div>
<p dir="auto">Upon further investigation is seems <code class="notranslate">STATICA_HACK == False</code> despite this</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# ...lib/python3.7/site-packages/celery/__init__.py

# This is never executed, but tricks static analyzers (PyDev, PyCharm,
# pylint, etc.) into knowing the types of these symbols, and what
# they contain.
STATICA_HACK = True
globals()['kcah_acitats'[::-1].upper()] = False
breakpoint()
if STATICA_HACK:  # pragma: no cover
    from celery._state import current_app, current_task
    from celery.app import shared_task
    from celery.app.base import Celery
    from celery.app.task import Task
    from celery.app.utils import bugreport
    from celery.canvas import (chain, chord, chunks, group,  # noqa
                               maybe_signature, signature, subtask, xmap,
                               xstarmap)
    from celery.utils import uuid"><pre class="notranslate"><span class="pl-c"># ...lib/python3.7/site-packages/celery/__init__.py</span>

<span class="pl-c"># This is never executed, but tricks static analyzers (PyDev, PyCharm,</span>
<span class="pl-c"># pylint, etc.) into knowing the types of these symbols, and what</span>
<span class="pl-c"># they contain.</span>
<span class="pl-v">STATICA_HACK</span> <span class="pl-c1">=</span> <span class="pl-c1">True</span>
<span class="pl-en">globals</span>()[<span class="pl-s">'kcah_acitats'</span>[::<span class="pl-c1">-</span><span class="pl-c1">1</span>].<span class="pl-en">upper</span>()] <span class="pl-c1">=</span> <span class="pl-c1">False</span>
<span class="pl-en">breakpoint</span>()
<span class="pl-k">if</span> <span class="pl-v">STATICA_HACK</span>:  <span class="pl-c"># pragma: no cover</span>
    <span class="pl-k">from</span> <span class="pl-s1">celery</span>.<span class="pl-s1">_state</span> <span class="pl-k">import</span> <span class="pl-s1">current_app</span>, <span class="pl-s1">current_task</span>
    <span class="pl-k">from</span> <span class="pl-s1">celery</span>.<span class="pl-s1">app</span> <span class="pl-k">import</span> <span class="pl-s1">shared_task</span>
    <span class="pl-k">from</span> <span class="pl-s1">celery</span>.<span class="pl-s1">app</span>.<span class="pl-s1">base</span> <span class="pl-k">import</span> <span class="pl-v">Celery</span>
    <span class="pl-k">from</span> <span class="pl-s1">celery</span>.<span class="pl-s1">app</span>.<span class="pl-s1">task</span> <span class="pl-k">import</span> <span class="pl-v">Task</span>
    <span class="pl-k">from</span> <span class="pl-s1">celery</span>.<span class="pl-s1">app</span>.<span class="pl-s1">utils</span> <span class="pl-k">import</span> <span class="pl-s1">bugreport</span>
    <span class="pl-k">from</span> <span class="pl-s1">celery</span>.<span class="pl-s1">canvas</span> <span class="pl-k">import</span> (<span class="pl-s1">chain</span>, <span class="pl-s1">chord</span>, <span class="pl-s1">chunks</span>, <span class="pl-s1">group</span>,  <span class="pl-c"># noqa</span>
                               <span class="pl-s1">maybe_signature</span>, <span class="pl-s1">signature</span>, <span class="pl-s1">subtask</span>, <span class="pl-s1">xmap</span>,
                               <span class="pl-s1">xstarmap</span>)
    <span class="pl-k">from</span> <span class="pl-s1">celery</span>.<span class="pl-s1">utils</span> <span class="pl-k">import</span> <span class="pl-s1">uuid</span></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="python --version
Python 3.7.9

pip list
Package            Version
------------------ -------
amqp               5.1.1
billiard           3.6.4.0
cached-property    1.5.2
celery             5.2.7
click              8.1.3
click-didyoumean   0.3.0
click-plugins      1.1.1
click-repl         0.2.0
importlib-metadata 5.0.0
kombu              5.2.4
pip                22.2.2
prompt-toolkit     3.0.31
pytz               2022.4
setuptools         65.4.1
six                1.16.0
typing_extensions  4.3.0
vine               5.0.0
wcwidth            0.2.5
wheel              0.37.1
zipp               3.8.1"><pre class="notranslate"><code class="notranslate">python --version
Python 3.7.9

pip list
Package            Version
------------------ -------
amqp               5.1.1
billiard           3.6.4.0
cached-property    1.5.2
celery             5.2.7
click              8.1.3
click-didyoumean   0.3.0
click-plugins      1.1.1
click-repl         0.2.0
importlib-metadata 5.0.0
kombu              5.2.4
pip                22.2.2
prompt-toolkit     3.0.31
pytz               2022.4
setuptools         65.4.1
six                1.16.0
typing_extensions  4.3.0
vine               5.0.0
wcwidth            0.2.5
wheel              0.37.1
zipp               3.8.1
</code></pre></div>
<p dir="auto">I don't see this issue with python 3.8 (and perhaps higher as well, didn't check)</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="python --version
Python 3.8.5

pip list
Package          Version
---------------- -------
amqp             5.1.1
billiard         3.6.4.0
celery           5.2.7
click            8.1.3
click-didyoumean 0.3.0
click-plugins    1.1.1
click-repl       0.2.0
kombu            5.2.4
pip              22.2.2
prompt-toolkit   3.0.31
pytz             2022.4
setuptools       65.4.1
six              1.16.0
vine             5.0.0
wcwidth          0.2.5
wheel            0.37.1"><pre class="notranslate"><code class="notranslate">python --version
Python 3.8.5

pip list
Package          Version
---------------- -------
amqp             5.1.1
billiard         3.6.4.0
celery           5.2.7
click            8.1.3
click-didyoumean 0.3.0
click-plugins    1.1.1
click-repl       0.2.0
kombu            5.2.4
pip              22.2.2
prompt-toolkit   3.0.31
pytz             2022.4
setuptools       65.4.1
six              1.16.0
vine             5.0.0
wcwidth          0.2.5
wheel            0.37.1
</code></pre></div>