<p dir="auto">The type of traceback is always string, and children is list.<br>
And An error occurred when decode it.<br>
It tried to set serializer to json or pickle, and Their results are the same.<br>
Thanks.</p>
<p dir="auto">$ celery --version<br>
5.0.2 (singularity)</p>
<hr>
<p dir="auto">Related Issues and Possible Duplicates<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="708774400" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/6364" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/6364/hovercard" href="https://github.com/celery/celery/issues/6364">#6364</a></p>
<blockquote>
<p dir="auto">File "msgpack_unpacker.pyx", line 178, in msgpack._cmsgpack.unpackb<br>
File "msgpack_unpacker.pyx", line 126, in msgpack._cmsgpack.get_data_from_buffer<br>
kombu.exceptions.DecodeError: a bytes-like object is required, not 'str'</p>
</blockquote>
<p dir="auto"></p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/celery/celery/blob/60ba37900a038420aec0fc76e60c55989f66c718/celery/backends/mongodb.py#L205">celery/celery/backends/mongodb.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 205
      in
      <a data-pjax="true" class="commit-tease-sha" href="/celery/celery/commit/60ba37900a038420aec0fc76e60c55989f66c718">60ba379</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L205" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="205"></td>
          <td id="LC205" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s">'traceback'</span>: <span class="pl-s1">self</span>.<span class="pl-en">decode</span>(<span class="pl-s1">obj</span>[<span class="pl-s">'traceback'</span>]), </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="            return self.meta_from_decoded({
                'task_id': obj['_id'],
                'status': obj['status'],
                'result': self.decode(obj['result']),
                'date_done': obj['date_done'],
                'traceback': self.decode(obj['traceback']),
                'children': self.decode(obj['children']),
            })"><pre class="notranslate"><code class="notranslate">            return self.meta_from_decoded({
                'task_id': obj['_id'],
                'status': obj['status'],
                'result': self.decode(obj['result']),
                'date_done': obj['date_done'],
                'traceback': self.decode(obj['traceback']),
                'children': self.decode(obj['children']),
            })
</code></pre></div>
<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> This has already been asked to the <a href="https://github.com/celery/celery/discussions">discussions forum</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="823132644" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/6661" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/6661/hovercard" href="https://github.com/celery/celery/issues/6661">#6661</a></p>
<h4 dir="auto">Possible Duplicates</h4>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="823132644" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/6661" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/6661/hovercard" href="https://github.com/celery/celery/issues/6661">#6661</a></p>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>: 5.2.5</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><p dir="auto">software -&gt; celery:5.2.5 (dawn-chorus) kombu:5.2.4 py:3.8.13<br>
billiard:3.6.4.0 py-amqp:5.1.0<br>
platform -&gt; system:Linux arch:64bit<br>
kernel version:5.4.0-1069-aws imp:CPython<br>
loader   -&gt; celery.loaders.default.Loader<br>
settings -&gt; transport:amqp results:disabled</p>
<p dir="auto">deprecated_settings: None</p>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: 3.8</li>
<li><strong>Minimal Celery Version</strong>: 5.2.3</li>
<li><strong>Minimal Kombu Version</strong>: 5.2.4</li>
<li><strong>Minimal Broker Version</strong>: Redis</li>
<li><strong>Minimal Result Backend Version</strong>: N/A</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: Debian GNU/Linux 10 (buster)</li>
<li><strong>Minimal Broker Client Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
</p><p dir="auto">alembic==1.7.7<br>
amqp==5.1.0<br>
atomicwrites==1.3.0<br>
attrs==19.3.0<br>
baseplate==2.1.1<br>
billiard==3.6.4.0<br>
boto3==1.21.35<br>
botocore==1.24.38<br>
celery==5.2.5<br>
certifi==2020.12.5<br>
cffi==1.14.5<br>
chardet==4.0.0<br>
click==8.1.2<br>
click-didyoumean==0.3.0<br>
click-plugins==1.1.1<br>
click-repl==0.2.0<br>
coverage==5.0<br>
cryptography==3.3.2<br>
gevent==21.1.2<br>
greenlet==1.0.0<br>
hiredis==2.0.0<br>
idna==2.10<br>
importlib-metadata==1.7.0<br>
importlib-resources==5.6.0<br>
jmespath==1.0.0<br>
kombu==5.2.4<br>
Mako==1.2.0<br>
MarkupSafe==2.1.1<br>
more-itertools==7.2.0<br>
mypy==0.910<br>
mypy-extensions==0.4.3<br>
packaging==19.1<br>
pluggy==0.12.0<br>
posix-ipc==1.0.5</p>
<h1 dir="auto">Editable install with no version control (product-catalog==0.0.0)</h1>
<p dir="auto">-e /src<br>
prompt-toolkit==3.0.29<br>
psycogreen==1.0.1<br>
psycopg2==2.8.5<br>
py==1.10.0<br>
pycparser==2.20<br>
PyJWT==2.0.1<br>
pyparsing==2.4.6<br>
pytest==5.1.2<br>
pytest-cov==2.6.1<br>
pytest-random-order==1.0.4<br>
python-dateutil==2.8.2<br>
python-json-logger==2.0.1<br>
pytz==2022.1<br>
reddit-edgecontext==1.0.0<br>
redis==3.5.3<br>
requests==2.25.1<br>
s3transfer==0.5.2<br>
sentry-sdk==0.20.3<br>
simpleflake==0.1.5<br>
six==1.15.0<br>
SQLAlchemy==1.3.19<br>
thrift-unofficial==0.14.1<br>
toml==0.10.2<br>
typed-ast==1.4.1<br>
typing-extensions==3.7.4.2<br>
urllib3==1.26.5<br>
vine==5.0.0<br>
wcwidth==0.1.7<br>
zipp==3.1.0<br>
zope.event==4.5.0<br>
zope.interface==5.2.0</p>
<p dir="auto"></p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
N/A
</p>
</details>
<h3 dir="auto">Expected Behavior</h3>

<p dir="auto">.delay  sending task method from app's docker container to worker's container produces connection error,<br>
however both containers are deployed correctly and working.</p>
<h3 dir="auto">Actual Behavior</h3>

<p dir="auto">I have following code reimplementing original Celery and Task classes in separate <code class="notranslate">app.py</code>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="def short_task_name(task_name: str) -&gt; str:
    return task_name.split(&quot;.&quot;)[-1]


def emit_celery_metric(context: baseplate.RequestContext, task_name: str, **tags: Any) -&gt; None:
    tags = {&quot;task_name&quot;: short_task_name(task_name), **tags}
    context.metrics.counter(&quot;celery_task&quot;, tags=tags).increment()
    context.metrics.flush()


class BaseplateTask(celery.Task):
    # this lets us get away with injecting the context task arg
    typing = False

    def __call__(self, *args, **kwargs) -&gt; None:
        baseplate = self._app._baseplate
        context = baseplate.make_context_object()

        with baseplate.make_server_span(context, f&quot;celery.{short_task_name(self.name)}&quot;):
            try:
                ret = super().__call__(context, *args, **kwargs)
            except Exception:
                emit_celery_metric(context, self.name, success=False)
                raise
            emit_celery_metric(context, self.name, success=True)

        return ret


class BaseplateCelery(celery.Celery):
    def __init__(self, *args, **kwargs) -&gt; None:
        self._baseplate = None
        super().__init__(*args, **kwargs)

    def set_baseplatel(self, baseplate, config) -&gt; None:
        self._baseplate = baseplate
        self.conf.update(broker_url=config['broker_url'])

    def run_workers(self, config: baseplate.lib.config.RawConfig) -&gt; None:
        &quot;&quot;&quot;Start the worker, sets up context for future use in workers.&quot;&quot;&quot;
        logger.info(&quot;Starting celery workers&quot;)

        self.start(config[&quot;celery_worker_command&quot;].split(&quot; &quot;))


celery = BaseplateCelery(&quot;product_catalog&quot;)  # I can't pass broker url directly here, because I have to take it from config and set later
celery.Task = BaseplateTask"><pre class="notranslate"><code class="notranslate">def short_task_name(task_name: str) -&gt; str:
    return task_name.split(".")[-1]


def emit_celery_metric(context: baseplate.RequestContext, task_name: str, **tags: Any) -&gt; None:
    tags = {"task_name": short_task_name(task_name), **tags}
    context.metrics.counter("celery_task", tags=tags).increment()
    context.metrics.flush()


class BaseplateTask(celery.Task):
    # this lets us get away with injecting the context task arg
    typing = False

    def __call__(self, *args, **kwargs) -&gt; None:
        baseplate = self._app._baseplate
        context = baseplate.make_context_object()

        with baseplate.make_server_span(context, f"celery.{short_task_name(self.name)}"):
            try:
                ret = super().__call__(context, *args, **kwargs)
            except Exception:
                emit_celery_metric(context, self.name, success=False)
                raise
            emit_celery_metric(context, self.name, success=True)

        return ret


class BaseplateCelery(celery.Celery):
    def __init__(self, *args, **kwargs) -&gt; None:
        self._baseplate = None
        super().__init__(*args, **kwargs)

    def set_baseplatel(self, baseplate, config) -&gt; None:
        self._baseplate = baseplate
        self.conf.update(broker_url=config['broker_url'])

    def run_workers(self, config: baseplate.lib.config.RawConfig) -&gt; None:
        """Start the worker, sets up context for future use in workers."""
        logger.info("Starting celery workers")

        self.start(config["celery_worker_command"].split(" "))


celery = BaseplateCelery("product_catalog")  # I can't pass broker url directly here, because I have to take it from config and set later
celery.Task = BaseplateTask
</code></pre></div>
<p dir="auto">Here enough just a simple task to reproduce the error case in <code class="notranslate">tasks.py</code>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from . import app


@app.celery.task
def test(context):
     print(type(context))
"><pre class="notranslate"><code class="notranslate">from . import app


@app.celery.task
def test(context):
     print(type(context))

</code></pre></div>
<p dir="auto">Just a func in call.py to get caledl from the shell:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from . import tasks


def foo():
    tasks.test.get(context)   # context here will be passed in reimplemented Task class"><pre class="notranslate"><code class="notranslate">from . import tasks


def foo():
    tasks.test.get(context)   # context here will be passed in reimplemented Task class
</code></pre></div>
<p dir="auto">Then main function in <code class="notranslate">__main__.py</code> is for initialization:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="
import baseplate

from . import app
from . import command


def main(): 
    baseplate = baseplate.Baseplate()
    app.celery.set_baseplate(baseplate, config={'broker_url': 'redis://redis:6379'})  # broker url is correct, works
    app.celery.run_workers(command.config)"><pre class="notranslate"><code class="notranslate">
import baseplate

from . import app
from . import command


def main(): 
    baseplate = baseplate.Baseplate()
    app.celery.set_baseplate(baseplate, config={'broker_url': 'redis://redis:6379'})  # broker url is correct, works
    app.celery.run_workers(command.config)
</code></pre></div>
<p dir="auto">After both containers get deployed, I can reproduces the behavior via shell:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from call import foo
foo()"><pre class="notranslate"><code class="notranslate">from call import foo
foo()
</code></pre></div>
<p dir="auto">It gives me a stack trace:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/kombu/utils/functional.py&quot;, line 30, in __call__
    return self.__value__
AttributeError: 'ChannelPromise' object has no attribute '__value__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/amqp/transport.py&quot;, line 188, in _connect
    entries = socket.getaddrinfo(
  File &quot;/usr/local/lib/python3.8/socket.py&quot;, line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -9] Address family for hostname not supported

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py&quot;, line 446, in _reraise_as_library_errors
    yield
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py&quot;, line 433, in _ensure_connection
    return retry_over_time(
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/kombu/utils/functional.py&quot;, line 312, in retry_over_time
    return fun(*args, **kwargs)
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py&quot;, line 877, in _connection_factory
    self._connection = self._establish_connection()
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py&quot;, line 812, in _establish_connection
    conn = self.transport.establish_connection()
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/kombu/transport/pyamqp.py&quot;, line 201, in establish_connection
    conn.connect()
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/amqp/connection.py&quot;, line 323, in connect
    self.transport.connect()
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/amqp/transport.py&quot;, line 129, in connect
    self._connect(self.host, self.port, self.connect_timeout)
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/amqp/transport.py&quot;, line 197, in _connect
    raise (e
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/amqp/transport.py&quot;, line 213, in _connect
    self.sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
  File &quot;/src/product_catalog/__main__&quot;, line 81, in main
     tasks.foo.delay()
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/celery/app/task.py&quot;, line 425, in delay
    return self.apply_async(args, kwargs)
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/celery/app/task.py&quot;, line 575, in apply_async
    return app.send_task(
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/celery/app/base.py&quot;, line 788, in send_task
    amqp.send_task_message(P, name, message, **options)
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/celery/app/amqp.py&quot;, line 510, in send_task_message
    ret = producer.publish(
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/kombu/messaging.py&quot;, line 177, in publish
    return _publish(
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py&quot;, line 523, in _ensured
    return fun(*args, **kwargs)
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/kombu/messaging.py&quot;, line 186, in _publish
    channel = self.channel
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/kombu/messaging.py&quot;, line 209, in _get_channel
    channel = self._channel = channel()
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/kombu/utils/functional.py&quot;, line 32, in __call__
    value = self.__value__ = self.__contract__()
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/kombu/messaging.py&quot;, line 225, in &lt;lambda&gt;
    channel = ChannelPromise(lambda: connection.default_channel)
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py&quot;, line 895, in default_channel
    self._ensure_connection(**conn_opts)
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py&quot;, line 433, in _ensure_connection
    return retry_over_time(
  File &quot;/usr/local/lib/python3.8/contextlib.py&quot;, line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File &quot;/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py&quot;, line 450, in _reraise_as_library_errors
    raise ConnectionError(str(exc)) from exc
kombu.exceptions.OperationalError: [Errno 111] Connection refused"><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):
  File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/utils/functional.py", line 30, in __call__
    return self.__value__
AttributeError: 'ChannelPromise' object has no attribute '__value__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/venv/lib/python3.8/site-packages/amqp/transport.py", line 188, in _connect
    entries = socket.getaddrinfo(
  File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -9] Address family for hostname not supported

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py", line 446, in _reraise_as_library_errors
    yield
  File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py", line 433, in _ensure_connection
    return retry_over_time(
  File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/utils/functional.py", line 312, in retry_over_time
    return fun(*args, **kwargs)
  File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py", line 877, in _connection_factory
    self._connection = self._establish_connection()
  File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py", line 812, in _establish_connection
    conn = self.transport.establish_connection()
  File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/transport/pyamqp.py", line 201, in establish_connection
    conn.connect()
  File "/usr/local/lib/venv/lib/python3.8/site-packages/amqp/connection.py", line 323, in connect
    self.transport.connect()
  File "/usr/local/lib/venv/lib/python3.8/site-packages/amqp/transport.py", line 129, in connect
    self._connect(self.host, self.port, self.connect_timeout)
  File "/usr/local/lib/venv/lib/python3.8/site-packages/amqp/transport.py", line 197, in _connect
    raise (e
  File "/usr/local/lib/venv/lib/python3.8/site-packages/amqp/transport.py", line 213, in _connect
    self.sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/src/product_catalog/__main__", line 81, in main
     tasks.foo.delay()
  File "/usr/local/lib/venv/lib/python3.8/site-packages/celery/app/task.py", line 425, in delay
    return self.apply_async(args, kwargs)
  File "/usr/local/lib/venv/lib/python3.8/site-packages/celery/app/task.py", line 575, in apply_async
    return app.send_task(
  File "/usr/local/lib/venv/lib/python3.8/site-packages/celery/app/base.py", line 788, in send_task
    amqp.send_task_message(P, name, message, **options)
  File "/usr/local/lib/venv/lib/python3.8/site-packages/celery/app/amqp.py", line 510, in send_task_message
    ret = producer.publish(
  File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/messaging.py", line 177, in publish
    return _publish(
  File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py", line 523, in _ensured
    return fun(*args, **kwargs)
  File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/messaging.py", line 186, in _publish
    channel = self.channel
  File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/messaging.py", line 209, in _get_channel
    channel = self._channel = channel()
  File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/utils/functional.py", line 32, in __call__
    value = self.__value__ = self.__contract__()
  File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/messaging.py", line 225, in &lt;lambda&gt;
    channel = ChannelPromise(lambda: connection.default_channel)
  File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py", line 895, in default_channel
    self._ensure_connection(**conn_opts)
  File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py", line 433, in _ensure_connection
    return retry_over_time(
  File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py", line 450, in _reraise_as_library_errors
    raise ConnectionError(str(exc)) from exc
kombu.exceptions.OperationalError: [Errno 111] Connection refused
</code></pre></div>
<p dir="auto">That is very strange. Because celery is running in container:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" -------------- celery@rcelery-workers-7d4d5bd5c8-4slh5 v5.2.5 (dawn-chorus)
--- ***** ----- 
-- ******* ---- Linux-5.4.0-1069-aws-x86_64-with-glibc2.2.5 2022-04-12 11:56:16
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .&gt; app:         __main__:0x7f0df9a15a30
- ** ---------- .&gt; transport:   redis://redis:6379/0
- ** ---------- .&gt; results:     disabled://
- *** --- * --- .&gt; concurrency: 1 (gevent)
-- ******* ---- .&gt; task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .&gt; celery           exchange=celery(direct) key=celery
                

[tasks]
  . product_catalog.celery.tasks.test

[2022-04-12 11:56:16,208: INFO/MainProcess] Connected to redis://redis:6379/0
[2022-04-12 11:56:16,211: INFO/MainProcess] mingle: searching for neighbors
[2022-04-12 11:56:17,226: INFO/MainProcess] mingle: sync with 1 nodes
[2022-04-12 11:56:17,226: INFO/MainProcess] mingle: sync complete
[2022-04-12 11:56:17,234: INFO/MainProcess] pidbox: Connected to redis://redis:6379/0.
[2022-04-12 11:56:17,241: INFO/MainProcess] celery@celery-workers-7d4d5bd5c8-4slh5 ready.
[2022-04-12 11:56:17,632: WARNING/MainProcess] Substantial drift from celery@ip-192-168-0-16.ec2.internal may mean clocks are out of sync.  Current drift is 21600 seconds.  [orig: 2022-04-12 11:56:17.632010 recv: 2022-04-12 17:56:17.339628]
[2022-04-12 11:56:52,239: INFO/MainProcess] missed heartbeat from celery@celery-workers-6f67f5499b-vp22v"><pre class="notranslate"><code class="notranslate"> -------------- celery@rcelery-workers-7d4d5bd5c8-4slh5 v5.2.5 (dawn-chorus)
--- ***** ----- 
-- ******* ---- Linux-5.4.0-1069-aws-x86_64-with-glibc2.2.5 2022-04-12 11:56:16
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .&gt; app:         __main__:0x7f0df9a15a30
- ** ---------- .&gt; transport:   redis://redis:6379/0
- ** ---------- .&gt; results:     disabled://
- *** --- * --- .&gt; concurrency: 1 (gevent)
-- ******* ---- .&gt; task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .&gt; celery           exchange=celery(direct) key=celery
                

[tasks]
  . product_catalog.celery.tasks.test

[2022-04-12 11:56:16,208: INFO/MainProcess] Connected to redis://redis:6379/0
[2022-04-12 11:56:16,211: INFO/MainProcess] mingle: searching for neighbors
[2022-04-12 11:56:17,226: INFO/MainProcess] mingle: sync with 1 nodes
[2022-04-12 11:56:17,226: INFO/MainProcess] mingle: sync complete
[2022-04-12 11:56:17,234: INFO/MainProcess] pidbox: Connected to redis://redis:6379/0.
[2022-04-12 11:56:17,241: INFO/MainProcess] celery@celery-workers-7d4d5bd5c8-4slh5 ready.
[2022-04-12 11:56:17,632: WARNING/MainProcess] Substantial drift from celery@ip-192-168-0-16.ec2.internal may mean clocks are out of sync.  Current drift is 21600 seconds.  [orig: 2022-04-12 11:56:17.632010 recv: 2022-04-12 17:56:17.339628]
[2022-04-12 11:56:52,239: INFO/MainProcess] missed heartbeat from celery@celery-workers-6f67f5499b-vp22v
</code></pre></div>
<p dir="auto">As I already commented in code, I can't pass broker url to <strong>init</strong> constructor due to some restrictions from top.<br>
But if it was passed to constructor without setting afterwards in <code class="notranslate">self.conf</code>, then everything works well and task will be sent to worker and processed properly! But I have to initialize the app first, and then set broker url.</p>
<p dir="auto">I suspect that problem is in decorator, because when I initialize celery app without broker set, this task gets wrapped with default broker address to use. That's why <code class="notranslate">kombu</code> uses the default broker address(not sure).</p>
<p dir="auto">Although I tried to patch <code class="notranslate">BaseplateCelery</code> with additional method to call later:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="def call_super(self):
     super().__init__(main='product_catalog', broker_url='redis://redis:6379/0')"><pre class="notranslate"><code class="notranslate">def call_super(self):
     super().__init__(main='product_catalog', broker_url='redis://redis:6379/0')
</code></pre></div>
<p dir="auto">i see that class state now is if it was initialized in working case, but it still produces the same error!</p>