<p dir="auto">Hi people,</p>
<p dir="auto">I have 2 Django projects (at different branchs/versions) on the same machine and both use celery for asynchronous tasks.</p>
<p dir="auto">I figure thanks to <strong>sentry</strong> that tasks from app A are sent to workers of app B. Each application runs under a different unix user and workers/wsgi run through <strong>supervisor</strong>.</p>
<p dir="auto">Here the different configurations:</p>
<ul dir="auto">
<li><strong>Application A</strong></li>
<li>Supervisor</li>
</ul>
<div class="highlight highlight-source-ini notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="[program:demo-a-worker]
command=/home/demo_a/start_worker.sh
numprocs=1
directory=/home/demo_a
autostart=true
autorestart=true
user=demo_a
redirect_stderr=true
stdout_logfile=/var/log/demo_a/worker.log
stdout_logfile_maxbytes=1MB
stdout_logfile_backups=10
stdout_capture_maxbytes=1MB"><pre class="notranslate"><span class="pl-en">[program:demo-a-worker]</span>
<span class="pl-k">command</span>=/home/demo_a/start_worker.sh
<span class="pl-k">numprocs</span>=1
<span class="pl-k">directory</span>=/home/demo_a
<span class="pl-k">autostart</span>=true
<span class="pl-k">autorestart</span>=true
<span class="pl-k">user</span>=demo_a
<span class="pl-k">redirect_stderr</span>=true
<span class="pl-k">stdout_logfile</span>=/var/log/demo_a/worker.log
<span class="pl-k">stdout_logfile_maxbytes</span>=1MB
<span class="pl-k">stdout_logfile_backups</span>=10
<span class="pl-k">stdout_capture_maxbytes</span>=1MB</pre></div>
<ul dir="auto">
<li>Script</li>
</ul>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="exec $PYENV/celery worker -A mydjangoproject -B -Q demo_a --loglevel=INFO --hostname demo_a.%h"><pre class="notranslate"><span class="pl-c1">exec</span> <span class="pl-smi">$PYENV</span>/celery worker -A mydjangoproject -B -Q demo_a --loglevel=INFO --hostname demo_a.%h</pre></div>
<ul dir="auto">
<li><strong>Application B</strong></li>
<li>Supervisord</li>
</ul>
<div class="highlight highlight-source-ini notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="[program:demo-b-worker]
command=/home/demo_b/start_worker.sh
numprocs=1
directory=/home/demo_b
autostart=true
autorestart=true
user=demo_b
redirect_stderr=true
stdout_logfile=/var/log/demo_b/worker.log
stdout_logfile_maxbytes=1MB
stdout_logfile_backups=10
stdout_capture_maxbytes=1MB"><pre class="notranslate"><span class="pl-en">[program:demo-b-worker]</span>
<span class="pl-k">command</span>=/home/demo_b/start_worker.sh
<span class="pl-k">numprocs</span>=1
<span class="pl-k">directory</span>=/home/demo_b
<span class="pl-k">autostart</span>=true
<span class="pl-k">autorestart</span>=true
<span class="pl-k">user</span>=demo_b
<span class="pl-k">redirect_stderr</span>=true
<span class="pl-k">stdout_logfile</span>=/var/log/demo_b/worker.log
<span class="pl-k">stdout_logfile_maxbytes</span>=1MB
<span class="pl-k">stdout_logfile_backups</span>=10
<span class="pl-k">stdout_capture_maxbytes</span>=1MB</pre></div>
<ul dir="auto">
<li>Script</li>
</ul>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="exec $PYENV/celery worker -A mydjangoproject -B -Q demo_b --loglevel=INFO --hostname demo_b.%h"><pre class="notranslate"><span class="pl-c1">exec</span> <span class="pl-smi">$PYENV</span>/celery worker -A mydjangoproject -B -Q demo_b --loglevel=INFO --hostname demo_b.%h</pre></div>
<ul dir="auto">
<li>Celery configuration in <code class="notranslate">mydjangoproject</code></li>
</ul>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# Set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mydjangoproject.settings')
os.environ.setdefault(&quot;DJANGO_CONFIGURATION&quot;, &quot;Dev&quot;)

# Load environment configuration
from configurations import importer
importer.install()


class Celery(celery.Celery):
    &quot;&quot;&quot;
    Celery configuration class
    &quot;&quot;&quot;
    def on_configure(self):
        # Get Raven configuration from Django settings
        if not hasattr(settings, 'RAVEN_CONFIG'):
            return
        # Initialize Raven client
        client = raven.Client(**settings.RAVEN_CONFIG)
        # Register a custom filter to filter out duplicate logs
        register_logger_signal(client)
        # Hook into the Celery error handler
        register_signal(client)


# Initialize Celery application
app = Celery('mydjangoproject')

# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')

# Auto-discover celery tasks inside Django applications
from django.conf import settings
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)"><pre class="notranslate"><span class="pl-c"># Set the default Django settings module for the 'celery' program.</span>
<span class="pl-s1">os</span>.<span class="pl-s1">environ</span>.<span class="pl-en">setdefault</span>(<span class="pl-s">'DJANGO_SETTINGS_MODULE'</span>, <span class="pl-s">'mydjangoproject.settings'</span>)
<span class="pl-s1">os</span>.<span class="pl-s1">environ</span>.<span class="pl-en">setdefault</span>(<span class="pl-s">"DJANGO_CONFIGURATION"</span>, <span class="pl-s">"Dev"</span>)

<span class="pl-c"># Load environment configuration</span>
<span class="pl-k">from</span> <span class="pl-s1">configurations</span> <span class="pl-k">import</span> <span class="pl-s1">importer</span>
<span class="pl-s1">importer</span>.<span class="pl-en">install</span>()


<span class="pl-k">class</span> <span class="pl-v">Celery</span>(<span class="pl-s1">celery</span>.<span class="pl-v">Celery</span>):
    <span class="pl-s">"""</span>
<span class="pl-s">    Celery configuration class</span>
<span class="pl-s">    """</span>
    <span class="pl-k">def</span> <span class="pl-en">on_configure</span>(<span class="pl-s1">self</span>):
        <span class="pl-c"># Get Raven configuration from Django settings</span>
        <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-en">hasattr</span>(<span class="pl-s1">settings</span>, <span class="pl-s">'RAVEN_CONFIG'</span>):
            <span class="pl-k">return</span>
        <span class="pl-c"># Initialize Raven client</span>
        <span class="pl-s1">client</span> <span class="pl-c1">=</span> <span class="pl-s1">raven</span>.<span class="pl-v">Client</span>(<span class="pl-c1">**</span><span class="pl-s1">settings</span>.<span class="pl-v">RAVEN_CONFIG</span>)
        <span class="pl-c"># Register a custom filter to filter out duplicate logs</span>
        <span class="pl-en">register_logger_signal</span>(<span class="pl-s1">client</span>)
        <span class="pl-c"># Hook into the Celery error handler</span>
        <span class="pl-en">register_signal</span>(<span class="pl-s1">client</span>)


<span class="pl-c"># Initialize Celery application</span>
<span class="pl-s1">app</span> <span class="pl-c1">=</span> <span class="pl-v">Celery</span>(<span class="pl-s">'mydjangoproject'</span>)

<span class="pl-c"># Using a string here means the worker will not have to</span>
<span class="pl-c"># pickle the object when using Windows.</span>
<span class="pl-s1">app</span>.<span class="pl-en">config_from_object</span>(<span class="pl-s">'django.conf:settings'</span>)

<span class="pl-c"># Auto-discover celery tasks inside Django applications</span>
<span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">conf</span> <span class="pl-k">import</span> <span class="pl-s1">settings</span>
<span class="pl-s1">app</span>.<span class="pl-en">autodiscover_tasks</span>(<span class="pl-k">lambda</span>: <span class="pl-s1">settings</span>.<span class="pl-v">INSTALLED_APPS</span>)</pre></div>
<ul dir="auto">
<li><strong>Settings</strong></li>
</ul>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="    # Celery configuration
    BROKER_URL = values.Value('redis://localhost:6379/0')
    BROKER_TRANSPORT_OPTIONS = {
        'visibility_timeout': 3600,
        'fanout_prefix': True,
        'fanout_patterns': True,
    }
    CELERY_TASK_SERIALIZER = 'pickle'
    CELERY_RESULT_SERIALIZER = 'pickle'
    CELERY_RESULT_BACKEND = values.Value('redis')
    CELERY_TASK_RESULT_EXPIRES = 3600
    CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']
    CELERY_DISABLE_RATE_LIMITS = True
    CELERY_CHORD_PROPAGATES = True
    CELERY_ALWAYS_EAGER = values.BooleanValue(True)
    CELERY_DEFAULT_QUEUE = values.Value('celery', environ_name='QUEUE_NAME')"><pre class="notranslate">    <span class="pl-c"># Celery configuration</span>
    <span class="pl-v">BROKER_URL</span> <span class="pl-c1">=</span> <span class="pl-s1">values</span>.<span class="pl-v">Value</span>(<span class="pl-s">'redis://localhost:6379/0'</span>)
    <span class="pl-v">BROKER_TRANSPORT_OPTIONS</span> <span class="pl-c1">=</span> {
        <span class="pl-s">'visibility_timeout'</span>: <span class="pl-c1">3600</span>,
        <span class="pl-s">'fanout_prefix'</span>: <span class="pl-c1">True</span>,
        <span class="pl-s">'fanout_patterns'</span>: <span class="pl-c1">True</span>,
    }
    <span class="pl-v">CELERY_TASK_SERIALIZER</span> <span class="pl-c1">=</span> <span class="pl-s">'pickle'</span>
    <span class="pl-v">CELERY_RESULT_SERIALIZER</span> <span class="pl-c1">=</span> <span class="pl-s">'pickle'</span>
    <span class="pl-v">CELERY_RESULT_BACKEND</span> <span class="pl-c1">=</span> <span class="pl-s1">values</span>.<span class="pl-v">Value</span>(<span class="pl-s">'redis'</span>)
    <span class="pl-v">CELERY_TASK_RESULT_EXPIRES</span> <span class="pl-c1">=</span> <span class="pl-c1">3600</span>
    <span class="pl-v">CELERY_ACCEPT_CONTENT</span> <span class="pl-c1">=</span> [<span class="pl-s">'pickle'</span>, <span class="pl-s">'json'</span>, <span class="pl-s">'msgpack'</span>, <span class="pl-s">'yaml'</span>]
    <span class="pl-v">CELERY_DISABLE_RATE_LIMITS</span> <span class="pl-c1">=</span> <span class="pl-c1">True</span>
    <span class="pl-v">CELERY_CHORD_PROPAGATES</span> <span class="pl-c1">=</span> <span class="pl-c1">True</span>
    <span class="pl-v">CELERY_ALWAYS_EAGER</span> <span class="pl-c1">=</span> <span class="pl-s1">values</span>.<span class="pl-v">BooleanValue</span>(<span class="pl-c1">True</span>)
    <span class="pl-v">CELERY_DEFAULT_QUEUE</span> <span class="pl-c1">=</span> <span class="pl-s1">values</span>.<span class="pl-v">Value</span>(<span class="pl-s">'celery'</span>, <span class="pl-s1">environ_name</span><span class="pl-c1">=</span><span class="pl-s">'QUEUE_NAME'</span>)</pre></div>
<p dir="auto">Values are environment configuration, in production CELERY_ALWAYS_EAGER=False and queue name is the same as in the command line.</p>
<p dir="auto">I take care that each worker has its own name and queue but it seems not to solve the issue.</p>
<p dir="auto">Thanks!</p>
<h1 dir="auto">Checklist</h1>

<ul dir="auto">
<li>[ x] I have checked the <a href="https://github.com/celery/celery/issues?utf8=%E2%9C%93&amp;q=is%3Aissue+label%3A%22Issue+Type%3A+Feature+Request%22+">issues list</a><br>
for similar or identical feature requests.</li>
<li>[ x] I have checked the <a href="https://github.com/celery/celery/pulls?utf8=%E2%9C%93&amp;q=is%3Apr+label%3A%22PR+Type%3A+Feature%22+">pull requests list</a><br>
for existing proposed implementations of this feature.</li>
<li>[ x] I have checked the <a href="https://github.com/celery/celery/commits/main">commit log</a><br>
to find out if the same feature was already implemented in the<br>
main branch.</li>
<li>[x ] I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h1 dir="auto">Brief Summary</h1>

<p dir="auto">This is related to another issue I filed: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1800454221" data-permission-text="Title is private" data-url="https://github.com/ovh/celery-director/issues/195" data-hovercard-type="issue" data-hovercard-url="/ovh/celery-director/issues/195/hovercard" href="https://github.com/ovh/celery-director/issues/195">ovh/celery-director#195</a>:  In a workflow, a step might need some info/value generated in a grand parent step(not a direct parent step), one solution is enable a step/task to save the info into payload, and then later step could access this payload. I am not good at celery, not sure whether this is possible. But the use case is very usefully.</p>
<h1 dir="auto">Design</h1>
<h2 dir="auto">Architectural Considerations</h2>

<p dir="auto">None</p>
<h2 dir="auto">Proposed Behavior</h2>

<h2 dir="auto">Proposed UI/UX</h2>

<h2 dir="auto">Diagrams</h2>

<p dir="auto">N/A</p>
<h2 dir="auto">Alternatives</h2>

<p dir="auto">None</p>