<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Enhancement%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical enhancement to an existing feature.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22Issue+Type%3A+Enhancement%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed enhancements.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the if the same enhancement was already implemented in the<br>
master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues in this issue<br>
(If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h1 dir="auto">Brief Summary</h1>

<p dir="auto">This proposes Celery package to provide a generic <code class="notranslate">delay(func, *args, **kwargs)</code> that wraps any func or method thrown at it.</p>
<h1 dir="auto">Design</h1>
<p dir="auto">Historically, to make a function "able" to be delayed via Celery you need to wrap it in a decorator at least. This can be ok for own code, but not for 3rd-party code. Also makes your code coupled with Celery or at least with Celery API for jobs.</p>
<p dir="auto">"Modern" Python versions can natively pickle functions, so this is no more needed for simpler uses. If Celery provides a shared task that receives a function and its arguments, no custom code is needed for standard user code.</p>
<p dir="auto">Example:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# mymodule.py
import requests
def ping_website(url):
    requests.get(url)

# main.py
from celery.contrib import delay

from .mymodule import ping_website
from .myothermodule import greet

delay(ping_website, 'http://www.google.com.br')
delay(greet)"><pre class="notranslate"><span class="pl-c"># mymodule.py</span>
<span class="pl-k">import</span> <span class="pl-s1">requests</span>
<span class="pl-k">def</span> <span class="pl-en">ping_website</span>(<span class="pl-s1">url</span>):
    <span class="pl-s1">requests</span>.<span class="pl-en">get</span>(<span class="pl-s1">url</span>)

<span class="pl-c"># main.py</span>
<span class="pl-k">from</span> <span class="pl-s1">celery</span>.<span class="pl-s1">contrib</span> <span class="pl-k">import</span> <span class="pl-s1">delay</span>

<span class="pl-k">from</span> .<span class="pl-s1">mymodule</span> <span class="pl-k">import</span> <span class="pl-s1">ping_website</span>
<span class="pl-k">from</span> .<span class="pl-s1">myothermodule</span> <span class="pl-k">import</span> <span class="pl-s1">greet</span>

<span class="pl-en">delay</span>(<span class="pl-s1">ping_website</span>, <span class="pl-s">'http://www.google.com.br'</span>)
<span class="pl-en">delay</span>(<span class="pl-s1">greet</span>)</pre></div>
<p dir="auto">Note how the Celery dependent part occur only on the main module, not on each of function definitions. Even better, as we do not care about the response, such call could be wrote as:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="delay(requests.get, 'http://www.google.com.br')"><pre class="notranslate"><span class="pl-en">delay</span>(<span class="pl-s1">requests</span>.<span class="pl-s1">get</span>, <span class="pl-s">'http://www.google.com.br'</span>)</pre></div>
<p dir="auto">This is not possible with today's prescription of "create a Task to be able to .delay it", you need to import the 3rd-party funcs and code in a task that wraps it to be able to send to Celery.</p>
<h2 dir="auto">Architectural Considerations</h2>

<p dir="auto">Code <code class="notranslate">delay()</code>ed can be considered more convenient. Yet could potentially carry more data than needed or more data than the former way of calling. People can optimize by writing in the former way if needed.</p>
<p dir="auto">Methods carry the whole <code class="notranslate">self</code> when serialized and this should be informed to the user, as they can consider if is acceptable to pay the cost to do so. Also is needed to inform that such feature needs pickle and reproduce the notice about the danger of accepting pickled jobs.</p>
<h2 dir="auto">Proposed Behavior</h2>

<p dir="auto">I propose some code along the following to be included on Celery release and the usage to be featured on Quickstart as the simpler way to sideload your code via Celery</p>
<p dir="auto">Code to be included:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# celery/contrib.py
@shared_task(serializer='pickle')
def _call(func, *args, **kwargs):
    return func(*args, **kwargs)


def delay(func, *args, **kwargs) -&gt; celery.result.AsyncResult:
    &quot;&quot;&quot;
    Defer a call of func(*args, **kwargs) to a Celery worker
    &quot;&quot;&quot;
    return _call.delay(func, *args, **kwargs)"><pre class="notranslate"><span class="pl-c"># celery/contrib.py</span>
<span class="pl-en">@<span class="pl-en">shared_task</span>(<span class="pl-s1">serializer</span><span class="pl-c1">=</span><span class="pl-s">'pickle'</span>)</span>
<span class="pl-k">def</span> <span class="pl-en">_call</span>(<span class="pl-s1">func</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
    <span class="pl-k">return</span> <span class="pl-en">func</span>(<span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)


<span class="pl-k">def</span> <span class="pl-en">delay</span>(<span class="pl-s1">func</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>) <span class="pl-c1">-&gt;</span> <span class="pl-s1">celery</span>.<span class="pl-s1">result</span>.<span class="pl-v">AsyncResult</span>:
    <span class="pl-s">"""</span>
<span class="pl-s">    Defer a call of func(*args, **kwargs) to a Celery worker</span>
<span class="pl-s">    """</span>
    <span class="pl-k">return</span> <span class="pl-s1">_call</span>.<span class="pl-en">delay</span>(<span class="pl-s1">func</span>, <span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)</pre></div>
<p dir="auto">Example to be included in docs:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from celery.contrib import delay

# synchronous version:
my_slow_procedure(3, 4, spam='eggs')

# sideloaded version:
delay(my_slow_procedure, 3, 4, spam='eggs')"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">celery</span>.<span class="pl-s1">contrib</span> <span class="pl-k">import</span> <span class="pl-s1">delay</span>

<span class="pl-c"># synchronous version:</span>
<span class="pl-en">my_slow_procedure</span>(<span class="pl-c1">3</span>, <span class="pl-c1">4</span>, <span class="pl-s1">spam</span><span class="pl-c1">=</span><span class="pl-s">'eggs'</span>)

<span class="pl-c"># sideloaded version:</span>
<span class="pl-en">delay</span>(<span class="pl-s1">my_slow_procedure</span>, <span class="pl-c1">3</span>, <span class="pl-c1">4</span>, <span class="pl-s1">spam</span><span class="pl-c1">=</span><span class="pl-s">'eggs'</span>)</pre></div>
<h2 dir="auto">Proposed UI/UX</h2>

<p dir="auto">The simpler usage is to <code class="notranslate">delay(fn, *args, **kwargs)</code> arbitrary functions/methods.</p>
<p dir="auto">Anything more advanced than this could be considered advanced enough to honor the today's way of do it.</p>
<h2 dir="auto">Diagrams</h2>

<p dir="auto">N/A</p>
<h2 dir="auto">Alternatives</h2>

<p dir="auto">None</p>
<p dir="auto">Hi,<br>
I am trying to update celery from 2.5.5 to 3.0.7, I am using redis as broker and latest kombu.<br>
What I notice is that when I switch the workers to 3.0.7 redis open files count goes up a lot.<br>
I noticed that because it hits the max open file user limit setting (is running on ubuntu) and kills redis when all workers are up and running :(</p>
<p dir="auto">Looking at strace -p redis_pid I see that redis is trying to open /var/log/redis/daemon.log (defined in redis.conf as the logfile value) but fails because of the too many open files limit.</p>
<p dir="auto">Looking at the logfile I see no output when running 1 single worker @3.0.7</p>
<p dir="auto">I did this 3 times, moving the same amount of workers (approx 5 workers with concurrency = 20) and when moving from 2.5.5 to 3.0.1 always produce this situation.</p>
<p dir="auto">I know I can fix this by set higher limits on the redis server but first I would like to know if this is normal behaviour or a side effect of some change.</p>
<p dir="auto">A [maybe] useful note about this problem:<br>
I was one of the user affected by the empty poll bug <a href="https://github.com/celery/celery/issues/940" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/940/hovercard">940</a>,  when running 3.0.6 with same settings and amount of workers as now I did not experienced this problem (and it run for few hours).</p>