<h2 dir="auto">Checklist</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
</ul>
<h2 dir="auto">Steps to reproduce</h2>
<p dir="auto"><strong>I am using celery version 3.0.23 and scheduling the tasks using apply_async the issue is that the i encountered that the tasks are being duplicated by 15x so our users are getting 15 push notifications instead of one.  The duplicacy of the tasks can be seen in the file attached below:</strong><br>
<a href="https://github.com/celery/celery/files/1821326/bug.txt">bug.txt</a></p>
<h2 dir="auto">Expected behavior</h2>
<h2 dir="auto">Actual behavior</h2>
<h1 dir="auto">Checklist</h1>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?utf8=%E2%9C%93&amp;q=is%3Aissue+label%3A%22Issue+Type%3A+Feature+Request%22+">issues list</a><br>
for similar or identical feature requests.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?utf8=%E2%9C%93&amp;q=is%3Apr+label%3A%22PR+Type%3A+Feature%22+">pull requests list</a><br>
for existing proposed implementations of this feature.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the if the same feature was already implemented in the<br>
master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>
<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="299009247" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/4551" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/4551/hovercard" href="https://github.com/celery/celery/issues/4551">#4551</a></li>
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="294771738" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/4525" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/4525/hovercard" href="https://github.com/celery/celery/issues/4525">#4525</a></li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="299009247" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/4551" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/4551/hovercard" href="https://github.com/celery/celery/issues/4551">#4551</a></li>
</ul>
<h1 dir="auto">Brief Summary</h1>
<p dir="auto">Memory leaks can be difficult to track down, especially when they are caused by 3rd party packages. In theory these obviously should be fixed, but in real life, where one needs to keep things running, one might be happy enough if the memory leaks were contained.</p>
<p dir="auto">So I'd like to propose a solution to <em>quarantene</em> memory leaks in task specific processes in order to avoid an increasing memory usage of the Celery workers, which appears to be a common reason for restarting the workers.</p>
<h1 dir="auto">Design</h1>
<h2 dir="auto">Architectural Considerations</h2>
<p dir="auto">None</p>
<h2 dir="auto">Proposed Behavior</h2>
<p dir="auto">My proposal looks as follows:</p>
<ul dir="auto">
<li>The Celery workers fork/spawn/... a new child process before executing a task.</li>
<li>Task execution is done in this child process.</li>
<li>The result is retrieved from the child process</li>
<li>The child process is terminated.</li>
<li>The results is handed to the result backend by the worker process</li>
</ul>
<p dir="auto">In a variation, this proposal might be extended to allow processing of a chunk in child process or the delegation of the task execution might be made optional.</p>
<p dir="auto">In all other regards (e.g. errors, exceptions) this feature should behave identically to the current behavior.</p>
<h2 dir="auto">Proposed UI/UX</h2>

<h2 dir="auto">Diagrams</h2>

<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/1128117/102881655-7883b480-444d-11eb-8a47-dddb27fb8e73.png"><img src="https://user-images.githubusercontent.com/1128117/102881655-7883b480-444d-11eb-8a47-dddb27fb8e73.png" alt="scratch_1" style="max-width: 100%;"></a></p>
<p dir="auto">This diagram illustrates the basic idea. If a task leaks memory, the memory consumption will rise until the Worker process (e.g. the Celery systemd service) is restarted. If the execution of such a task is contained in a separate child process, memory is freed after each task.</p>
<h2 dir="auto">Alternatives</h2>
<p dir="auto">Instead of including this feature in Celery itself, one could implement this feature in a subclass of <code class="notranslate">celery.Task</code>?</p>