<p dir="auto">celery failed to run the command in subprocess. when restarting the celery worker the command is running in the background.<br>
the subprocess is working without calling as celery task</p>
<h1 dir="auto">Checklist</h1>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked="">  I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> This has already been asked to the <a href="https://groups.google.com/forum/#!forum/celery-users" rel="nofollow">discussion group</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>
<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="5895750" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/891" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/891/hovercard" href="https://github.com/celery/celery/issues/891">#891</a></li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>:</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><p dir="auto">software -&gt; celery:4.4.0 (cliffs) kombu:4.6.11 py:3.6.11<br>
billiard:3.6.3.0 py-amqp:2.6.1<br>
platform -&gt; system:Linux arch:64bit, ELF<br>
kernel version:5.3.0-51-generic imp:CPython<br>
loader   -&gt; celery.loaders.app.AppLoader<br>
settings -&gt; transport:pyamqp results:rpc:///</p>
<p dir="auto">broker_url: 'amqp://guest:********@192.168.43.150:5672//'<br>
result_backend: 'rpc:///'<br>
task_queue: &lt;unbound Queue test_subprocess -&gt; &lt;unbound Exchange test_subprocess(direct)&gt; -&gt; model_builder.#&gt;<br>
task_routes: {<br>
'*': {'queue': 'test_subprocess'}}</p>
<p dir="auto"></p>
</details>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
absl-py==0.9.0
affine==2.3.0
amqp==2.6.1
astor==0.8.1
attrs==20.1.0
Babel==2.8.0
billiard==3.6.3.0
celery==4.4.0
certifi==2020.6.20
chardet==3.0.4
click==7.1.2
click-plugins==1.1.1
cligj==0.5.0
cycler==0.10.0
Cython==3.0a6
Flask==1.1.1
Flask-Cors==3.0.8
flower==0.9.3
gast==0.3.3
gevent==1.4.0
google-pasta==0.2.0
greenlet==0.4.16
grpcio==1.25.0
grpcio-tools==1.25.0
h5py==2.10.0
idna==2.10
importlib-metadata==1.7.0
itsdangerous==1.1.0
Jinja2==2.11.2
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.2
kiwisolver==1.2.0
kombu==4.6.11
Markdown==3.2.2
MarkupSafe==1.1.1
matplotlib==3.3.1
minio==5.0.6
numpy==1.19.1
object-detection==0.1
opencv-python==4.1.2.30
pandas==1.1.0
pika==1.1.0
Pillow==7.0.0
protobuf==3.12.2
pyparsing==2.4.7
python-dateutil==2.8.1
python-dotenv==0.11.0
pytz==2020.1
rasterio==1.1.3
redis==3.3.11
requests==2.23.0
scipy==1.5.2
simplification==0.4.4
six==1.15.0
snuggs==1.4.7
tensorboard==1.14.0
tensorflow==1.14.0
tensorflow-estimator==1.14.0
termcolor==1.1.0
tf-slim==1.1.0
tornado==5.1.1
urllib3==1.25.10
vine==1.3.0
Werkzeug==1.0.1
wrapt==1.12.1
zipp==3.1.0
</p>
</details>
<h1 dir="auto">reference subprocess code</h1>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="    train_obj = subprocess.Popen(
    cmd,
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    shell=True
    )
    for line in iter(train_obj.stdout.read(1), ''):  # replace '' with b'' for Python 3
        sys.stdout.write(line)"><pre class="notranslate"><code class="notranslate">    train_obj = subprocess.Popen(
    cmd,
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    shell=True
    )
    for line in iter(train_obj.stdout.read(1), ''):  # replace '' with b'' for Python 3
        sys.stdout.write(line)
</code></pre></div>
<h1 dir="auto">Expected Behavior</h1>
<p dir="auto">subprocess need to be start and stdout can able to see in the logs</p>
<h1 dir="auto">Actual Behavior</h1>
<p dir="auto">stopped before the line of subprocess<br>
restarting the worker. the command is running in the background. not able to see in the logs</p>
<h1 dir="auto">similar question in stack overflow</h1>
<p dir="auto"><a href="https://stackoverflow.com/questions/36995868/how-to-execute-a-long-running-subprocess-inside-celery-task" rel="nofollow">https://stackoverflow.com/questions/36995868/how-to-execute-a-long-running-subprocess-inside-celery-task</a></p>
<h2 dir="auto">Steps to reproduce</h2>
<p dir="auto">celery 3.1.25 with redis</p>
<p dir="auto">celery worker -A celery_worker.celery --loglevel=info --logfile=/tmp/celery.log --beat</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from celery.schedules import crontab

CELERYBEAT_SCHEDULE = {
    'every-monthly-blog-digest': {
        'task': 'app.users.views.monthly_blog_digest',
        'schedule': crontab(hour=9, minute=0, day_of_month=15),
    },
}

CELERY_TIMEZONE = &quot;Europe/Lisbon&quot;"><pre class="notranslate"><code class="notranslate">from celery.schedules import crontab

CELERYBEAT_SCHEDULE = {
    'every-monthly-blog-digest': {
        'task': 'app.users.views.monthly_blog_digest',
        'schedule': crontab(hour=9, minute=0, day_of_month=15),
    },
}

CELERY_TIMEZONE = "Europe/Lisbon"
</code></pre></div>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">Unique execution by task</p>
<h2 dir="auto">Actual behavior</h2>
<p dir="auto">Duplicated tasks</p>
<p dir="auto">Example:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2018-11-15 08:00:00,012: INFO/Beat] Scheduler: Sending due task every-monthly-blog-digest (app.users.views.monthly_blog_digest)
[2018-11-15 08:00:00,031: INFO/MainProcess] Received task: app.users.views.monthly_blog_digest[df2de4fb-35d8-4b67-bec8-aa9b806cbe1f]
[2018-11-15 08:00:00,707: INFO/MainProcess] Received task: app.mymail.send_async_email_celery[7fd76326-49c0-491c-bec4-aa534ff48411] eta:[2018-11-15 08:10:00.689367+00:00]

...

[2018-11-15 09:00:00,027: INFO/Beat] Scheduler: Sending due task every-monthly-blog-digest (app.users.views.monthly_blog_digest)
[2018-11-15 09:00:00,076: INFO/MainProcess] Received task: app.users.views.monthly_blog_digest[0979c42c-be1b-455a-853e-449601124227]
[2018-11-15 09:00:00,218: INFO/MainProcess] Received task: app.mymail.send_async_email_celery[53fe3a81-9119-4669-8af4-080253abafb8] eta:[2018-11-15 09:10:00.211812+00:00]"><pre class="notranslate"><code class="notranslate">[2018-11-15 08:00:00,012: INFO/Beat] Scheduler: Sending due task every-monthly-blog-digest (app.users.views.monthly_blog_digest)
[2018-11-15 08:00:00,031: INFO/MainProcess] Received task: app.users.views.monthly_blog_digest[df2de4fb-35d8-4b67-bec8-aa9b806cbe1f]
[2018-11-15 08:00:00,707: INFO/MainProcess] Received task: app.mymail.send_async_email_celery[7fd76326-49c0-491c-bec4-aa534ff48411] eta:[2018-11-15 08:10:00.689367+00:00]

...

[2018-11-15 09:00:00,027: INFO/Beat] Scheduler: Sending due task every-monthly-blog-digest (app.users.views.monthly_blog_digest)
[2018-11-15 09:00:00,076: INFO/MainProcess] Received task: app.users.views.monthly_blog_digest[0979c42c-be1b-455a-853e-449601124227]
[2018-11-15 09:00:00,218: INFO/MainProcess] Received task: app.mymail.send_async_email_celery[53fe3a81-9119-4669-8af4-080253abafb8] eta:[2018-11-15 09:10:00.211812+00:00]
</code></pre></div>