<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> This has already been asked to the <a href="https://groups.google.com/forum/#!forum/celery-users" rel="nofollow">discussion group</a> first. <a href="https://groups.google.com/forum/#!topic/celery-users/TBCxDemhSdE" rel="nofollow">https://groups.google.com/forum/#!topic/celery-users/TBCxDemhSdE</a></li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="444739232" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/5527" data-hovercard-type="pull_request" data-hovercard-url="/celery/celery/pull/5527/hovercard" href="https://github.com/celery/celery/pull/5527">#5527</a>, <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="282410055" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/4454" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/4454/hovercard" href="https://github.com/celery/celery/issues/4454">#4454</a></li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>: 4.4.0</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ celery -A celery_tacks report

software -&gt; celery:4.4.0 (cliffs) kombu:4.6.7 py:3.8.1
            billiard:3.6.1.0 py-amqp:2.5.2
platform -&gt; system:Linux arch:64bit, ELF
            kernel version:4.15.0-47-generic imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:pyamqp results:mongodb+srv://remains:**@mongo.example.com/database_name

broker_url: 'amqp://remains:********@rabbitmq.example.com:5672//'
result_backend: 'mongodb+srv://remains:********@mongo.example.com/database_name'
include: [
  'store_remains', 'providers', 'mongo_utils', 'afinalux', 'afinalux-diff',
  'christmass', 'danio', 'danio-diff', 'elgo-msk', 'evan-diff', 'globus',
  'lightstar', 'lussole', 'mw-light', 'prorab', 'quadro-stroy', 'sds-group',
  'sonex', 'stluce', 'stluce-diff', 'technolight', 'contactplus', 'lex-c', 'lex-c-diff',
 'greentrees', 'lifan-moto', 'triumph-nord-rus', 'crawl_site', 'mail2csv', 'mail2diff',
  'rotate_mb', 'store_out_of_stock'
]
accept_content: [
 'mongo_json', 'application/x-mongo-json', 'json', 'application/json'
]
task_serializer: 'mongo_json'
result_serializer: 'mongo_json'
task_default_queue: 'remains_tasks'
worker_prefetch_multiplier: 1
worker_max_tasks_per_child: 45
worker_hijack_root_logger: False
mongodb_backend_settings: {
 'compressors': 'snappy', 'database': '********'}
CELERY_MONGO_SCHEDULER_URI: 'mongodb+srv://remains:********@mongo.example.com/database_name?ssl=false'
CELERY_MONGO_SCHEDULER_DB: 'remains'
CELERY_MONGO_SCHEDULER_ENTRIES_COL: 'schedules_entries'
CELERY_MONGO_SCHEDULER_META_COL: 'schedules_meta'
beat_schedule: {
    'afinalux': {   'args': ['afinalux'],
                    'comment': '',
                    'is_enabled': True,
                    'schedule': &lt;crontab: 7 0 * * * (m/h/d/dM/MY)&gt;,
                    'task': 'celery_tacks.load_link',
                    'title': 'Афина Гарден'},
    'afinalux-diff': {   'args': ['afinalux-diff'],
                         'comment': 'рассылка с изменениями прайса Афина '
                                    'Гарден',
                         'is_enabled': True,
                         'schedule': &lt;crontab: 7 0 * * * (m/h/d/dM/MY)&gt;,
                         'task': 'celery_tacks.load_link',
                         'title': 'Афина Гарден'},
    'allpools': {   'args': ['allpools'],
                    'comment': '',
                    'is_enabled': True,
                    'schedule': &lt;crontab: 19 1 * * * (m/h/d/dM/MY)&gt;,
                    'task': 'celery_tacks.load_site',
                    'title': 'Олпулс'},
    'aquapolis': {   'args': ['aquapolis'],
                     'comment': '',
                     'is_enabled': True,
                     'schedule': &lt;crontab: 31 0 * * * (m/h/d/dM/MY)&gt;,
                     'task': 'celery_tacks.load_site',
                     'title': 'Акваполис'},
    'aquapolis_bestway': {   'args': ['aquapolis_bestway'],
                             'comment': '',
                             'is_enabled': True,
                             'schedule': &lt;crontab: 31 0 * * * (m/h/d/dM/MY)&gt;,
                             'task': 'celery_tacks.load_site',
                             'title': 'Акваполис_Bestway'},
    'audiomania': {   'args': ['audiomania'],
                      'comment': '',
                      'is_enabled': True,
                      'schedule': &lt;crontab: 17 0 * * * (m/h/d/dM/MY)&gt;,
                      'task': 'celery_tacks.load_site',
                      'title': 'Аудиомания'},
    'christmass': {   'args': ['christmass'],
                      'comment': '',
                      'is_enabled': True,
                      'schedule': &lt;crontab: 13-59/30 * * * * (m/h/d/dM/MY)&gt;,
                      'task': 'celery_tacks.load_link',
                      'title': 'Актив групп'},
    'contactplus': {   'args': ['contactplus'],
                       'comment': 'Остатки Контакт Плюс',
                       'is_enabled': True,
                       'schedule': &lt;crontab: 17-59/30 * * * * (m/h/d/dM/MY)&gt;,
                       'task': 'celery_tacks.load_link',
                       'title': 'Контакт Плюс'},
    'danio': {   'args': ['danio'],
                 'comment': '',
                 'is_enabled': True,
                 'schedule': &lt;crontab: 13 2 * * * (m/h/d/dM/MY)&gt;,
                 'task': 'celery_tacks.load_link',
                 'title': 'Данио'},
    'danio-diff': {   'args': ['danio-diff'],
                      'comment': 'Рассылка с изменениями прайса Данио',
                      'is_enabled': True,
                      'schedule': &lt;crontab: 17 2 * * * (m/h/d/dM/MY)&gt;,
                      'task': 'celery_tacks.load_link',
                      'title': 'Данио'},
    'decomo': {   'args': ['decomo'],
                  'comment': '',
                  'is_enabled': True,
                  'schedule': &lt;crontab: 23 0 * * * (m/h/d/dM/MY)&gt;,
                  'task': 'celery_tacks.load_site',
                  'title': 'Декомо'},
    'delta': {   'args': ['delta'],
                 'comment': '',
                 'is_enabled': True,
                 'schedule': &lt;crontab: 19 0 * * * (m/h/d/dM/MY)&gt;,
                 'task': 'celery_tacks.load_site',
                 'title': 'Дельта'},
    'elgo-msk': {   'args': ['elgo-msk'],
                    'comment': 'Отключено по просьбе заказчика',
                    'is_enabled': False,
                    'schedule': &lt;crontab: 29-59/30 * * * * (m/h/d/dM/MY)&gt;,
                    'task': 'celery_tacks.load_link',
                    'title': 'Эгло'},
    'evan-diff': {   'args': ['evan-diff'],
                     'comment': 'Рассылка с изменениями прайса Эван',
                     'is_enabled': True,
                     'schedule': &lt;crontab: 17 3 0 * * (m/h/d/dM/MY)&gt;,
                     'task': 'celery_tacks.load_link',
                     'title': 'Эван'},
    'floraimport': {   'args': ['floraimport'],
                       'comment': '',
                       'is_enabled': True,
                       'schedule': &lt;crontab: 11 0 * * * (m/h/d/dM/MY)&gt;,
                       'task': 'celery_tacks.load_site',
                       'title': 'Флора Импорт'},
    'globus': {   'args': ['globus'],
                  'comment': '',
                  'is_enabled': True,
                  'schedule': &lt;crontab: 5-59/30 * * * * (m/h/d/dM/MY)&gt;,
                  'task': 'celery_tacks.load_link',
                  'title': 'Глобус'},
    'greentrees': {   'args': ['greentrees'],
                      'comment': 'Остатки Green Trees',
                      'is_enabled': True,
                      'schedule': &lt;crontab: 37 1 * * * (m/h/d/dM/MY)&gt;,
                      'task': 'celery_tacks.load_link',
                      'title': 'Green Trees'},
    'housetech': {   'args': ['housetech'],
                     'comment': '',
                     'is_enabled': True,
                     'schedule': &lt;crontab: 13 0 * * * (m/h/d/dM/MY)&gt;,
                     'task': 'celery_tacks.load_site',
                     'title': 'Домотехника'},
    'housetech_nsk': {   'args': ['housetech_nsk'],
                         'comment': '',
                         'is_enabled': True,
                         'schedule': &lt;crontab: 23 0 * * * (m/h/d/dM/MY)&gt;,
                         'task': 'celery_tacks.load_site',
                         'title': 'Домотехника_НСК'},
    'housetech_spb': {   'args': ['housetech_spb'],
                         'comment': '',
                         'is_enabled': True,
                         'schedule': &lt;crontab: 31 0 * * * (m/h/d/dM/MY)&gt;,
                         'task': 'celery_tacks.load_site',
                         'title': 'Домотехника_СПБ'},
    'huter-gmbh': {   'args': ['huter-gmbh'],
                      'comment': '',
                      'is_enabled': True,
                      'schedule': &lt;crontab: 43 0 0 * * (m/h/d/dM/MY)&gt;,
                      'task': 'celery_tacks.load_site',
                      'title': 'Континент ГК'},
    'iatec': {   'args': ['iatec'],
                 'comment': '',
                 'is_enabled': True,
                 'schedule': &lt;crontab: 41 0 * * * (m/h/d/dM/MY)&gt;,
                 'task': 'celery_tacks.load_site',
                 'title': 'Три Пятых'},
    'interinstrument': {   'args': ['interinstrument'],
                           'comment': '',
                           'is_enabled': True,
                           'schedule': &lt;crontab: 47 0 * * * (m/h/d/dM/MY)&gt;,
                           'task': 'celery_tacks.load_site',
                           'title': 'ИнтерИнструмент'},
    'lex-c': {   'args': ['lex-c'],
                 'comment': 'Остатки Лекс-с',
                 'is_enabled': True,
                 'schedule': &lt;crontab: 21 2 0 * * (m/h/d/dM/MY)&gt;,
                 'task': 'celery_tacks.load_link',
                 'title': 'Лекс-С'},
    'lex-c-diff': {   'args': ['lex-c-diff'],
                      'comment': 'Рассылка с изменениями прайса Лекс-с',
                      'is_enabled': True,
                      'schedule': &lt;crontab: 23 2 0 * * (m/h/d/dM/MY)&gt;,
                      'task': 'celery_tacks.load_link',
                      'title': 'Лекс-С'},
    'lifan-moto': {   'args': ['lifan-moto'],
                      'comment': 'Остатки lifan-moto',
                      'is_enabled': True,
                      'schedule': &lt;crontab: 37 1 0 * * (m/h/d/dM/MY)&gt;,
                      'task': 'celery_tacks.load_link',
                      'title': 'Мото северо-запад'},
    'lightstar': {   'args': ['lightstar'],
                     'comment': 'Отключено по просьбе заказчика',
                     'is_enabled': False,
                     'schedule': &lt;crontab: 17-59/30 * * * * (m/h/d/dM/MY)&gt;,
                     'task': 'celery_tacks.load_link',
                     'title': 'Лайтстар'},
    'lussole': {   'args': ['lussole'],
                   'comment': 'Отключено по просьбе заказчика',
                   'is_enabled': False,
                   'schedule': &lt;crontab: 29 */2 * * * (m/h/d/dM/MY)&gt;,
                   'task': 'celery_tacks.load_link',
                   'title': 'Lussole'},
    'mail2csv': {   'is_enabled': True,
                    'is_system': True,
                    'schedule': &lt;crontab: 3-59/30 * * * * (m/h/d/dM/MY)&gt;,
                    'task': 'celery_tacks.load_mail2csv',
                    'title': 'Остатки из писем'},
    'mail2diff': {   'is_enabled': True,
                     'is_system': True,
                     'schedule': &lt;crontab: 13 * * * * (m/h/d/dM/MY)&gt;,
                     'task': 'celery_tacks.load_mail2diff',
                     'title': 'Diff прайса из писем'},
    'mon_site': {   'args': ['poolfactory'],
                    'is_enabled': False,
                    'is_system': False,
                    'schedule': &lt;crontab: 23 3 * * * (m/h/d/dM/MY)&gt;,
                    'task': 'celery_tacks.load_mon_site',
                    'title': 'Мониторинг продаж с poolfactory'},
    'mw-light': {   'args': ['mw-light'],
                    'comment': 'Отключено по просьбе заказчика',
                    'is_enabled': False,
                    'schedule': &lt;crontab: 1-59/30 * * * * (m/h/d/dM/MY)&gt;,
                    'task': 'celery_tacks.load_link',
                    'title': 'MW-Light'},
    'novum': {   'args': ['novum'],
                 'comment': '',
                 'is_enabled': True,
                 'schedule': &lt;crontab: 7 0 * * * (m/h/d/dM/MY)&gt;,
                 'task': 'celery_tacks.load_site',
                 'title': 'Novum'},
    'pompina': {   'args': ['pompina'],
                   'comment': '',
                   'is_enabled': True,
                   'schedule': &lt;crontab: 13 0 0 * * (m/h/d/dM/MY)&gt;,
                   'task': 'celery_tacks.load_site',
                   'title': 'Саблайн сервис'},
    'price.allpools.ru': {   'args': ['allpools.ru'],
                             'comment': '',
                             'is_enabled': True,
                             'schedule': &lt;crontab: 31 3 0 * * (m/h/d/dM/MY)&gt;,
                             'task': 'celery_tacks.load_price_site',
                             'title': 'Цены с сайта allpools.ru'},
    'price.aquapolis.ru': {   'args': ['aquapolis.ru'],
                              'comment': '',
                              'is_enabled': True,
                              'schedule': &lt;crontab: 29 3 0 * * (m/h/d/dM/MY)&gt;,
                              'task': 'celery_tacks.load_price_site',
                              'title': 'Цены с сайта aquapolis.ru'},
    'price.decomo': {   'args': ['decomo'],
                        'comment': '',
                        'is_enabled': True,
                        'schedule': &lt;crontab: 13 1 * 1 * (m/h/d/dM/MY)&gt;,
                        'task': 'celery_tacks.load_price_site',
                        'title': 'Цены с сайта decomo'},
    'price.decomo_rrc': {   'args': ['decomo_rrc'],
                            'comment': '',
                            'is_enabled': True,
                            'schedule': &lt;crontab: 13 1 * * * (m/h/d/dM/MY)&gt;,
                            'task': 'celery_tacks.load_price_site',
                            'title': 'Цены с сайта decomo_rrc'},
    'price.ekopromgroup.ru': {   'args': ['ekopromgroup.ru'],
                                 'comment': 'Цены ЭкоПром',
                                 'is_enabled': True,
                                 'schedule': &lt;crontab: 41 3 0 * * (m/h/d/dM/MY)&gt;,
                                 'task': 'celery_tacks.load_price_site',
                                 'title': 'Цены с сайта ekopromgroup.ru'},
    'price.floraimport': {   'args': ['floraimport'],
                             'comment': '',
                             'is_enabled': True,
                             'schedule': &lt;crontab: 11 0 0 1-7,15-21,29-31 * (m/h/d/dM/MY)&gt;,
                             'task': 'celery_tacks.load_price_site',
                             'title': 'Цены с сайта floraimport'},
    'price.kamnirus': {   'args': ['kamnirus'],
                          'comment': 'Отключено по просьбе заказчика',
                          'is_enabled': False,
                          'schedule': &lt;crontab: 19 3 0 * * (m/h/d/dM/MY)&gt;,
                          'task': 'celery_tacks.load_price_site',
                          'title': 'Цены с сайта kamnirus'},
    'price.novum': {   'args': ['novum'],
                       'comment': 'Цены Novum',
                       'is_enabled': False,
                       'schedule': &lt;crontab: 17 2 0 * * (m/h/d/dM/MY)&gt;,
                       'task': 'celery_tacks.load_price_site',
                       'title': 'Цены с сайта novum'},
    'price.polimer-group.com': {   'args': ['polimer-group.com'],
                                   'comment': 'Цены Polimer Group',
                                   'is_enabled': True,
                                   'schedule': &lt;crontab: 37 3 0 * * (m/h/d/dM/MY)&gt;,
                                   'task': 'celery_tacks.load_price_site',
                                   'title': 'Цены с сайта polimer-group.com'},
    'price.prorab': {   'args': ['prorab'],
                        'comment': '',
                        'is_enabled': False,
                        'schedule': &lt;crontab: 37 3 0 * * (m/h/d/dM/MY)&gt;,
                        'task': 'celery_tacks.load_price_site',
                        'title': 'Цены с сайта prorab'},
    'price.unisaw': {   'args': ['unisaw'],
                        'comment': '',
                        'is_enabled': True,
                        'schedule': &lt;crontab: 23 3 0 * * (m/h/d/dM/MY)&gt;,
                        'task': 'celery_tacks.load_price_site',
                        'title': 'Цены с сайта unisaw'},
    'price.Домотехника': {   'args': ['Домотехника'],
                             'comment': 'Изменения в составе прайсов',
                             'is_enabled': True,
                             'schedule': &lt;crontab: 41 3 0 * * (m/h/d/dM/MY)&gt;,
                             'task': 'celery_tacks.load_price_site',
                             'title': 'Цены с сайта Домотехника'},
    'prorab': {   'args': ['prorab'],
                  'comment': '',
                  'is_enabled': False,
                  'schedule': &lt;crontab: 7-59/30 * * * * (m/h/d/dM/MY)&gt;,
                  'task': 'celery_tacks.load_link',
                  'title': 'Prorab'},
    'quadro-stroy': {   'args': ['quadro-stroy'],
                        'comment': '',
                        'is_enabled': True,
                        'schedule': &lt;crontab: 17 1 * * * (m/h/d/dM/MY)&gt;,
                        'task': 'celery_tacks.load_link',
                        'title': 'Квадро-Строй'},
    'rotate_mb': {   'is_enabled': True,
                     'is_system': True,
                     'schedule': &lt;crontab: 23 0 0 * * (m/h/d/dM/MY)&gt;,
                     'task': 'celery_tacks.rotate_mb',
                     'title': 'Очистка устаревших файлов'},
    'sds-group': {   'args': ['sds-group'],
                     'comment': '',
                     'is_enabled': True,
                     'schedule': &lt;crontab: 29 0 * * * (m/h/d/dM/MY)&gt;,
                     'task': 'celery_tacks.load_link',
                     'title': 'СДС-Группа'},
    'sima-land': {   'args': ['sima-land'],
                     'comment': '',
                     'is_enabled': True,
                     'schedule': &lt;crontab: 53 0 * * * (m/h/d/dM/MY)&gt;,
                     'task': 'celery_tacks.load_site',
                     'title': 'Сима-ленд'},
    'sonex': {   'args': ['sonex'],
                 'comment': '',
                 'is_enabled': True,
                 'schedule': &lt;crontab: 11-59/30 * * * * (m/h/d/dM/MY)&gt;,
                 'task': 'celery_tacks.load_link',
                 'title': 'Сонекс'},
    'stluce': {   'args': ['stluce'],
                  'comment': 'Остатки ST Luce Velante',
                  'is_enabled': True,
                  'schedule': &lt;crontab: 19 1 * * * (m/h/d/dM/MY)&gt;,
                  'task': 'celery_tacks.load_link',
                  'title': 'ST Luce'},
    'stluce-diff': {   'args': ['stluce-diff'],
                       'comment': 'рассылка с изменениями прайса ST Luce '
                                  'Velante',
                       'is_enabled': True,
                       'schedule': &lt;crontab: 23 1 * * * (m/h/d/dM/MY)&gt;,
                       'task': 'celery_tacks.load_link',
                       'title': 'ST Luce'},
    'store_outofstock': {   'is_enabled': True,
                            'is_system': True,
                            'schedule': &lt;crontab: 49 0-20,23 * * * (m/h/d/dM/MY)&gt;,
                            'task': 'celery_tacks.store_outofstock',
                            'title': 'Автозаливка &quot;нет в наличии&quot;'},
    'technolight': {   'args': ['technolight'],
                       'comment': 'Отключено по просьбе заказчика',
                       'is_enabled': False,
                       'schedule': &lt;crontab: 19-59/30 * * * * (m/h/d/dM/MY)&gt;,
                       'task': 'celery_tacks.load_link',
                       'title': 'Тренд Света'},
    'triumph-nord-rus': {   'args': ['triumph-nord-rus'],
                            'comment': '',
                            'is_enabled': True,
                            'schedule': &lt;crontab: 11 1 * * * (m/h/d/dM/MY)&gt;,
                            'task': 'celery_tacks.load_link',
                            'title': 'Триумф Норд Рус'},
    'unisaw': {   'args': ['unisaw'],
                  'comment': '',
                  'is_enabled': True,
                  'schedule': &lt;crontab: 3 0 * * * (m/h/d/dM/MY)&gt;,
                  'task': 'celery_tacks.load_site',
                  'title': 'Unisaw'},
    'vodalux-fontan': {   'args': ['vodalux-fontan'],
                          'comment': '',
                          'is_enabled': True,
                          'schedule': &lt;crontab: 37 0 * * * (m/h/d/dM/MY)&gt;,
                          'task': 'celery_tacks.load_site',
                          'title': 'Водалюкс'},
    'vodalux-prud': {   'args': ['vodalux-prud'],
                        'comment': '',
                        'is_enabled': True,
                        'schedule': &lt;crontab: 13 0 * * * (m/h/d/dM/MY)&gt;,
                        'task': 'celery_tacks.load_site',
                        'title': 'Водалюкс'},
    'y-skazki': {   'args': ['y-skazki'],
                    'comment': 'Отключено по просьбе заказчика',
                    'is_enabled': False,
                    'schedule': &lt;crontab: 17 1 * * * (m/h/d/dM/MY)&gt;,
                    'task': 'celery_tacks.load_site',
                    'title': 'Bestway Юг'}}"><pre class="notranslate"><code class="notranslate">$ celery -A celery_tacks report

software -&gt; celery:4.4.0 (cliffs) kombu:4.6.7 py:3.8.1
            billiard:3.6.1.0 py-amqp:2.5.2
platform -&gt; system:Linux arch:64bit, ELF
            kernel version:4.15.0-47-generic imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:pyamqp results:mongodb+srv://remains:**@mongo.example.com/database_name

broker_url: 'amqp://remains:********@rabbitmq.example.com:5672//'
result_backend: 'mongodb+srv://remains:********@mongo.example.com/database_name'
include: [
  'store_remains', 'providers', 'mongo_utils', 'afinalux', 'afinalux-diff',
  'christmass', 'danio', 'danio-diff', 'elgo-msk', 'evan-diff', 'globus',
  'lightstar', 'lussole', 'mw-light', 'prorab', 'quadro-stroy', 'sds-group',
  'sonex', 'stluce', 'stluce-diff', 'technolight', 'contactplus', 'lex-c', 'lex-c-diff',
 'greentrees', 'lifan-moto', 'triumph-nord-rus', 'crawl_site', 'mail2csv', 'mail2diff',
  'rotate_mb', 'store_out_of_stock'
]
accept_content: [
 'mongo_json', 'application/x-mongo-json', 'json', 'application/json'
]
task_serializer: 'mongo_json'
result_serializer: 'mongo_json'
task_default_queue: 'remains_tasks'
worker_prefetch_multiplier: 1
worker_max_tasks_per_child: 45
worker_hijack_root_logger: False
mongodb_backend_settings: {
 'compressors': 'snappy', 'database': '********'}
CELERY_MONGO_SCHEDULER_URI: 'mongodb+srv://remains:********@mongo.example.com/database_name?ssl=false'
CELERY_MONGO_SCHEDULER_DB: 'remains'
CELERY_MONGO_SCHEDULER_ENTRIES_COL: 'schedules_entries'
CELERY_MONGO_SCHEDULER_META_COL: 'schedules_meta'
beat_schedule: {
    'afinalux': {   'args': ['afinalux'],
                    'comment': '',
                    'is_enabled': True,
                    'schedule': &lt;crontab: 7 0 * * * (m/h/d/dM/MY)&gt;,
                    'task': 'celery_tacks.load_link',
                    'title': 'Афина Гарден'},
    'afinalux-diff': {   'args': ['afinalux-diff'],
                         'comment': 'рассылка с изменениями прайса Афина '
                                    'Гарден',
                         'is_enabled': True,
                         'schedule': &lt;crontab: 7 0 * * * (m/h/d/dM/MY)&gt;,
                         'task': 'celery_tacks.load_link',
                         'title': 'Афина Гарден'},
    'allpools': {   'args': ['allpools'],
                    'comment': '',
                    'is_enabled': True,
                    'schedule': &lt;crontab: 19 1 * * * (m/h/d/dM/MY)&gt;,
                    'task': 'celery_tacks.load_site',
                    'title': 'Олпулс'},
    'aquapolis': {   'args': ['aquapolis'],
                     'comment': '',
                     'is_enabled': True,
                     'schedule': &lt;crontab: 31 0 * * * (m/h/d/dM/MY)&gt;,
                     'task': 'celery_tacks.load_site',
                     'title': 'Акваполис'},
    'aquapolis_bestway': {   'args': ['aquapolis_bestway'],
                             'comment': '',
                             'is_enabled': True,
                             'schedule': &lt;crontab: 31 0 * * * (m/h/d/dM/MY)&gt;,
                             'task': 'celery_tacks.load_site',
                             'title': 'Акваполис_Bestway'},
    'audiomania': {   'args': ['audiomania'],
                      'comment': '',
                      'is_enabled': True,
                      'schedule': &lt;crontab: 17 0 * * * (m/h/d/dM/MY)&gt;,
                      'task': 'celery_tacks.load_site',
                      'title': 'Аудиомания'},
    'christmass': {   'args': ['christmass'],
                      'comment': '',
                      'is_enabled': True,
                      'schedule': &lt;crontab: 13-59/30 * * * * (m/h/d/dM/MY)&gt;,
                      'task': 'celery_tacks.load_link',
                      'title': 'Актив групп'},
    'contactplus': {   'args': ['contactplus'],
                       'comment': 'Остатки Контакт Плюс',
                       'is_enabled': True,
                       'schedule': &lt;crontab: 17-59/30 * * * * (m/h/d/dM/MY)&gt;,
                       'task': 'celery_tacks.load_link',
                       'title': 'Контакт Плюс'},
    'danio': {   'args': ['danio'],
                 'comment': '',
                 'is_enabled': True,
                 'schedule': &lt;crontab: 13 2 * * * (m/h/d/dM/MY)&gt;,
                 'task': 'celery_tacks.load_link',
                 'title': 'Данио'},
    'danio-diff': {   'args': ['danio-diff'],
                      'comment': 'Рассылка с изменениями прайса Данио',
                      'is_enabled': True,
                      'schedule': &lt;crontab: 17 2 * * * (m/h/d/dM/MY)&gt;,
                      'task': 'celery_tacks.load_link',
                      'title': 'Данио'},
    'decomo': {   'args': ['decomo'],
                  'comment': '',
                  'is_enabled': True,
                  'schedule': &lt;crontab: 23 0 * * * (m/h/d/dM/MY)&gt;,
                  'task': 'celery_tacks.load_site',
                  'title': 'Декомо'},
    'delta': {   'args': ['delta'],
                 'comment': '',
                 'is_enabled': True,
                 'schedule': &lt;crontab: 19 0 * * * (m/h/d/dM/MY)&gt;,
                 'task': 'celery_tacks.load_site',
                 'title': 'Дельта'},
    'elgo-msk': {   'args': ['elgo-msk'],
                    'comment': 'Отключено по просьбе заказчика',
                    'is_enabled': False,
                    'schedule': &lt;crontab: 29-59/30 * * * * (m/h/d/dM/MY)&gt;,
                    'task': 'celery_tacks.load_link',
                    'title': 'Эгло'},
    'evan-diff': {   'args': ['evan-diff'],
                     'comment': 'Рассылка с изменениями прайса Эван',
                     'is_enabled': True,
                     'schedule': &lt;crontab: 17 3 0 * * (m/h/d/dM/MY)&gt;,
                     'task': 'celery_tacks.load_link',
                     'title': 'Эван'},
    'floraimport': {   'args': ['floraimport'],
                       'comment': '',
                       'is_enabled': True,
                       'schedule': &lt;crontab: 11 0 * * * (m/h/d/dM/MY)&gt;,
                       'task': 'celery_tacks.load_site',
                       'title': 'Флора Импорт'},
    'globus': {   'args': ['globus'],
                  'comment': '',
                  'is_enabled': True,
                  'schedule': &lt;crontab: 5-59/30 * * * * (m/h/d/dM/MY)&gt;,
                  'task': 'celery_tacks.load_link',
                  'title': 'Глобус'},
    'greentrees': {   'args': ['greentrees'],
                      'comment': 'Остатки Green Trees',
                      'is_enabled': True,
                      'schedule': &lt;crontab: 37 1 * * * (m/h/d/dM/MY)&gt;,
                      'task': 'celery_tacks.load_link',
                      'title': 'Green Trees'},
    'housetech': {   'args': ['housetech'],
                     'comment': '',
                     'is_enabled': True,
                     'schedule': &lt;crontab: 13 0 * * * (m/h/d/dM/MY)&gt;,
                     'task': 'celery_tacks.load_site',
                     'title': 'Домотехника'},
    'housetech_nsk': {   'args': ['housetech_nsk'],
                         'comment': '',
                         'is_enabled': True,
                         'schedule': &lt;crontab: 23 0 * * * (m/h/d/dM/MY)&gt;,
                         'task': 'celery_tacks.load_site',
                         'title': 'Домотехника_НСК'},
    'housetech_spb': {   'args': ['housetech_spb'],
                         'comment': '',
                         'is_enabled': True,
                         'schedule': &lt;crontab: 31 0 * * * (m/h/d/dM/MY)&gt;,
                         'task': 'celery_tacks.load_site',
                         'title': 'Домотехника_СПБ'},
    'huter-gmbh': {   'args': ['huter-gmbh'],
                      'comment': '',
                      'is_enabled': True,
                      'schedule': &lt;crontab: 43 0 0 * * (m/h/d/dM/MY)&gt;,
                      'task': 'celery_tacks.load_site',
                      'title': 'Континент ГК'},
    'iatec': {   'args': ['iatec'],
                 'comment': '',
                 'is_enabled': True,
                 'schedule': &lt;crontab: 41 0 * * * (m/h/d/dM/MY)&gt;,
                 'task': 'celery_tacks.load_site',
                 'title': 'Три Пятых'},
    'interinstrument': {   'args': ['interinstrument'],
                           'comment': '',
                           'is_enabled': True,
                           'schedule': &lt;crontab: 47 0 * * * (m/h/d/dM/MY)&gt;,
                           'task': 'celery_tacks.load_site',
                           'title': 'ИнтерИнструмент'},
    'lex-c': {   'args': ['lex-c'],
                 'comment': 'Остатки Лекс-с',
                 'is_enabled': True,
                 'schedule': &lt;crontab: 21 2 0 * * (m/h/d/dM/MY)&gt;,
                 'task': 'celery_tacks.load_link',
                 'title': 'Лекс-С'},
    'lex-c-diff': {   'args': ['lex-c-diff'],
                      'comment': 'Рассылка с изменениями прайса Лекс-с',
                      'is_enabled': True,
                      'schedule': &lt;crontab: 23 2 0 * * (m/h/d/dM/MY)&gt;,
                      'task': 'celery_tacks.load_link',
                      'title': 'Лекс-С'},
    'lifan-moto': {   'args': ['lifan-moto'],
                      'comment': 'Остатки lifan-moto',
                      'is_enabled': True,
                      'schedule': &lt;crontab: 37 1 0 * * (m/h/d/dM/MY)&gt;,
                      'task': 'celery_tacks.load_link',
                      'title': 'Мото северо-запад'},
    'lightstar': {   'args': ['lightstar'],
                     'comment': 'Отключено по просьбе заказчика',
                     'is_enabled': False,
                     'schedule': &lt;crontab: 17-59/30 * * * * (m/h/d/dM/MY)&gt;,
                     'task': 'celery_tacks.load_link',
                     'title': 'Лайтстар'},
    'lussole': {   'args': ['lussole'],
                   'comment': 'Отключено по просьбе заказчика',
                   'is_enabled': False,
                   'schedule': &lt;crontab: 29 */2 * * * (m/h/d/dM/MY)&gt;,
                   'task': 'celery_tacks.load_link',
                   'title': 'Lussole'},
    'mail2csv': {   'is_enabled': True,
                    'is_system': True,
                    'schedule': &lt;crontab: 3-59/30 * * * * (m/h/d/dM/MY)&gt;,
                    'task': 'celery_tacks.load_mail2csv',
                    'title': 'Остатки из писем'},
    'mail2diff': {   'is_enabled': True,
                     'is_system': True,
                     'schedule': &lt;crontab: 13 * * * * (m/h/d/dM/MY)&gt;,
                     'task': 'celery_tacks.load_mail2diff',
                     'title': 'Diff прайса из писем'},
    'mon_site': {   'args': ['poolfactory'],
                    'is_enabled': False,
                    'is_system': False,
                    'schedule': &lt;crontab: 23 3 * * * (m/h/d/dM/MY)&gt;,
                    'task': 'celery_tacks.load_mon_site',
                    'title': 'Мониторинг продаж с poolfactory'},
    'mw-light': {   'args': ['mw-light'],
                    'comment': 'Отключено по просьбе заказчика',
                    'is_enabled': False,
                    'schedule': &lt;crontab: 1-59/30 * * * * (m/h/d/dM/MY)&gt;,
                    'task': 'celery_tacks.load_link',
                    'title': 'MW-Light'},
    'novum': {   'args': ['novum'],
                 'comment': '',
                 'is_enabled': True,
                 'schedule': &lt;crontab: 7 0 * * * (m/h/d/dM/MY)&gt;,
                 'task': 'celery_tacks.load_site',
                 'title': 'Novum'},
    'pompina': {   'args': ['pompina'],
                   'comment': '',
                   'is_enabled': True,
                   'schedule': &lt;crontab: 13 0 0 * * (m/h/d/dM/MY)&gt;,
                   'task': 'celery_tacks.load_site',
                   'title': 'Саблайн сервис'},
    'price.allpools.ru': {   'args': ['allpools.ru'],
                             'comment': '',
                             'is_enabled': True,
                             'schedule': &lt;crontab: 31 3 0 * * (m/h/d/dM/MY)&gt;,
                             'task': 'celery_tacks.load_price_site',
                             'title': 'Цены с сайта allpools.ru'},
    'price.aquapolis.ru': {   'args': ['aquapolis.ru'],
                              'comment': '',
                              'is_enabled': True,
                              'schedule': &lt;crontab: 29 3 0 * * (m/h/d/dM/MY)&gt;,
                              'task': 'celery_tacks.load_price_site',
                              'title': 'Цены с сайта aquapolis.ru'},
    'price.decomo': {   'args': ['decomo'],
                        'comment': '',
                        'is_enabled': True,
                        'schedule': &lt;crontab: 13 1 * 1 * (m/h/d/dM/MY)&gt;,
                        'task': 'celery_tacks.load_price_site',
                        'title': 'Цены с сайта decomo'},
    'price.decomo_rrc': {   'args': ['decomo_rrc'],
                            'comment': '',
                            'is_enabled': True,
                            'schedule': &lt;crontab: 13 1 * * * (m/h/d/dM/MY)&gt;,
                            'task': 'celery_tacks.load_price_site',
                            'title': 'Цены с сайта decomo_rrc'},
    'price.ekopromgroup.ru': {   'args': ['ekopromgroup.ru'],
                                 'comment': 'Цены ЭкоПром',
                                 'is_enabled': True,
                                 'schedule': &lt;crontab: 41 3 0 * * (m/h/d/dM/MY)&gt;,
                                 'task': 'celery_tacks.load_price_site',
                                 'title': 'Цены с сайта ekopromgroup.ru'},
    'price.floraimport': {   'args': ['floraimport'],
                             'comment': '',
                             'is_enabled': True,
                             'schedule': &lt;crontab: 11 0 0 1-7,15-21,29-31 * (m/h/d/dM/MY)&gt;,
                             'task': 'celery_tacks.load_price_site',
                             'title': 'Цены с сайта floraimport'},
    'price.kamnirus': {   'args': ['kamnirus'],
                          'comment': 'Отключено по просьбе заказчика',
                          'is_enabled': False,
                          'schedule': &lt;crontab: 19 3 0 * * (m/h/d/dM/MY)&gt;,
                          'task': 'celery_tacks.load_price_site',
                          'title': 'Цены с сайта kamnirus'},
    'price.novum': {   'args': ['novum'],
                       'comment': 'Цены Novum',
                       'is_enabled': False,
                       'schedule': &lt;crontab: 17 2 0 * * (m/h/d/dM/MY)&gt;,
                       'task': 'celery_tacks.load_price_site',
                       'title': 'Цены с сайта novum'},
    'price.polimer-group.com': {   'args': ['polimer-group.com'],
                                   'comment': 'Цены Polimer Group',
                                   'is_enabled': True,
                                   'schedule': &lt;crontab: 37 3 0 * * (m/h/d/dM/MY)&gt;,
                                   'task': 'celery_tacks.load_price_site',
                                   'title': 'Цены с сайта polimer-group.com'},
    'price.prorab': {   'args': ['prorab'],
                        'comment': '',
                        'is_enabled': False,
                        'schedule': &lt;crontab: 37 3 0 * * (m/h/d/dM/MY)&gt;,
                        'task': 'celery_tacks.load_price_site',
                        'title': 'Цены с сайта prorab'},
    'price.unisaw': {   'args': ['unisaw'],
                        'comment': '',
                        'is_enabled': True,
                        'schedule': &lt;crontab: 23 3 0 * * (m/h/d/dM/MY)&gt;,
                        'task': 'celery_tacks.load_price_site',
                        'title': 'Цены с сайта unisaw'},
    'price.Домотехника': {   'args': ['Домотехника'],
                             'comment': 'Изменения в составе прайсов',
                             'is_enabled': True,
                             'schedule': &lt;crontab: 41 3 0 * * (m/h/d/dM/MY)&gt;,
                             'task': 'celery_tacks.load_price_site',
                             'title': 'Цены с сайта Домотехника'},
    'prorab': {   'args': ['prorab'],
                  'comment': '',
                  'is_enabled': False,
                  'schedule': &lt;crontab: 7-59/30 * * * * (m/h/d/dM/MY)&gt;,
                  'task': 'celery_tacks.load_link',
                  'title': 'Prorab'},
    'quadro-stroy': {   'args': ['quadro-stroy'],
                        'comment': '',
                        'is_enabled': True,
                        'schedule': &lt;crontab: 17 1 * * * (m/h/d/dM/MY)&gt;,
                        'task': 'celery_tacks.load_link',
                        'title': 'Квадро-Строй'},
    'rotate_mb': {   'is_enabled': True,
                     'is_system': True,
                     'schedule': &lt;crontab: 23 0 0 * * (m/h/d/dM/MY)&gt;,
                     'task': 'celery_tacks.rotate_mb',
                     'title': 'Очистка устаревших файлов'},
    'sds-group': {   'args': ['sds-group'],
                     'comment': '',
                     'is_enabled': True,
                     'schedule': &lt;crontab: 29 0 * * * (m/h/d/dM/MY)&gt;,
                     'task': 'celery_tacks.load_link',
                     'title': 'СДС-Группа'},
    'sima-land': {   'args': ['sima-land'],
                     'comment': '',
                     'is_enabled': True,
                     'schedule': &lt;crontab: 53 0 * * * (m/h/d/dM/MY)&gt;,
                     'task': 'celery_tacks.load_site',
                     'title': 'Сима-ленд'},
    'sonex': {   'args': ['sonex'],
                 'comment': '',
                 'is_enabled': True,
                 'schedule': &lt;crontab: 11-59/30 * * * * (m/h/d/dM/MY)&gt;,
                 'task': 'celery_tacks.load_link',
                 'title': 'Сонекс'},
    'stluce': {   'args': ['stluce'],
                  'comment': 'Остатки ST Luce Velante',
                  'is_enabled': True,
                  'schedule': &lt;crontab: 19 1 * * * (m/h/d/dM/MY)&gt;,
                  'task': 'celery_tacks.load_link',
                  'title': 'ST Luce'},
    'stluce-diff': {   'args': ['stluce-diff'],
                       'comment': 'рассылка с изменениями прайса ST Luce '
                                  'Velante',
                       'is_enabled': True,
                       'schedule': &lt;crontab: 23 1 * * * (m/h/d/dM/MY)&gt;,
                       'task': 'celery_tacks.load_link',
                       'title': 'ST Luce'},
    'store_outofstock': {   'is_enabled': True,
                            'is_system': True,
                            'schedule': &lt;crontab: 49 0-20,23 * * * (m/h/d/dM/MY)&gt;,
                            'task': 'celery_tacks.store_outofstock',
                            'title': 'Автозаливка "нет в наличии"'},
    'technolight': {   'args': ['technolight'],
                       'comment': 'Отключено по просьбе заказчика',
                       'is_enabled': False,
                       'schedule': &lt;crontab: 19-59/30 * * * * (m/h/d/dM/MY)&gt;,
                       'task': 'celery_tacks.load_link',
                       'title': 'Тренд Света'},
    'triumph-nord-rus': {   'args': ['triumph-nord-rus'],
                            'comment': '',
                            'is_enabled': True,
                            'schedule': &lt;crontab: 11 1 * * * (m/h/d/dM/MY)&gt;,
                            'task': 'celery_tacks.load_link',
                            'title': 'Триумф Норд Рус'},
    'unisaw': {   'args': ['unisaw'],
                  'comment': '',
                  'is_enabled': True,
                  'schedule': &lt;crontab: 3 0 * * * (m/h/d/dM/MY)&gt;,
                  'task': 'celery_tacks.load_site',
                  'title': 'Unisaw'},
    'vodalux-fontan': {   'args': ['vodalux-fontan'],
                          'comment': '',
                          'is_enabled': True,
                          'schedule': &lt;crontab: 37 0 * * * (m/h/d/dM/MY)&gt;,
                          'task': 'celery_tacks.load_site',
                          'title': 'Водалюкс'},
    'vodalux-prud': {   'args': ['vodalux-prud'],
                        'comment': '',
                        'is_enabled': True,
                        'schedule': &lt;crontab: 13 0 * * * (m/h/d/dM/MY)&gt;,
                        'task': 'celery_tacks.load_site',
                        'title': 'Водалюкс'},
    'y-skazki': {   'args': ['y-skazki'],
                    'comment': 'Отключено по просьбе заказчика',
                    'is_enabled': False,
                    'schedule': &lt;crontab: 17 1 * * * (m/h/d/dM/MY)&gt;,
                    'task': 'celery_tacks.load_site',
                    'title': 'Bestway Юг'}}
</code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: 3.7.4</li>
<li><strong>Minimal Celery Version</strong>: 4.4.0</li>
<li><strong>Minimal Kombu Version</strong>: 4.6.7</li>
<li><strong>Minimal Broker Version</strong>: rabbitmq-server 3.6.15-1</li>
<li><strong>Minimal Result Backend Version</strong>: mongodb 4.2.0</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: ubuntu 18.04</li>
<li><strong>Minimal Broker Client Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="amqp==2.5.2
attrs==19.3.0
Automat==0.8.0
billiard==3.6.1.0
Brotli==1.0.7
cchardet==2.1.5
celery==4.4.0
certifi==2019.11.28
cffi==1.13.2
chardet==3.0.4
Click==7.0
clickhouse-cityhash==1.0.2.3
clickhouse-driver==0.1.2
constantly==15.1.0
cryptography==2.8
cssselect==1.1.0
dnspython==1.16.0
et-xmlfile==1.0.1
fake-useragent==0.1.11
fastnumbers==3.0.0
hyperlink==19.0.0
idna==2.8
importlib-metadata==1.3.0
incremental==17.5.0
jdcal==1.4.1
jmespath==0.9.4
jsonlines==1.2.0
kombu==4.6.7
lxml==4.4.2
lz4==3.0.2
marshmallow==2.20.5
more-itertools==8.0.2
mysqlclient==1.4.6
natsort==7.0.0
openpyxl==3.0.3
parsel==1.5.2
pip-tools==4.3.0
Protego==0.1.16
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.19
PyDispatcher==2.0.5
PyHamcrest==1.10.1
PyICU==2.4.2
pymongo==3.10.1
pyOpenSSL==19.1.0
python-dateutil==2.8.1
python-snappy==0.5.4
pytz==2019.3
pyxlsb==1.0.6
PyYAML==5.3
queuelib==1.5.0
requests==2.22.0
Scrapy==1.8.0
sentry-sdk==0.14.0
service-identity==18.1.0
six==1.14.0
Twisted==19.10.0
txmongo==19.2.0
tzlocal==2.0.0
urllib3==1.25.7
vine==1.3.0
w3lib==1.21.0
xlrd==1.2.0
XlsxWriter==1.2.7
zipp==0.6.0
zope.interface==4.7.1
zstandard==0.13.0
zstd==1.4.4.0"><pre class="notranslate"><code class="notranslate">amqp==2.5.2
attrs==19.3.0
Automat==0.8.0
billiard==3.6.1.0
Brotli==1.0.7
cchardet==2.1.5
celery==4.4.0
certifi==2019.11.28
cffi==1.13.2
chardet==3.0.4
Click==7.0
clickhouse-cityhash==1.0.2.3
clickhouse-driver==0.1.2
constantly==15.1.0
cryptography==2.8
cssselect==1.1.0
dnspython==1.16.0
et-xmlfile==1.0.1
fake-useragent==0.1.11
fastnumbers==3.0.0
hyperlink==19.0.0
idna==2.8
importlib-metadata==1.3.0
incremental==17.5.0
jdcal==1.4.1
jmespath==0.9.4
jsonlines==1.2.0
kombu==4.6.7
lxml==4.4.2
lz4==3.0.2
marshmallow==2.20.5
more-itertools==8.0.2
mysqlclient==1.4.6
natsort==7.0.0
openpyxl==3.0.3
parsel==1.5.2
pip-tools==4.3.0
Protego==0.1.16
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.19
PyDispatcher==2.0.5
PyHamcrest==1.10.1
PyICU==2.4.2
pymongo==3.10.1
pyOpenSSL==19.1.0
python-dateutil==2.8.1
python-snappy==0.5.4
pytz==2019.3
pyxlsb==1.0.6
PyYAML==5.3
queuelib==1.5.0
requests==2.22.0
Scrapy==1.8.0
sentry-sdk==0.14.0
service-identity==18.1.0
six==1.14.0
Twisted==19.10.0
txmongo==19.2.0
tzlocal==2.0.0
urllib3==1.25.7
vine==1.3.0
w3lib==1.21.0
xlrd==1.2.0
XlsxWriter==1.2.7
zipp==0.6.0
zope.interface==4.7.1
zstandard==0.13.0
zstd==1.4.4.0
</code></pre></div>
<p dir="auto"></p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
```
```
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>

<p dir="auto">Code for initialization celery app:</p>
<details>
<p dir="auto">
</p><div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from functools import partial
from bson import json_util

json_options=json_util.JSONOptions(
  json_mode=json_util.JSONMode.RELAXED, tz_aware=True, tzinfo=utc)
json_dump = partial(
  json_util.dumps, ensure_ascii=False, json_options=json_options)
json_load = partial(json_util.loads, json_options=json_options)

register(
  'mongo_json',
  encoder=(lambda obj: json_dump(obj).encode('utf-8')),
  decoder=json_load,
  content_type='application/x-mongo-json')

app = Celery(
  'remains_tasks',
  broker='pyamqp://...',
  backend = 'mongodb://...',
  include=[
    # many modules
  ]
)
app.conf.update(
  accept_content=[
    'mongo_json', 'application/x-mongo-json', 'json', 'application/json'],
  task_serializer='mongo_json',
  result_serializer='mongo_json',
  task_default_queue='remains_tasks',
  worker_prefetch_multiplier=1,
  worker_max_tasks_per_child=45,
  worker_hijack_root_logger=False,

  mongodb_backend_settings=dict(database='remains', compressors='snappy'),
)"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">functools</span> <span class="pl-k">import</span> <span class="pl-s1">partial</span>
<span class="pl-k">from</span> <span class="pl-s1">bson</span> <span class="pl-k">import</span> <span class="pl-s1">json_util</span>

<span class="pl-s1">json_options</span><span class="pl-c1">=</span><span class="pl-s1">json_util</span>.<span class="pl-v">JSONOptions</span>(
  <span class="pl-s1">json_mode</span><span class="pl-c1">=</span><span class="pl-s1">json_util</span>.<span class="pl-v">JSONMode</span>.<span class="pl-v">RELAXED</span>, <span class="pl-s1">tz_aware</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">tzinfo</span><span class="pl-c1">=</span><span class="pl-s1">utc</span>)
<span class="pl-s1">json_dump</span> <span class="pl-c1">=</span> <span class="pl-en">partial</span>(
  <span class="pl-s1">json_util</span>.<span class="pl-s1">dumps</span>, <span class="pl-s1">ensure_ascii</span><span class="pl-c1">=</span><span class="pl-c1">False</span>, <span class="pl-s1">json_options</span><span class="pl-c1">=</span><span class="pl-s1">json_options</span>)
<span class="pl-s1">json_load</span> <span class="pl-c1">=</span> <span class="pl-en">partial</span>(<span class="pl-s1">json_util</span>.<span class="pl-s1">loads</span>, <span class="pl-s1">json_options</span><span class="pl-c1">=</span><span class="pl-s1">json_options</span>)

<span class="pl-en">register</span>(
  <span class="pl-s">'mongo_json'</span>,
  <span class="pl-s1">encoder</span><span class="pl-c1">=</span>(<span class="pl-k">lambda</span> <span class="pl-s1">obj</span>: <span class="pl-en">json_dump</span>(<span class="pl-s1">obj</span>).<span class="pl-en">encode</span>(<span class="pl-s">'utf-8'</span>)),
  <span class="pl-s1">decoder</span><span class="pl-c1">=</span><span class="pl-s1">json_load</span>,
  <span class="pl-s1">content_type</span><span class="pl-c1">=</span><span class="pl-s">'application/x-mongo-json'</span>)

<span class="pl-s1">app</span> <span class="pl-c1">=</span> <span class="pl-v">Celery</span>(
  <span class="pl-s">'remains_tasks'</span>,
  <span class="pl-s1">broker</span><span class="pl-c1">=</span><span class="pl-s">'pyamqp://...'</span>,
  <span class="pl-s1">backend</span> <span class="pl-c1">=</span> <span class="pl-s">'mongodb://...'</span>,
  <span class="pl-s1">include</span><span class="pl-c1">=</span>[
    <span class="pl-c"># many modules</span>
  ]
)
<span class="pl-s1">app</span>.<span class="pl-s1">conf</span>.<span class="pl-en">update</span>(
  <span class="pl-s1">accept_content</span><span class="pl-c1">=</span>[
    <span class="pl-s">'mongo_json'</span>, <span class="pl-s">'application/x-mongo-json'</span>, <span class="pl-s">'json'</span>, <span class="pl-s">'application/json'</span>],
  <span class="pl-s1">task_serializer</span><span class="pl-c1">=</span><span class="pl-s">'mongo_json'</span>,
  <span class="pl-s1">result_serializer</span><span class="pl-c1">=</span><span class="pl-s">'mongo_json'</span>,
  <span class="pl-s1">task_default_queue</span><span class="pl-c1">=</span><span class="pl-s">'remains_tasks'</span>,
  <span class="pl-s1">worker_prefetch_multiplier</span><span class="pl-c1">=</span><span class="pl-c1">1</span>,
  <span class="pl-s1">worker_max_tasks_per_child</span><span class="pl-c1">=</span><span class="pl-c1">45</span>,
  <span class="pl-s1">worker_hijack_root_logger</span><span class="pl-c1">=</span><span class="pl-c1">False</span>,

  <span class="pl-s1">mongodb_backend_settings</span><span class="pl-c1">=</span><span class="pl-en">dict</span>(<span class="pl-s1">database</span><span class="pl-c1">=</span><span class="pl-s">'remains'</span>, <span class="pl-s1">compressors</span><span class="pl-c1">=</span><span class="pl-s">'snappy'</span>),
)</pre></div>
<p dir="auto">Connection string look:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="mongodb+srv://username:password@mongo.example.com/database?ssl=false"><pre class="notranslate"><code class="notranslate">mongodb+srv://username:password@mongo.example.com/database?ssl=false
</code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>

<p dir="auto">No exception</p>
<h1 dir="auto">Actual Behavior</h1>

<p dir="auto">Exception after exit tack</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="KeyError: 'collection'
  File &quot;kombu/utils/objects.py&quot;, line 42, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'database'
  File &quot;kombu/utils/objects.py&quot;, line 42, in __get__
    return obj.__dict__[self.__name__]
OperationFailure: Authentication failed.
  File &quot;billiard/pool.py&quot;, line 1791, in safe_apply_callback
    fun(*args, **kwargs)
  File &quot;celery/worker/request.py&quot;, line 526, in on_failure
    self.task.backend.mark_as_failure(
  File &quot;celery/backends/base.py&quot;, line 159, in mark_as_failure
    self.store_result(task_id, exc, state,
  File &quot;celery/backends/base.py&quot;, line 406, in store_result
    self._store_result(task_id, result, state, traceback,
  File &quot;celery/backends/mongodb.py&quot;, line 194, in _store_result
    self.collection.replace_one({'_id': task_id}, meta, upsert=True)
  File &quot;kombu/utils/objects.py&quot;, line 44, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File &quot;celery/backends/mongodb.py&quot;, line 291, in collection
    collection = self.database[self.taskmeta_collection]
  File &quot;kombu/utils/objects.py&quot;, line 44, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File &quot;celery/backends/mongodb.py&quot;, line 286, in database
    return self._get_database()
  File &quot;celery/backends/mongodb.py&quot;, line 275, in _get_database
    if not db.authenticate(self.user, self.password, source=source):
  File &quot;pymongo/database.py&quot;, line 1468, in authenticate
    self.client._cache_credentials(
  File &quot;pymongo/mongo_client.py&quot;, line 754, in _cache_credentials
    with server.get_socket(all_credentials) as sock_info:
  File &quot;contextlib.py&quot;, line 113, in __enter__
    return next(self.gen)
  File &quot;pymongo/pool.py&quot;, line 1138, in get_socket
    sock_info.check_auth(all_credentials)
  File &quot;pymongo/pool.py&quot;, line 712, in check_auth
    auth.authenticate(credentials, self)
  File &quot;pymongo/auth.py&quot;, line 564, in authenticate
    auth_func(credentials, sock_info)
  File &quot;pymongo/auth.py&quot;, line 539, in _authenticate_default
    return _authenticate_scram(credentials, sock_info, 'SCRAM-SHA-1')
  File &quot;pymongo/auth.py&quot;, line 263, in _authenticate_scram
    res = sock_info.command(source, cmd)
  File &quot;pymongo/pool.py&quot;, line 603, in command
    return command(self.sock, dbname, spec, slave_ok,
  File &quot;pymongo/network.py&quot;, line 165, in command
    helpers._check_command_response(
  File &quot;pymongo/helpers.py&quot;, line 159, in _check_command_response
    raise OperationFailure(msg % errmsg, code, response)"><pre class="notranslate"><code class="notranslate">KeyError: 'collection'
  File "kombu/utils/objects.py", line 42, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'database'
  File "kombu/utils/objects.py", line 42, in __get__
    return obj.__dict__[self.__name__]
OperationFailure: Authentication failed.
  File "billiard/pool.py", line 1791, in safe_apply_callback
    fun(*args, **kwargs)
  File "celery/worker/request.py", line 526, in on_failure
    self.task.backend.mark_as_failure(
  File "celery/backends/base.py", line 159, in mark_as_failure
    self.store_result(task_id, exc, state,
  File "celery/backends/base.py", line 406, in store_result
    self._store_result(task_id, result, state, traceback,
  File "celery/backends/mongodb.py", line 194, in _store_result
    self.collection.replace_one({'_id': task_id}, meta, upsert=True)
  File "kombu/utils/objects.py", line 44, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "celery/backends/mongodb.py", line 291, in collection
    collection = self.database[self.taskmeta_collection]
  File "kombu/utils/objects.py", line 44, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "celery/backends/mongodb.py", line 286, in database
    return self._get_database()
  File "celery/backends/mongodb.py", line 275, in _get_database
    if not db.authenticate(self.user, self.password, source=source):
  File "pymongo/database.py", line 1468, in authenticate
    self.client._cache_credentials(
  File "pymongo/mongo_client.py", line 754, in _cache_credentials
    with server.get_socket(all_credentials) as sock_info:
  File "contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "pymongo/pool.py", line 1138, in get_socket
    sock_info.check_auth(all_credentials)
  File "pymongo/pool.py", line 712, in check_auth
    auth.authenticate(credentials, self)
  File "pymongo/auth.py", line 564, in authenticate
    auth_func(credentials, sock_info)
  File "pymongo/auth.py", line 539, in _authenticate_default
    return _authenticate_scram(credentials, sock_info, 'SCRAM-SHA-1')
  File "pymongo/auth.py", line 263, in _authenticate_scram
    res = sock_info.command(source, cmd)
  File "pymongo/pool.py", line 603, in command
    return command(self.sock, dbname, spec, slave_ok,
  File "pymongo/network.py", line 165, in command
    helpers._check_command_response(
  File "pymongo/helpers.py", line 159, in _check_command_response
    raise OperationFailure(msg % errmsg, code, response)
</code></pre></div>
<p dir="auto">PR for proposed fix to this issue: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="592251302" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/6020" data-hovercard-type="pull_request" data-hovercard-url="/celery/celery/pull/6020/hovercard" href="https://github.com/celery/celery/pull/6020">#6020</a></p>
<h1 dir="auto">Checklist</h1>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> This has already been asked to the <a href="https://groups.google.com/forum/#!forum/celery-users" rel="nofollow">discussion group</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>
<p dir="auto"><a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/celery/celery/commit/94dae1b899aae6ae2ca333773fddbc6dd603213c/hovercard" href="https://github.com/celery/celery/commit/94dae1b899aae6ae2ca333773fddbc6dd603213c"><tt>94dae1b</tt></a><br>
This PR was made to address the following issue, which has resulted in the issue I am having now.   <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="30048263" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/1930" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/1930/hovercard" href="https://github.com/celery/celery/issues/1930">#1930</a></p>
<h4 dir="auto">Related Issues</h4>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="30048263" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/1930" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/1930/hovercard" href="https://github.com/celery/celery/issues/1930">#1930</a></p>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>: celery&gt;=4.0.0 (using it in Airflow)</p>
<p dir="auto"></p>

<h1 dir="auto">Steps to Reproduce</h1>
<p dir="auto">(see Minimally Reproducible Test Case for step by step commands. This contains information leading to the issue and a proposed fix)</p>
<p dir="auto">In Airflow, you can set celery configs. I was setting up cloudsql to use a private IP instead of a proxy. Currently, we use mysql as the <code class="notranslate">results_backend</code>. Changing the host address from local host to the private IP caused some errors, as expected.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="OperationalError: (_mysql_exceptions.OperationalError) (1045, &quot;Access denied for user 'airflow'@'10.x.x.xxx' (using password: YES)&quot;)"><pre class="notranslate"><code class="notranslate">OperationalError: (_mysql_exceptions.OperationalError) (1045, "Access denied for user 'airflow'@'10.x.x.xxx' (using password: YES)")
</code></pre></div>
<p dir="auto">In order to use the private IP, I need to use the SSL cert, key, and ca. I confirmed that by logging into the Airflow worker and scheduler pods that my url and engine arg params worked.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from airflow.models import DagRun 
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
e = create_engine({AIRFLOW__CELERY__SQL_ALCHEMY_CONN},connect_args= {'ssl': {'ca': '/path-to-mysql-sslcert/server-ca', 'cert': '/path-to-mysql-sslcert/client-cert',  'key': '/path-to-mysql-sslcert/client-key'}})
s = scoped_session(sessionmaker(autocommit=False, autoflush=False, bind=e))
s.query(DagRun).all()"><pre class="notranslate"><code class="notranslate">from airflow.models import DagRun 
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
e = create_engine({AIRFLOW__CELERY__SQL_ALCHEMY_CONN},connect_args= {'ssl': {'ca': '/path-to-mysql-sslcert/server-ca', 'cert': '/path-to-mysql-sslcert/client-cert',  'key': '/path-to-mysql-sslcert/client-key'}})
s = scoped_session(sessionmaker(autocommit=False, autoflush=False, bind=e))
s.query(DagRun).all()
</code></pre></div>
<p dir="auto">This worked fine, so I know that the my ssl certs are accessible, the engine can be created, and a session used. Non-celery mysql connections no longer gave an error.</p>
<p dir="auto">The Celery documentation (<a href="https://docs.celeryproject.org/en/stable/userguide/configuration.html#conf-database-result-backend" rel="nofollow">https://docs.celeryproject.org/en/stable/userguide/configuration.html#conf-database-result-backend</a>) outlines how to add engine args to via <code class="notranslate">database_engine_options</code>. Therefore, I added</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="'database_engine_options': {
        'connect_args': {'ssl': {'ca': '/path-to-mysql-sslcert/server-ca', 'cert': '/path-to-mysql-sslcert/client-cert', 'key': '/path-to-mysql-sslcert/client-key'}}}"><pre class="notranslate"><code class="notranslate">'database_engine_options': {
        'connect_args': {'ssl': {'ca': '/path-to-mysql-sslcert/server-ca', 'cert': '/path-to-mysql-sslcert/client-cert', 'key': '/path-to-mysql-sslcert/client-key'}}}
</code></pre></div>
<p dir="auto">However, I still get the same error.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="OperationalError: (_mysql_exceptions.OperationalError) (1045, &quot;Access denied for user 'airflow'@'10.x.x.xxx' (using password: YES)&quot;)"><pre class="notranslate"><code class="notranslate">OperationalError: (_mysql_exceptions.OperationalError) (1045, "Access denied for user 'airflow'@'10.x.x.xxx' (using password: YES)")
</code></pre></div>
<p dir="auto">Additionally, I get logs in the scheduler like the following:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" {{__init__.py:56}} WARNING - Failed operation _get_task_meta_for.  Retrying 1 more times.
68918-Traceback (most recent call last):
68919-  File &quot;/usr/local/lib/python2.7/dist-packages/celery/backends/database/__init__.py&quot;, line 51, in _inner
68920-    return fun(*args, **kwargs)
68921-  File &quot;/usr/local/lib/python2.7/dist-packages/celery/backends/database/__init__.py&quot;, line 154, in _get_task_meta_for
68922:    session = self.ResultSession()
68923:  File &quot;/usr/local/lib/python2.7/dist-packages/celery/backends/database/__init__.py&quot;, line 113, in ResultSession
68924-    **self.engine_options)
68925-  File &quot;/usr/local/lib/python2.7/dist-packages/celery/backends/database/session.py&quot;, line 59, in session_factory
68926-    self.prepare_models(engine)
68927-  File &quot;/usr/local/lib/python2.7/dist-packages/celery/backends/database/session.py&quot;, line 54, in prepare_models
68928-    ResultModelBase.metadata.create_all(engine)"><pre class="notranslate"><code class="notranslate"> {{__init__.py:56}} WARNING - Failed operation _get_task_meta_for.  Retrying 1 more times.
68918-Traceback (most recent call last):
68919-  File "/usr/local/lib/python2.7/dist-packages/celery/backends/database/__init__.py", line 51, in _inner
68920-    return fun(*args, **kwargs)
68921-  File "/usr/local/lib/python2.7/dist-packages/celery/backends/database/__init__.py", line 154, in _get_task_meta_for
68922:    session = self.ResultSession()
68923:  File "/usr/local/lib/python2.7/dist-packages/celery/backends/database/__init__.py", line 113, in ResultSession
68924-    **self.engine_options)
68925-  File "/usr/local/lib/python2.7/dist-packages/celery/backends/database/session.py", line 59, in session_factory
68926-    self.prepare_models(engine)
68927-  File "/usr/local/lib/python2.7/dist-packages/celery/backends/database/session.py", line 54, in prepare_models
68928-    ResultModelBase.metadata.create_all(engine)
</code></pre></div>
<p dir="auto">After digging through the code with <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/dangermike/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/dangermike">@dangermike</a>, we noticed that <code class="notranslate">get_engine</code> will not use the kwargs passed to it unless it has been forked.(<a href="https://github.com/celery/celery/blob/master/celery/backends/database/session.py#L34">https://github.com/celery/celery/blob/master/celery/backends/database/session.py#L34</a>) Therefore, the SSL params will not be passed in our case. The only place that self.forked = True is after the fork cleanup session. This used to not be the case (<a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/celery/celery/commit/94dae1b899aae6ae2ca333773fddbc6dd603213c/hovercard" href="https://github.com/celery/celery/commit/94dae1b899aae6ae2ca333773fddbc6dd603213c"><tt>94dae1b</tt></a>), but after an issue was made about passing pool_size (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="30048263" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/1930" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/1930/hovercard" href="https://github.com/celery/celery/issues/1930">#1930</a>), <code class="notranslate">**kwargs</code> were taken out of create_engine() entirely.<br>
Possibly something like the following would allow for kwargs to be passed in, while still addressing the pool params issue.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="class SessionManager(object):
    # ...
    def get_engine(self, dburi, **kwargs):
        if self.forked:
            try:
                return self._engines[dburi]
            except KeyError:
                engine = self._engines[dburi] = create_engine(dburi, **kwargs)
                return engine
        else:
            kwargs = dict([(k, v) for k, v in kwargs.items() if not k.startswith('pool')])
            return create_engine(dburi, poolclass=NullPool, **kwargs)"><pre class="notranslate"><code class="notranslate">class SessionManager(object):
    # ...
    def get_engine(self, dburi, **kwargs):
        if self.forked:
            try:
                return self._engines[dburi]
            except KeyError:
                engine = self._engines[dburi] = create_engine(dburi, **kwargs)
                return engine
        else:
            kwargs = dict([(k, v) for k, v in kwargs.items() if not k.startswith('pool')])
            return create_engine(dburi, poolclass=NullPool, **kwargs)
</code></pre></div>
<p dir="auto">where <code class="notranslate">kwargs = dict([(k, v) for k, v in kwargs.items() if not k.startswith('pool')])</code> omits any pool args while keeping the rest.</p>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: &gt;=2.7</li>
<li><strong>Minimal Celery Version</strong>: &gt;=4.0.0</li>
<li><strong>Minimal Kombu Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Version</strong>: N/A or Unknown</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Client Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>
<p dir="auto">Used Airflow</p>
<h3 dir="auto">Other Dependencies</h3>
<p dir="auto">N/A</p>
<h2 dir="auto">Minimally Reproducible Test Case</h2>
<p dir="auto">In a python shell,</p>
<p dir="auto">get the url with a private mysql IP to make result_backend, giving something like <code class="notranslate">db+mysql://airflow:***@10.x.xx.xx/airflow</code></p>
<p dir="auto">and the celery config</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="celery_configuration = 
{'broker_transport_options': {'visibility_timeout': 21600},
 'result_serializer': 'pickle',
 'task_acks_late': True, 
'database_engine_options': { 'connect_args': {'ssl': {'ca': '/path-to-mysql-sslcert/server-ca', 'cert': '/path-to-mysql-sslcert/client-cert', 'key': '/path-to-mysql-sslcert/client-key'}}}, 
'task_default_queue': 'default',
 'worker_concurrency': 32,
 'worker_prefetch_multiplier': 1, 
'event_serializer': 'json', 
'accept_content': ['json', 'pickle'], 
'broker_url': 'redis://{URL}/1', 
'result_backend': 'db+mysql://airflow:***@10.x.xx.xx/airflow', 
'task_default_exchange': 'default'}"><pre class="notranslate"><code class="notranslate">celery_configuration = 
{'broker_transport_options': {'visibility_timeout': 21600},
 'result_serializer': 'pickle',
 'task_acks_late': True, 
'database_engine_options': { 'connect_args': {'ssl': {'ca': '/path-to-mysql-sslcert/server-ca', 'cert': '/path-to-mysql-sslcert/client-cert', 'key': '/path-to-mysql-sslcert/client-key'}}}, 
'task_default_queue': 'default',
 'worker_concurrency': 32,
 'worker_prefetch_multiplier': 1, 
'event_serializer': 'json', 
'accept_content': ['json', 'pickle'], 
'broker_url': 'redis://{URL}/1', 
'result_backend': 'db+mysql://airflow:***@10.x.xx.xx/airflow', 
'task_default_exchange': 'default'}
</code></pre></div>
<p dir="auto">the line most important here is:<br>
<code class="notranslate"> 'database_engine_options': { 'connect_args': {'ssl': {'ca': '/path-to-mysql-sslcert/server-ca', 'cert': '/path-to-mysql-sslcert/client-cert', 'key': '/path-to-mysql-sslcert/client-key'}}}</code></p>
<p dir="auto">then try to connect to result_backend by creating app.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="app = Celery(celery_app_name=airflow.executors.celery_executor,
    config_source=celery_configuration)"><pre class="notranslate"><code class="notranslate">app = Celery(celery_app_name=airflow.executors.celery_executor,
    config_source=celery_configuration)
</code></pre></div>
<p dir="auto">create a database backend</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="dbbe = database.DatabaseBackend(url={results_backend url without the 'db+' in the beginning}, engine_options=celery_configuration['database_engine_options'], app=app)"><pre class="notranslate"><code class="notranslate">dbbe = database.DatabaseBackend(url={results_backend url without the 'db+' in the beginning}, engine_options=celery_configuration['database_engine_options'], app=app)
</code></pre></div>
<p dir="auto">and you will get the error again</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1045, &quot;Access denied for user 'airflow'@'10.xx.xx.xxx' (using password: YES)&quot;)
(Background on this error at: http://sqlalche.me/e/e3q8)"><pre class="notranslate"><code class="notranslate">sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1045, "Access denied for user 'airflow'@'10.xx.xx.xxx' (using password: YES)")
(Background on this error at: http://sqlalche.me/e/e3q8)
</code></pre></div>
<h1 dir="auto">Expected Behavior</h1>
<p dir="auto">It seems like the expected behavior here would be for the connection to be successful and use the SSL certs in the **kwargs passed into <code class="notranslate">get_engine</code>.</p>
<h1 dir="auto">Actual Behavior</h1>
<p dir="auto">Since self.fork is not True, and will not be True, create_engine is made by:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="            return create_engine(dburi, poolclass=NullPool)"><pre class="notranslate"><code class="notranslate">            return create_engine(dburi, poolclass=NullPool)
</code></pre></div>
<p dir="auto">since the SSL certs are not included, an error is returned and the connection is <em>not</em> successful.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1045, &quot;Access denied for user 'airflow'@'10.xx.xx.xxx' (using password: YES)&quot;)
(Background on this error at: http://sqlalche.me/e/e3q8)"><pre class="notranslate"><code class="notranslate">sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1045, "Access denied for user 'airflow'@'10.xx.xx.xxx' (using password: YES)")
(Background on this error at: http://sqlalche.me/e/e3q8)
</code></pre></div>