title: Error with celery worker in elastic beanstalk (using django and SQS) [ImportError: The curl client requires the pycurl library.]
body: <p>I'm trying to deploy to elastic beanstalk a django project that uses celery periodic tasks, using SQS.</p>
  
  <p>I've been more or less following the instructions here:</p>
  
  <p><a href="https://stackoverflow.com/questions/41161691/how-to-run-a-celery-worker-with-django-app-scalable-by-aws-elastic-beanstalk?noredirect=1&amp;lq=1">How to run a celery worker with Django app scalable by AWS Elastic Beanstalk?</a></p>
  
  <p>When I deploy to eb, the periodic tasks are not being executed.</p>
  
  <p>Checking the celery-beat log, everything seems right:</p>
  
  <pre><code>celery beat v4.2.1 (windowlicker) is starting.
  __    -    ... __   -        _
  LocalTime -&gt; 2019-01-27 09:48:16
  Configuration -&gt;
      . broker -&gt; sqs://AKIAIVCNK32ABCHNNZSQ:**@localhost//
      . loader -&gt; celery.loaders.app.AppLoader
      . scheduler -&gt; django_celery_beat.schedulers.DatabaseScheduler
  
      . logfile -&gt; [stderr]@%INFO
      . maxinterval -&gt; 5.00 seconds (5s)
  /opt/python/run/venv/local/lib64/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: &lt;http://initd.org/psycopg/docs/install.html#binary-install-from-pypi&gt;.
    """)
  [2019-01-27 09:48:44,659: INFO/MainProcess] beat: Starting...
  [2019-01-27 09:48:44,660: INFO/MainProcess] Writing entries...
  [2019-01-27 09:48:44,809: INFO/MainProcess] DatabaseScheduler: Schedule changed.
  [2019-01-27 09:48:44,809: INFO/MainProcess] Writing entries...
  [2019-01-27 09:48:49,865: INFO/MainProcess] Writing entries...
  [2019-01-27 09:49:00,409: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)
  [2019-01-27 09:50:00,050: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)
  [2019-01-27 09:51:00,045: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)
  [2019-01-27 09:51:50,543: INFO/MainProcess] Writing entries...
  [2019-01-27 09:52:00,048: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)
  [2019-01-27 09:53:00,045: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)
  </code></pre>
  
  <p>but there seems to be some problem with the celery worker. This is the error I get in celery-worker.log:</p>
  
  <pre><code>/opt/python/run/venv/local/lib64/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: &lt;http://initd.org/psycopg/docs/install.html#binary-install-from-pypi&gt;.
    """)
  [2019-01-27 09:49:12,513: INFO/MainProcess] Connected to sqs://AKIAIVCNK32ABCHNNZSQ:**@localhost//
  [2019-01-27 09:49:12,950: WARNING/MainProcess] /opt/python/run/venv/local/lib/python3.6/site-packages/celery/fixups/django.py:200: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
    warnings.warn('Using settings.DEBUG leads to a memory leak, never '
  [2019-01-27 09:49:12,951: INFO/MainProcess] celery@ip-172-31-23-20 ready.
  [2019-01-27 09:49:12,952: CRITICAL/MainProcess] Unrecoverable error: ImportError('The curl client requires the pycurl library.',)
  Traceback (most recent call last):
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/http/__init__.py", line 20, in get_client
      return hub._current_http_client
  AttributeError: 'Hub' object has no attribute '_current_http_client'
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/worker/worker.py", line 205, in start
      self.blueprint.start(self)
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
      step.start(parent)
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bootsteps.py", line 369, in start
      return self.obj.start()
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 317, in start
      blueprint.start(self)
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
      step.start(parent)
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 593, in start
      c.loop(*c.loop_args())
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/worker/loops.py", line 91, in asynloop
      next(loop)
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 291, in create_loop
      item()
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/vine/promises.py", line 163, in __call__
      return self.throw()
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/vine/promises.py", line 160, in __call__
      retval = fun(*final_args, **final_kwargs)
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 316, in _schedule_queue
      queue, callback=promise(self._loop1, (queue,)),
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 332, in _get_bulk_async
      return self._get_async(queue, maxcount, callback=callback)
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 342, in _get_async
      qname, count=count, connection=self.asynsqs,
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 436, in asynsqs
      region=self.region
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/aws/sqs/connection.py", line 27, in __init__
      **kwargs
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/aws/connection.py", line 178, in __init__
      **http_client_params)
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/aws/connection.py", line 151, in __init__
      self._httpclient = http_client or get_client()
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/http/__init__.py", line 22, in get_client
      client = hub._current_http_client = Client(hub, **kwargs)
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/http/__init__.py", line 13, in Client
      return CurlClient(hub, **kwargs)
    File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/http/curl.py", line 43, in __init__
      raise ImportError('The curl client requires the pycurl library.')
  ImportError: The curl client requires the pycurl library.
  
   -------------- celery@ip-172-31-23-20 v4.2.1 (windowlicker)
  ---- **** -----
  --- * ***  * -- Linux-4.14.88-72.76.amzn1.x86_64-x86_64-with-glibc2.3.4 2019-01-27 09:49:12
  -- * - **** ---
  - ** ---------- [config]
  - ** ---------- .&gt; app:         raiseflags:0x7fbf06309f28
  - ** ---------- .&gt; transport:   sqs://AKIAIVCNK32ABCHNNZSQ:**@localhost//
  - ** ---------- .&gt; results:
  - *** --- * --- .&gt; concurrency: 1 (prefork)
  -- ******* ---- .&gt; task events: OFF (enable -E to monitor tasks in this worker)
  --- ***** -----
   -------------- [queues]
                  .&gt; celery           exchange=celery(direct) key=celery
  
  
  [tasks]
    . check_users_inboxes
    . periodic_send
    . raiseflags.celery.debug_task
    . scrapeURLs
    . spreadsheets_update
    . sum_two_numbers
  </code></pre>
  
  <p>I've been searching for answers and some people seem to have the same problem. I've tried deploying different versions of celery and kombu but can't solve it. </p>
  
  <p>This is my requirements.txt file:</p>
  
  <pre><code>amqp==2.4.0
  anyjson==0.3.3
  beautifulsoup4==4.7.1
  billiard==3.5.0.5
  boto==2.49.0
  bs4==0.0.1
  cachetools==3.0.0
  celery==4.2.1
  certifi==2018.11.29
  chardet==3.0.4
  Django==2.1.5
  django-celery-beat==1.4.0
  django-celery-results==1.0.4
  django-timezone-field==3.0
  google-api-python-client==1.7.7
  google-auth==1.6.2
  google-auth-httplib2==0.0.3
  httplib2==0.12.0
  idna==2.8
  kombu==4.2.2.post1
  oauth2client==4.1.3
  psycopg2==2.7.7
  pyasn1==0.4.5
  pyasn1-modules==0.2.3
  python-dateutil==2.7.5
  pycurl==7.43.0 --global-option="--with-nss"
  python-crontab==2.3.6
  pytz==2018.9
  requests==2.21.0
  rsa==4.0
  six==1.12.0
  soupsieve==1.7.3
  stripe==2.18.1
  uritemplate==3.0.0
  urllib3==1.24.1
  vine==1.2.0
  </code></pre>
  
  <p>If I ssh connect to the instance and type "python -c 'import pycurl'" everything seems right, no problem there.</p>
  
  <p>Does someone know how to solve it? Any help would be appreciated.</p>
  
  <p>Thank you very much.</p>