title: Airflow with celery executor for memory hungry Dags
body: <p>We need to execute 3000 Dag runs in a specific hour of the day.
  almost all the work inside the Dag is waiting for other micro services to perform their work. (for example AWS sagemaker)</p>
  
  <p>But our Dag consume about 250MB due to all the python imports we need (sagemaker,tensorflow etc)
  we noticed that the 250MB memory is consumed even before the first task is executed. ( checked that by printing <code>process.memory_full_info()</code>)</p>
  
  <p>we have tried to run 8 concurrent runs on a Celery cluster which have 2 workers (4GB ram each) and <code>worker_concurrency=8</code> and failed.
  the worker machine crashed for OOM error.</p>
  
  <p>drilling down into this i understand that airflow actually work in prefork method , meaning it creates duplicates of its main worker process for each sub worker it initiates. this subprocess consume more or less the memory our Dag consumes (around 300MB) , there is no memory sharing between the sub processes.</p>
  
  <p>so i have started to search for a solution since this one obviously cannot scale unless we put something like 500 workers. (each can take 6 dag runs concurrently from our tests)</p>
  
  <p>i read about different pool worker classes that can be set like <code>gevent</code> or <code>eventlet</code> and how you can reuse the memory so not every sub worker will duplicate its memory .i thought i have found the solution i need.
  but then i have tried to set this up and didn't see this behavior in work - i would only see 2 concurrent tasks being executed (with gevent configured)
  in addition - our airflow cloud provider explain to me that this is not airflow is working and in its base executor class it still create a subprocess for each task.
  this is the reference he sent me  <a href="https://github.com/apache/airflow/blob/919bb8c1cbb36679c02ca3f8890c300e1527c08b/airflow/task/task_runner/base_task_runner.py#L112-L142" rel="nofollow noreferrer">https://github.com/apache/airflow/blob/919bb8c1cbb36679c02ca3f8890c300e1527c08b/airflow/task/task_runner/base_task_runner.py#L112-L142</a></p>
  
  <p>my questions are:</p>
  
  <ol>
  <li>is it possible to scale the way we need , given our Dag consume much memory to begin with ? </li>
  <li>does working with Celery executor actually can benefit memory hungry dag runs.</li>
  <li>Is the cloud provider right by saying - "this is not how airflow works with celery "</li>
  <li>i'm hearing people run thousands of tasks concurrently , even with minimum DAG import i get 130MB memory usage - how can this scale to thousands ?</li>
  </ol>
  
  <p>thanks</p>