<h3 dir="auto">Overview</h3>
<p dir="auto">Some Executors, currently a subset of the local Executors, run in a single threaded fashion and have certain limitations and requirements, many of which are hardcoded. To add a new single threaded Executor would require changes to core Airflow code.<br>
Note: This coupling often shows up with SQLite compatibility checks since it does not support multiple connections.</p>
<h3 dir="auto">Examples</h3>
<ul dir="auto">
<li><strong>2a</strong>) SQLite check done in configuration.py: <div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/apache/airflow/blob/26f94c5370587f73ebd935cecf208c6a36bdf9b6/airflow/configuration.py#L412-L419">airflow/airflow/configuration.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
        Lines 412 to 419
      in
      <a data-pjax="true" class="commit-tease-sha" href="/apache/airflow/commit/26f94c5370587f73ebd935cecf208c6a36bdf9b6">26f94c5</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L412" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="412"></td>
          <td id="LC412" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">is_executor_without_sqlite_support</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">get</span>(<span class="pl-s">"core"</span>, <span class="pl-s">"executor"</span>) <span class="pl-c1">not</span> <span class="pl-c1">in</span> ( </td>
        </tr>

        <tr class="border-0">
          <td id="L413" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="413"></td>
          <td id="LC413" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s">'DebugExecutor'</span>, </td>
        </tr>

        <tr class="border-0">
          <td id="L414" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="414"></td>
          <td id="LC414" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s">'SequentialExecutor'</span>, </td>
        </tr>

        <tr class="border-0">
          <td id="L415" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="415"></td>
          <td id="LC415" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> ) </td>
        </tr>

        <tr class="border-0">
          <td id="L416" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="416"></td>
          <td id="LC416" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">is_sqlite</span> <span class="pl-c1">=</span> <span class="pl-s">"sqlite"</span> <span class="pl-c1">in</span> <span class="pl-s1">self</span>.<span class="pl-en">get</span>(<span class="pl-s">'database'</span>, <span class="pl-s">'sql_alchemy_conn'</span>) </td>
        </tr>

        <tr class="border-0">
          <td id="L417" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="417"></td>
          <td id="LC417" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">if</span> <span class="pl-s1">is_sqlite</span> <span class="pl-c1">and</span> <span class="pl-s1">is_executor_without_sqlite_support</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L418" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="418"></td>
          <td id="LC418" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-k">raise</span> <span class="pl-v">AirflowConfigException</span>(<span class="pl-s">f"error: cannot use sqlite with the <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">self</span>.<span class="pl-en">get</span>(<span class="pl-s">'core'</span>, <span class="pl-s">'executor'</span>)<span class="pl-kos">}</span></span>"</span>) </td>
        </tr>

        <tr class="border-0">
          <td id="L419" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="419"></td>
          <td id="LC419" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">if</span> <span class="pl-s1">is_sqlite</span>: </td>
        </tr>
    </tbody></table>
  </div>
</div>
</li>
<li><strong>2b</strong>) When running in standalone mode SQLite compatibility is checked: <div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/apache/airflow/blob/26f94c5370587f73ebd935cecf208c6a36bdf9b6/airflow/cli/commands/standalone_command.py#L160-L165">airflow/airflow/cli/commands/standalone_command.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
        Lines 160 to 165
      in
      <a data-pjax="true" class="commit-tease-sha" href="/apache/airflow/commit/26f94c5370587f73ebd935cecf208c6a36bdf9b6">26f94c5</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L160" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="160"></td>
          <td id="LC160" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">if</span> <span class="pl-s">"sqlite"</span> <span class="pl-c1">in</span> <span class="pl-s1">conf</span>.<span class="pl-en">get</span>(<span class="pl-s">"database"</span>, <span class="pl-s">"sql_alchemy_conn"</span>): </td>
        </tr>

        <tr class="border-0">
          <td id="L161" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="161"></td>
          <td id="LC161" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">self</span>.<span class="pl-en">print_output</span>(<span class="pl-s">"standalone"</span>, <span class="pl-s">"Forcing executor to SequentialExecutor"</span>) </td>
        </tr>

        <tr class="border-0">
          <td id="L162" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="162"></td>
          <td id="LC162" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">env</span>[<span class="pl-s">"AIRFLOW__CORE__EXECUTOR"</span>] <span class="pl-c1">=</span> <span class="pl-s1">executor_constants</span>.<span class="pl-v">SEQUENTIAL_EXECUTOR</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L163" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="163"></td>
          <td id="LC163" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">else</span>: </td>
        </tr>

        <tr class="border-0">
          <td id="L164" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="164"></td>
          <td id="LC164" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">self</span>.<span class="pl-en">print_output</span>(<span class="pl-s">"standalone"</span>, <span class="pl-s">"Forcing executor to LocalExecutor"</span>) </td>
        </tr>

        <tr class="border-0">
          <td id="L165" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="165"></td>
          <td id="LC165" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">env</span>[<span class="pl-s">"AIRFLOW__CORE__EXECUTOR"</span>] <span class="pl-c1">=</span> <span class="pl-s1">executor_constants</span>.<span class="pl-v">LOCAL_EXECUTOR</span> </td>
        </tr>
    </tbody></table>
  </div>
</div>
</li>
<li><strong>2c</strong>) Sensors in <code class="notranslate">poke</code> mode can block execution of DAGs when running with single process Executors, currently hardcoded to DebugExecutor (although should also include SequentialExecutor): <div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/apache/airflow/blob/27e2101f6ee5567b2843cbccf1dca0b0e7c96186/airflow/sensors/base.py#L243">airflow/airflow/sensors/base.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 243
      in
      <a data-pjax="true" class="commit-tease-sha" href="/apache/airflow/commit/27e2101f6ee5567b2843cbccf1dca0b0e7c96186">27e2101</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L243" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="243"></td>
          <td id="LC243" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">if</span> <span class="pl-s1">conf</span>.<span class="pl-en">get</span>(<span class="pl-s">'core'</span>, <span class="pl-s">'executor'</span>) <span class="pl-c1">==</span> <span class="pl-s">"DebugExecutor"</span>: </td>
        </tr>
    </tbody></table>
  </div>
</div>
</li>
</ul>
<h3 dir="auto">Proposal</h3>
<p dir="auto">A static method or attribute on the Executor class which can be checked by core code.</p>
<p dir="auto">There is a precedent already set with the <code class="notranslate">supports_ad_hoc_ti_run</code> attribute, see:<br>
</p><div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/apache/airflow/blob/fb741fd87254e235f99d7d67e558dafad601f253/airflow/executors/kubernetes_executor.py#L435">airflow/airflow/executors/kubernetes_executor.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
         Line 435
      in
      <a data-pjax="true" class="commit-tease-sha" href="/apache/airflow/commit/fb741fd87254e235f99d7d67e558dafad601f253">fb741fd</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L435" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="435"></td>
          <td id="LC435" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-s1">supports_ad_hoc_ti_run</span>: <span class="pl-s1">bool</span> <span class="pl-c1">=</span> <span class="pl-c1">True</span> </td>
        </tr>
    </tbody></table>
  </div>
</div>
 <div class="Box Box--condensed my-2">
  <div class="Box-header f6">
    <p class="mb-0 text-bold">
      <a href="https://github.com/apache/airflow/blob/26f94c5370587f73ebd935cecf208c6a36bdf9b6/airflow/www/views.py#L1735-L1737">airflow/airflow/www/views.py</a>
    </p>
    <p class="mb-0 color-fg-muted">
        Lines 1735 to 1737
      in
      <a data-pjax="true" class="commit-tease-sha" href="/apache/airflow/commit/26f94c5370587f73ebd935cecf208c6a36bdf9b6">26f94c5</a>
    </p>
  </div>
  <div itemprop="text" class="Box-body p-0 blob-wrapper blob-wrapper-embedded data">
    <table class="highlight tab-size mb-0 js-file-line-container" data-tab-size="8" data-paste-markdown-skip="">

        <tbody><tr class="border-0">
          <td id="L1735" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1735"></td>
          <td id="LC1735" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line"> <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-en">getattr</span>(<span class="pl-s1">executor</span>, <span class="pl-s">"supports_ad_hoc_ti_run"</span>, <span class="pl-c1">False</span>): </td>
        </tr>

        <tr class="border-0">
          <td id="L1736" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1736"></td>
          <td id="LC1736" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-s1">msg</span> <span class="pl-c1">=</span> <span class="pl-s">"Only works with the Celery, CeleryKubernetes or Kubernetes executors"</span> </td>
        </tr>

        <tr class="border-0">
          <td id="L1737" class="blob-num border-0 px-3 py-0 color-bg-default" data-line-number="1737"></td>
          <td id="LC1737" class="blob-code border-0 px-3 py-0 color-bg-default blob-code-inner js-file-line">     <span class="pl-k">return</span> <span class="pl-en">redirect_or_json</span>(<span class="pl-s1">origin</span>, <span class="pl-s1">msg</span>, <span class="pl-s">"error"</span>, <span class="pl-c1">400</span>) </td>
        </tr>
    </tbody></table>
  </div>
</div>
<p></p>
<h3 dir="auto">Description</h3>
<p dir="auto">The databricks provider can be updated to a deferrable task introduced in airflow 2.2.0. This will significnatly reduce cpu usage, memory usage on LocalExecutor and improve reliability since state is stored in the airflow metastore.</p>
<h3 dir="auto">Use case/motivation</h3>
<p dir="auto">Currently the databricks operators works by calling the databricks REST API to submit a spark job and polling it every 20 secs to check when it is done. This creates the following inefficiencies:</p>
<ul dir="auto">
<li>If the airflow executor process crashes, a duplicate job can be created in databricks since airflow doesn't save the databricks job run id.</li>
<li>Each task instance of the databricks operators run its own process, however 95% of the time the process is idle and waiting to repoll the databricks API. If you want to 50 jobs in parallel, you will use a non trivial amount of memory / CPU just to poll a rest API.</li>
</ul>
<h3 dir="auto">Related issues</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Are you willing to submit a PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Yes I am willing to submit a PR!</li>
</ul>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>