<h3 dir="auto">Apache Airflow version</h3>
<p dir="auto">2.3.1</p>
<h3 dir="auto">What happened</h3>
<p dir="auto">When attempting to delete a DAG using Airflow 2.3.1 I get the following error, and the DAG along with its metadata are effectively not deleted:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Something bad has happened.

Airflow is used by many users, and it is very likely that others had similar problems and you can easily find
a solution to your problem.

Consider following these steps:

  * gather the relevant information (detailed logs with errors, reproduction steps, details of your deployment)

  * find similar issues using:
     * [GitHub Discussions](https://github.com/apache/airflow/discussions)
     * [GitHub Issues](https://github.com/apache/airflow/issues)
     * [Stack Overflow](https://stackoverflow.com/questions/tagged/airflow)
     * the usual search engine you use on a daily basis

  * if you run Airflow on a Managed Service, consider opening an issue using the service support channels

  * if you tried and have difficulty with diagnosing and fixing the problem yourself, consider creating a [bug report](https://github.com/apache/airflow/issues/new/choose).
    Make sure however, to include all relevant details and results of your investigation so far.

Python version: 3.7.13
Airflow version: 2.3.1
Node: airflow-webserver-766ff4b954-p7m8m
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File &quot;/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py&quot;, line 1706, in _execute_context
    cursor, statement, parameters, context
  File &quot;/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py&quot;, line 716, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.ForeignKeyViolation: update or delete on table &quot;dag&quot; violates foreign key constraint &quot;dag_tag_dag_id_fkey&quot; on table &quot;dag_tag&quot;
DETAIL:  Key (dag_id)=(etl_daily_stg) is still referenced from table &quot;dag_tag&quot;.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File &quot;/home/airflow/.local/lib/python3.7/site-packages/flask/app.py&quot;, line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File &quot;/home/airflow/.local/lib/python3.7/site-packages/flask/app.py&quot;, line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File &quot;/home/airflow/.local/lib/python3.7/site-packages/flask/app.py&quot;, line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File &quot;/home/airflow/.local/lib/python3.7/site-packages/flask/_compat.py&quot;, line 39, in reraise
    raise value
  File &quot;/home/airflow/.local/lib/python3.7/site-packages/flask/app.py&quot;, line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File &quot;/home/airflow/.local/lib/python3.7/site-packages/flask/app.py&quot;, line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File &quot;/home/airflow/.local/lib/python3.7/site-packages/airflow/www/auth.py&quot;, line 43, in decorated
    return func(*args, **kwargs)
  File &quot;/home/airflow/.local/lib/python3.7/site-packages/airflow/www/decorators.py&quot;, line 80, in wrapper
    return f(*args, **kwargs)
  File &quot;/home/airflow/.local/lib/python3.7/site-packages/airflow/www/views.py&quot;, line 1844, in delete
    delete_dag.delete_dag(dag_id)
  File &quot;/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py&quot;, line 71, in wrapper
    return func(*args, session=session, **kwargs)
  File &quot;/home/airflow/.local/lib/python3.7/site-packages/airflow/api/common/delete_dag.py&quot;, line 80, in delete_dag
    .delete(synchronize_session='fetch')
  File &quot;/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/query.py&quot;, line 3111, in delete
    execution_options={&quot;synchronize_session&quot;: synchronize_session},
  File &quot;/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/session.py&quot;, line 1670, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File &quot;/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py&quot;, line 1520, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File &quot;/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py&quot;, line 314, in _execute_on_connection
    self, multiparams, params, execution_options
  File &quot;/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py&quot;, line 1399, in _execute_clauseelement
    cache_hit=cache_hit,
  File &quot;/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py&quot;, line 1749, in _execute_context
    e, statement, parameters, cursor, context
  File &quot;/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py&quot;, line 1930, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File &quot;/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/compat.py&quot;, line 211, in raise_
    raise exception
  File &quot;/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py&quot;, line 1706, in _execute_context
    cursor, statement, parameters, context
  File &quot;/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py&quot;, line 716, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.ForeignKeyViolation) update or delete on table &quot;dag&quot; violates foreign key constraint &quot;dag_tag_dag_id_fkey&quot; on table &quot;dag_tag&quot;
DETAIL:  Key (dag_id)=(etl_daily_stg) is still referenced from table &quot;dag_tag&quot;.

[SQL: DELETE FROM dag WHERE dag.dag_id IN (%(dag_id_1_1)s) RETURNING dag.dag_id]
[parameters: {'dag_id_1_1': 'etl_daily_stg'}]
(Background on this error at: http://sqlalche.me/e/14/gkpj)"><pre class="notranslate"><code class="notranslate">Something bad has happened.

Airflow is used by many users, and it is very likely that others had similar problems and you can easily find
a solution to your problem.

Consider following these steps:

  * gather the relevant information (detailed logs with errors, reproduction steps, details of your deployment)

  * find similar issues using:
     * [GitHub Discussions](https://github.com/apache/airflow/discussions)
     * [GitHub Issues](https://github.com/apache/airflow/issues)
     * [Stack Overflow](https://stackoverflow.com/questions/tagged/airflow)
     * the usual search engine you use on a daily basis

  * if you run Airflow on a Managed Service, consider opening an issue using the service support channels

  * if you tried and have difficulty with diagnosing and fixing the problem yourself, consider creating a [bug report](https://github.com/apache/airflow/issues/new/choose).
    Make sure however, to include all relevant details and results of your investigation so far.

Python version: 3.7.13
Airflow version: 2.3.1
Node: airflow-webserver-766ff4b954-p7m8m
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1706, in _execute_context
    cursor, statement, parameters, context
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 716, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.ForeignKeyViolation: update or delete on table "dag" violates foreign key constraint "dag_tag_dag_id_fkey" on table "dag_tag"
DETAIL:  Key (dag_id)=(etl_daily_stg) is still referenced from table "dag_tag".


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www/auth.py", line 43, in decorated
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www/decorators.py", line 80, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www/views.py", line 1844, in delete
    delete_dag.delete_dag(dag_id)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 71, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/api/common/delete_dag.py", line 80, in delete_dag
    .delete(synchronize_session='fetch')
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3111, in delete
    execution_options={"synchronize_session": synchronize_session},
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1670, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1520, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 314, in _execute_on_connection
    self, multiparams, params, execution_options
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1399, in _execute_clauseelement
    cache_hit=cache_hit,
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1749, in _execute_context
    e, statement, parameters, cursor, context
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1930, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1706, in _execute_context
    cursor, statement, parameters, context
  File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 716, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.ForeignKeyViolation) update or delete on table "dag" violates foreign key constraint "dag_tag_dag_id_fkey" on table "dag_tag"
DETAIL:  Key (dag_id)=(etl_daily_stg) is still referenced from table "dag_tag".

[SQL: DELETE FROM dag WHERE dag.dag_id IN (%(dag_id_1_1)s) RETURNING dag.dag_id]
[parameters: {'dag_id_1_1': 'etl_daily_stg'}]
(Background on this error at: http://sqlalche.me/e/14/gkpj)
</code></pre></div>
<h3 dir="auto">What you think should happen instead</h3>
<p dir="auto">DAG should be deleted along with the associated metadata</p>
<h3 dir="auto">How to reproduce</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Operating System</h3>
<p dir="auto">UNIX</p>
<h3 dir="auto">Versions of Apache Airflow Providers</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Deployment</h3>
<p dir="auto">Official Apache Airflow Helm Chart</p>
<h3 dir="auto">Deployment details</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Anything else</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Are you willing to submit PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Yes I am willing to submit a PR!</li>
</ul>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>
<p dir="auto">Hi, I have a very strange and specific behaviour of Airflow on AWS EKS cluster after deploying Calico to enforce network policies.  I have also created AWS support case, but I also need support from Airflow team. I will be very appreciated for any help.<br>
<strong>What happened</strong>:<br>
I have Airflow set-up running as 2 k8s pods (Airflow webserver and scheduler). Both Airflow pods use git-sync sidecar container to get DAGs from git and store it at k8s <code class="notranslate">emptyDir</code> volume. All works well on fresh EKS cluster without errors. But at the moment of deploing Calico <a href="https://docs.aws.amazon.com/eks/latest/userguide/calico.html" rel="nofollow">https://docs.aws.amazon.com/eks/latest/userguide/calico.html</a> to EKS cluster all DAGs with local imports become broken. Airflow has default k8s Network policy which allow all ingress/egress traffic without restrictions, and Airflow UI is accessible. But in the Airflow there is a message <code class="notranslate">DAG "helloWorld" seems to be missing.</code> and Airflow webserver became to generate an error in the logs:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2020-07-08 14:43:38,784] {__init__.py:51} INFO - Using executor SequentialExecutor                                                                          │
│ [2020-07-08 14:43:38,784] {dagbag.py:396} INFO - Filling up the DagBag from /usr/local/airflow/dags/repo                                                     │
│ [2020-07-08 14:43:38,785] {dagbag.py:225} DEBUG - Importing /usr/local/airflow/dags/repo/airflow_dags/dag_test.py                                            │
│ [2020-07-08 14:43:39,016] {dagbag.py:239} ERROR - Failed to import: /usr/local/airflow/dags/repo/airflow_dags/dag_test.py                                    │
│ Traceback (most recent call last):                                                                                                                           │
│   File &quot;/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py&quot;, line 236, in process_file                                                          │
│     m = imp.load_source(mod_name, filepath)                                                                                                                  │
│   File &quot;/usr/local/lib/python3.7/imp.py&quot;, line 171, in load_source                                                                                           │
│     module = _load(spec)                                                                                                                                     │
│   File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 696, in _load                                                                                                   │
│   File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 677, in _load_unlocked                                                                                          │
│   File &quot;&lt;frozen importlib._bootstrap_external&gt;&quot;, line 728, in exec_module                                                                                    │
│   File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 219, in _call_with_frames_removed                                                                               │
│   File &quot;/usr/local/airflow/dags/repo/airflow_dags/dag_test.py&quot;, line 5, in &lt;module&gt;                                                                          │
│     from airflow_dags.common import DEFAULT_ARGS                                                                                                             │
│ ModuleNotFoundError: No module named 'airflow_dags'"><pre class="notranslate"><code class="notranslate">[2020-07-08 14:43:38,784] {__init__.py:51} INFO - Using executor SequentialExecutor                                                                          │
│ [2020-07-08 14:43:38,784] {dagbag.py:396} INFO - Filling up the DagBag from /usr/local/airflow/dags/repo                                                     │
│ [2020-07-08 14:43:38,785] {dagbag.py:225} DEBUG - Importing /usr/local/airflow/dags/repo/airflow_dags/dag_test.py                                            │
│ [2020-07-08 14:43:39,016] {dagbag.py:239} ERROR - Failed to import: /usr/local/airflow/dags/repo/airflow_dags/dag_test.py                                    │
│ Traceback (most recent call last):                                                                                                                           │
│   File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 236, in process_file                                                          │
│     m = imp.load_source(mod_name, filepath)                                                                                                                  │
│   File "/usr/local/lib/python3.7/imp.py", line 171, in load_source                                                                                           │
│     module = _load(spec)                                                                                                                                     │
│   File "&lt;frozen importlib._bootstrap&gt;", line 696, in _load                                                                                                   │
│   File "&lt;frozen importlib._bootstrap&gt;", line 677, in _load_unlocked                                                                                          │
│   File "&lt;frozen importlib._bootstrap_external&gt;", line 728, in exec_module                                                                                    │
│   File "&lt;frozen importlib._bootstrap&gt;", line 219, in _call_with_frames_removed                                                                               │
│   File "/usr/local/airflow/dags/repo/airflow_dags/dag_test.py", line 5, in &lt;module&gt;                                                                          │
│     from airflow_dags.common import DEFAULT_ARGS                                                                                                             │
│ ModuleNotFoundError: No module named 'airflow_dags'
</code></pre></div>
<p dir="auto">The DAG itself consists of 2 files: <code class="notranslate">dag_test.py</code> and <code class="notranslate">common.py</code>. Content of the files are:<br>
<code class="notranslate">common.py</code></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from datetime import datetime, timedelta

DEFAULT_ARGS = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2020, 3, 26),
    'retry_delay': timedelta(minutes=1),
}"><pre class="notranslate"><code class="notranslate">from datetime import datetime, timedelta

DEFAULT_ARGS = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2020, 3, 26),
    'retry_delay': timedelta(minutes=1),
}
</code></pre></div>
<p dir="auto"><code class="notranslate">dag_test.py</code></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from airflow import DAG
from airflow.operators.bash_operator import BashOperator

from airflow_dags.common import DEFAULT_ARGS

dag = DAG('helloWorld', schedule_interval='*/5 * * * *', default_args=DEFAULT_ARGS)

t1 = BashOperator(
    task_id='task_1',
    bash_command='echo &quot;Hello World from Task 1&quot;; sleep 30',
    dag=dag
)"><pre class="notranslate"><code class="notranslate">from airflow import DAG
from airflow.operators.bash_operator import BashOperator

from airflow_dags.common import DEFAULT_ARGS

dag = DAG('helloWorld', schedule_interval='*/5 * * * *', default_args=DEFAULT_ARGS)

t1 = BashOperator(
    task_id='task_1',
    bash_command='echo "Hello World from Task 1"; sleep 30',
    dag=dag
)
</code></pre></div>
<p dir="auto"><em>What I have already tried at the webserver and scheduler pods</em>:</p>
<ul dir="auto">
<li>ssh to Airflow pod and enter Python shell. All imports work fine, for example:</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="airflow@airflow-webserver-78bc695cc7-l7z9s:~$ pwd
/usr/local/airflow
airflow@airflow-webserver-78bc695cc7-l7z9s:~$ python
Python 3.7.4 (default, Oct 17 2019, 06:10:02)
[GCC 8.3.0] on linux
Type &quot;help&quot;, &quot;copyright&quot;, &quot;credits&quot; or &quot;license&quot; for more information.
&gt;&gt;&gt; from airflow_dags.common import DEFAULT_ARGS
&gt;&gt;&gt; print(DEFAULT_ARGS)
{'owner': 'airflow', 'depends_on_past': False, 'start_date': datetime.datetime(2020, 3, 26, 0, 0), 'retry_delay': datetime.timedelta(seconds=60)}
&gt;&gt;&gt;"><pre class="notranslate"><code class="notranslate">airflow@airflow-webserver-78bc695cc7-l7z9s:~$ pwd
/usr/local/airflow
airflow@airflow-webserver-78bc695cc7-l7z9s:~$ python
Python 3.7.4 (default, Oct 17 2019, 06:10:02)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; from airflow_dags.common import DEFAULT_ARGS
&gt;&gt;&gt; print(DEFAULT_ARGS)
{'owner': 'airflow', 'depends_on_past': False, 'start_date': datetime.datetime(2020, 3, 26, 0, 0), 'retry_delay': datetime.timedelta(seconds=60)}
&gt;&gt;&gt;
</code></pre></div>
<ul dir="auto">
<li>from pod bash shell, I can execute airflow command and <code class="notranslate">list_tasks</code>, and DAG is not broken:</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="airflow@airflow-webserver-78bc695cc7-l7z9s:~$ airflow list_tasks helloWorld
[2020-07-08 15:37:24,309] {settings.py:212} DEBUG - Setting up DB connection pool (PID 275)
[2020-07-08 15:37:24,310] {settings.py:253} DEBUG - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=275
[2020-07-08 15:37:24,366] {cli_action_loggers.py:42} DEBUG - Adding &lt;function default_action_log at 0x7fb9b5a4f710&gt; to pre execution callback
[2020-07-08 15:37:24,817] {cli_action_loggers.py:68} DEBUG - Calling callbacks: [&lt;function default_action_log at 0x7fb9b5a4f710&gt;]
[2020-07-08 15:37:24,847] {__init__.py:51} INFO - Using executor SequentialExecutor
[2020-07-08 15:37:24,848] {dagbag.py:396} INFO - Filling up the DagBag from /usr/local/airflow/dags/repo
[2020-07-08 15:37:24,849] {dagbag.py:225} DEBUG - Importing /usr/local/airflow/dags/repo/airflow_dags/dag_test.py
[2020-07-08 15:37:25,081] {dagbag.py:363} DEBUG - Loaded DAG &lt;DAG: helloWorld&gt;
[2020-07-08 15:37:25,082] {dagbag.py:225} DEBUG - Importing /usr/local/airflow/dags/repo/airflow_dags/dagbg_add.py
task_1
[2020-07-08 15:37:25,083] {cli_action_loggers.py:86} DEBUG - Calling callbacks: []
[2020-07-08 15:37:25,083] {settings.py:278} DEBUG - Disposing DB connection pool (PID 275)

airflow@airflow-webserver-78bc695cc7-l7z9s:~$ airflow trigger_dag helloWorld
[2020-07-08 15:50:25,446] {settings.py:212} DEBUG - Setting up DB connection pool (PID 717)
[2020-07-08 15:50:25,446] {settings.py:253} DEBUG - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=717
[2020-07-08 15:50:25,502] {cli_action_loggers.py:42} DEBUG - Adding &lt;function default_action_log at 0x7fe05c254710&gt; to pre execution callback
[2020-07-08 15:50:25,986] {cli_action_loggers.py:68} DEBUG - Calling callbacks: [&lt;function default_action_log at 0x7fe05c254710&gt;]
[2020-07-08 15:50:26,024] {__init__.py:51} INFO - Using executor SequentialExecutor
[2020-07-08 15:50:26,024] {dagbag.py:396} INFO - Filling up the DagBag from /usr/local/airflow/dags/repo/airflow_dags/dag_test.py
[2020-07-08 15:50:26,024] {dagbag.py:225} DEBUG - Importing /usr/local/airflow/dags/repo/airflow_dags/dag_test.py
[2020-07-08 15:50:26,253] {dagbag.py:363} DEBUG - Loaded DAG &lt;DAG: helloWorld&gt;
Created &lt;DagRun helloWorld @ 2020-07-08 15:50:26+00:00: manual__2020-07-08T15:50:26+00:00, externally triggered: True&gt;
[2020-07-08 15:50:26,289] {cli_action_loggers.py:86} DEBUG - Calling callbacks: []
[2020-07-08 15:50:26,289] {settings.py:278} DEBUG - Disposing DB connection pool (PID 717)"><pre class="notranslate"><code class="notranslate">airflow@airflow-webserver-78bc695cc7-l7z9s:~$ airflow list_tasks helloWorld
[2020-07-08 15:37:24,309] {settings.py:212} DEBUG - Setting up DB connection pool (PID 275)
[2020-07-08 15:37:24,310] {settings.py:253} DEBUG - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=275
[2020-07-08 15:37:24,366] {cli_action_loggers.py:42} DEBUG - Adding &lt;function default_action_log at 0x7fb9b5a4f710&gt; to pre execution callback
[2020-07-08 15:37:24,817] {cli_action_loggers.py:68} DEBUG - Calling callbacks: [&lt;function default_action_log at 0x7fb9b5a4f710&gt;]
[2020-07-08 15:37:24,847] {__init__.py:51} INFO - Using executor SequentialExecutor
[2020-07-08 15:37:24,848] {dagbag.py:396} INFO - Filling up the DagBag from /usr/local/airflow/dags/repo
[2020-07-08 15:37:24,849] {dagbag.py:225} DEBUG - Importing /usr/local/airflow/dags/repo/airflow_dags/dag_test.py
[2020-07-08 15:37:25,081] {dagbag.py:363} DEBUG - Loaded DAG &lt;DAG: helloWorld&gt;
[2020-07-08 15:37:25,082] {dagbag.py:225} DEBUG - Importing /usr/local/airflow/dags/repo/airflow_dags/dagbg_add.py
task_1
[2020-07-08 15:37:25,083] {cli_action_loggers.py:86} DEBUG - Calling callbacks: []
[2020-07-08 15:37:25,083] {settings.py:278} DEBUG - Disposing DB connection pool (PID 275)

airflow@airflow-webserver-78bc695cc7-l7z9s:~$ airflow trigger_dag helloWorld
[2020-07-08 15:50:25,446] {settings.py:212} DEBUG - Setting up DB connection pool (PID 717)
[2020-07-08 15:50:25,446] {settings.py:253} DEBUG - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=717
[2020-07-08 15:50:25,502] {cli_action_loggers.py:42} DEBUG - Adding &lt;function default_action_log at 0x7fe05c254710&gt; to pre execution callback
[2020-07-08 15:50:25,986] {cli_action_loggers.py:68} DEBUG - Calling callbacks: [&lt;function default_action_log at 0x7fe05c254710&gt;]
[2020-07-08 15:50:26,024] {__init__.py:51} INFO - Using executor SequentialExecutor
[2020-07-08 15:50:26,024] {dagbag.py:396} INFO - Filling up the DagBag from /usr/local/airflow/dags/repo/airflow_dags/dag_test.py
[2020-07-08 15:50:26,024] {dagbag.py:225} DEBUG - Importing /usr/local/airflow/dags/repo/airflow_dags/dag_test.py
[2020-07-08 15:50:26,253] {dagbag.py:363} DEBUG - Loaded DAG &lt;DAG: helloWorld&gt;
Created &lt;DagRun helloWorld @ 2020-07-08 15:50:26+00:00: manual__2020-07-08T15:50:26+00:00, externally triggered: True&gt;
[2020-07-08 15:50:26,289] {cli_action_loggers.py:86} DEBUG - Calling callbacks: []
[2020-07-08 15:50:26,289] {settings.py:278} DEBUG - Disposing DB connection pool (PID 717)
</code></pre></div>
<p dir="auto"><em>To summarise</em>: Airflow DAGs which has local imports become broken in UI and in webserver logs, but is executable from a manual trigger when using EKS cluster with Calico network policies.</p>
<p dir="auto">Please help me to understand why Airflow DAGs imports become broken in UI.</p>
<p dir="auto"><strong>Apache Airflow version</strong>: 1.10.10<br>
<strong>Kubernetes version (if you are using kubernetes)</strong> (use <code class="notranslate">kubectl version</code>):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;15+&quot;, GitVersion:&quot;v1.15.11-eks-af3caf&quot;, GitCommit:&quot;af3caf6136cd355f467083651cc1010a499f59b1&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2020-03-27T21:51:36Z&quot;, GoVersion:&quot;go1.12.17&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}"><pre class="notranslate"><code class="notranslate">Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.11-eks-af3caf", GitCommit:"af3caf6136cd355f467083651cc1010a499f59b1", GitTreeState:"clean", BuildDate:"2020-03-27T21:51:36Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}
</code></pre></div>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: AWS, EKS</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):<br>
EKS workers nodes, EC2 instances:</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="NAME=&quot;Amazon Linux&quot;
VERSION=&quot;2&quot;
ID=&quot;amzn&quot;
ID_LIKE=&quot;centos rhel fedora&quot;
VERSION_ID=&quot;2&quot;
PRETTY_NAME=&quot;Amazon Linux 2&quot;
ANSI_COLOR=&quot;0;33&quot;
CPE_NAME=&quot;cpe:2.3:o:amazon:amazon_linux:2&quot;
HOME_URL=&quot;https://amazonlinux.com/&quot;"><pre class="notranslate"><code class="notranslate">NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
</code></pre></div>
<p dir="auto">Docker image with installed Airflow:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="PRETTY_NAME=&quot;Debian GNU/Linux 10 (buster)&quot;
NAME=&quot;Debian GNU/Linux&quot;
VERSION_ID=&quot;10&quot;
VERSION=&quot;10 (buster)&quot;
VERSION_CODENAME=buster
ID=debian
HOME_URL=&quot;https://www.debian.org/&quot;
SUPPORT_URL=&quot;https://www.debian.org/support&quot;
BUG_REPORT_URL=&quot;https://bugs.debian.org/&quot;"><pre class="notranslate"><code class="notranslate">PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
</code></pre></div>
<ul dir="auto">
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Linux airflow-webserver-78bc695cc7-dmzh2 4.14.181-140.257.amzn2.x86_64 #1 SMP Wed May 27 02:17:36 UTC 2020 x86_64 GNU/Linux"><pre class="notranslate"><code class="notranslate">Linux airflow-webserver-78bc695cc7-dmzh2 4.14.181-140.257.amzn2.x86_64 #1 SMP Wed May 27 02:17:36 UTC 2020 x86_64 GNU/Linux
</code></pre></div>
<ul dir="auto">
<li><strong>Install tools</strong>: we use <code class="notranslate">pipenv</code> to install Airflow to system <code class="notranslate">pipenv install --system --deploy --clear</code></li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>How to reproduce it</strong>:<br>
Create EKS cluster and deploy Calico. Use DAG with local imports.</p>
<p dir="auto"><strong>Anything else we need to know</strong>:<br>
I have all required env, such as <code class="notranslate">AIRFLOW_HOME=/usr/local/airflow, AIRFLOW_DAGS_FOLDER=/usr/local/airflow/dags/repo, PYTHONPATH=/usr/local/airflow/dags/repo</code>  and on EKS cluster without network policies all works fine.</p>