Answers: <p>I spent a long time on this problem and made 2 custom solutions as nothing out of the box exists.</p>
  
  <p>Both steps require you to either use both the experimental and the extra plugin API <strong>or</strong> to have access to the remote database.</p>
  
  <p>At that point, basically either:</p>
  
  <ul>
  <li>Make a GUI plugin on a separate page that hits the APIs and shows all tasks (list exec dates, list tasks in exec date, get status of each).</li>
  <li>Make python operator steps in your dag to call out to the state of the remote dag steps with the same name.  It's hard to reflect anything outside of running/done though as you need your step to keep checking in case the remote end restarts the step/tries again.</li>
  </ul>
  
  <p>The APIs are slow, so this isn't fun.  I'd recommend going for the DB if you have the option.</p>
  
  <p>Airflow 2.0 will fix the APIs apparently, and then this may be a lot easier.</p>
  
Answers: CC BY-SA 4.0
Answers: 2020-04-27T02:45:35.007Z
Answers: 2020-04-27T02:45:35.007Z
Answers: John Humphreys
Body: <p>We have numerous airflow instances running in different organizations/teams around the company.  While we have built a highly scaleable centralized airflow service, we do have cases where we need to depend on another airflow instance to run a dag.</p>
  
  <p>Our DevOps team often gets tasked with running larger orchestrations across teams, and we prefer to keep a highly detailed view of all tasks, even if they are in a remote airflow.</p>
  
  <p>Is there any good way to monitor, with full fidelity, all of the tasks in a remote airflow?  Something similar to the sub-dag operator for a remote airflow would be sufficient.</p>
  
Tags: <orchestration><apache-airflow>
Title: Apache Airflow federation - observe remote dag in full detail?
UserDisplayName: John Humphreys
ThreadText: # Apache Airflow federation - observe remote dag in full detail?
  
  (Asked by: username_0 on 2020-03-26)
  
  We have numerous airflow instances running in different organizations/teams around the company. While we have built a highly scaleable centralized airflow service, we do have cases where we need to depend on another airflow instance to run a dag.
  
  Our DevOps team often gets tasked with running larger orchestrations across teams, and we prefer to keep a highly detailed view of all tasks, even if they are in a remote airflow.
  
  Is there any good way to monitor, with full fidelity, all of the tasks in a remote airflow? Something similar to the sub-dag operator for a remote airflow would be sufficient.
  
  **Tags:** orchestration, apache-airflow
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-04-27. Score: 1 | Accepted answer
  
  I spent a long time on this problem and made 2 custom solutions as nothing out of the box exists.
  
  Both steps require you to either use both the experimental and the extra plugin API **or** to have access to the remote database.
  
  At that point, basically either:
  
  * Make a GUI plugin on a separate page that hits the APIs and shows all tasks (list exec dates, list tasks in exec date, get status of each).
  * Make python operator steps in your dag to call out to the state of the remote dag steps with the same name. It's hard to reflect anything outside of running/done though as you need your step to keep checking in case the remote end restarts the step/tries again.
  
  The APIs are slow, so this isn't fun. I'd recommend going for the DB if you have the option.
  
  Airflow 2.0 will fix the APIs apparently, and then this may be a lot easier.
  
  
  ---
  
  