<p dir="auto">This github issue is meant to track the big roadmap items ahead of use and for the community to provide ideas, feedback, steer and comment</p>
<h2 dir="auto">Lineage Annotations</h2>
<p dir="auto">Airflow knows a lot about your job dependencies, but how much does it know about your data objects (databases, tables, reports, ...)? Lineage annotations would expose a way for users to define lineage between data objects explicitly and tie that to tasks and/or DAG. The framework may include hooks for programatic inference (HiveOperator could introspect the HQL and guess table names and lineage), and a way to override or complement these. The framework will most likely be fairly agnostic about your data objects, letting you namespace them however you want, and simply consider them as an array of parent/child relationship between strings. It may be nice to use dot notation and reserve the first part of the expression to <code class="notranslate">object_type</code>, allowing for color coding in a graph view and tying actions (links) and the like. This will course ship with nice graph visualization, browsing and navigation features.</p>
<h2 dir="auto">Picklin'</h2>
<p dir="auto">stateless/codeless web servers and workers</p>
<h2 dir="auto">Backfilll UI</h2>
<p dir="auto">Trigger a backfill from the UI</p>
<h2 dir="auto">REST API</h2>
<p dir="auto">Essentially offering features similar to what is available in the CLI through a REST API, there may be some automagic solutions here that can figure out how to build REST specs from argparse, it would also insure consistency between the two</p>
<h1 dir="auto">[done!]</h1>
<h2 dir="auto">Continuous integration with Travis-CI</h2>
<p dir="auto">Systematically run all unit tests against CDH and HDP on Python 2.7 and 3.5</p>
<h2 dir="auto">Externally Triggered DAGs</h2>
<p dir="auto">Airflow currently assumes that you run your workflows on a fixed schedule interval. This is perfect for hourly, daily and weekly jobs. When thinking about "analytics as a service" and "analysis automation", many use cases are more of the "on demand" nature. For instance if you have a workflow that processes someone's genome when ordered to,  or a workflow that builds a dataset on demand for data scientists based on parameters they provide, .... This requires a new class of DAGs that are triggered externally, not on a fixed schedule interval.</p>
<p dir="auto"><strong>Apache Airflow version</strong>: None given</p>
<p dir="auto">Ticket was created 26/Jan/17 23:20</p>
<p dir="auto"><strong>Kubernetes version (if you are using kubernetes)</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>:</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):</li>
<li><strong>Install tools</strong>:</li>
<li><strong>Others</strong>:<br>
<strong>What happened</strong>:</li>
</ul>
<p dir="auto">following piece of code is the root cause of it.</p>
<p dir="auto">line = ''<br>
for line in iter(sp.stdout.readline, b''):<br>
line = line.decode(self.output_encoding).strip()<br>
logging.info(line)</p>
<p dir="auto">I plan to fix it using string buffer instead of just 1 line string variable here.</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto"><strong>How to reproduce it</strong>:</p>
<p dir="auto"><strong>Anything else we need to know</strong>:</p>
<p dir="auto">Moved here from <a href="https://issues.apache.org/jira/browse/AIRFLOW-833" rel="nofollow">https://issues.apache.org/jira/browse/AIRFLOW-833</a></p>