Launch HN: DAGWorks – ML platform for data science teams
Ah, it seems like you&#x27;re imagining Hamilton to be used to structure sort of &quot;small&quot; pieces, which might then be orchestrated into a Big Picture thing by another tool, something in the Airflow&#x2F;Dagster&#x2F;Argo&#x2F;Flyte class. Or perhaps a paid service offered by DAGWorks in the future...<p>One, that&#x27;s reasonable, and as you say there are code organization and testing benefits. I would emphasize that that&#x27;s the recommended pattern. I would also work to establish, and document, the details of how folks should go about doing that, and provide solid examples. (BTW your &quot;air quality analysis&quot; example is quite good, being far from trivial yet still example-sized in complexity.)<p>Two, ehhhhhhh I&#x27;m a little skeptical of most teams&#x27; ability to factor their projects that well. Folks will want to re-use outputs that are seen as useful, especially if they are expensive to compute. This causes DAG scope to grow and grow and grow. DBT in particular is vulnerable to this, and I have been told of 1000-model projects, which is just <i>yuck</i>. This isn&#x27;t a problem you have to solve right now, but it&#x27;s worth thinking about. As a motivating example, what if someone wanted to take the p-value output by the air quality example, and use that as an input into [some other thing]? What would be the &quot;right&quot; way to express that?