Launch HN: DAGWorks – ML platform for data science teams
yo congrats again on the launch! Anders dbt Labs here with a &quot;tough&quot; question for you. Apologies for 1) my response being half-baked,and 2) if i haven&#x27;t done my homework about Hamilton&#x27;s features.<p>coincidentally, my PR to the dbt viewpoint was closed by the docs team as &quot;closed, won&#x27;t do&quot; [1]<p>I really like the convention of data plane (where you describe how the data should be transformed) and the control plane (i.e. the configuration of the DAG, do this before this). In this paradigm, I believe that the control plane should be as simple as possible, and even perhaps limited in what can be done with the goal of pushing the user to take data transformation as tantamount. Maybe this is why I fell in love with dbt in the first place is because it does exactly this.<p>&quot;spicy&quot; take:
allowing users to write imperative code (e.g. using loops) that dynamically generates DAGs are never a good idea. I say this as someone who personally used to pester framework PMs for this exact feature before. While things like task groups (formerly subDAGs) [2] appear initially to be right answer, I always ended up regretting them. They&#x27;re a scheduling&#x2F;orchestration solution to a data transformation problem<p>Can y&#x27;all speak to how Hamilton views the data and control plane, and how it&#x27;s design philosophy encourages users to use the right tool for the job?<p>p.s. thanks for humoring my pedantry and merging this! [3]<p>[1]: <a href="https:&#x2F;&#x2F;github.com&#x2F;dbt-labs&#x2F;docs.getdbt.com&#x2F;pull&#x2F;2390">https:&#x2F;&#x2F;github.com&#x2F;dbt-labs&#x2F;docs.getdbt.com&#x2F;pull&#x2F;2390</a>
[2]: <a href="http:&#x2F;&#x2F;apache-airflow-docs.s3-website.eu-central-1.amazonaws.com&#x2F;docs&#x2F;apache-airflow&#x2F;latest&#x2F;core-concepts&#x2F;dags.html#taskgroups-vs-subdags" rel="nofollow">http:&#x2F;&#x2F;apache-airflow-docs.s3-website.eu-central-1.amazonaws...</a>
[3]: <a href="https:&#x2F;&#x2F;github.com&#x2F;DAGWorks-Inc&#x2F;hamilton&#x2F;pull&#x2F;105">https:&#x2F;&#x2F;github.com&#x2F;DAGWorks-Inc&#x2F;hamilton&#x2F;pull&#x2F;105</a>