Rewrites are Waterfall
I like Agile, but I think the industry needs lot more Waterfall. Imagine building a house with Agile, you would start from the roof (rain is the most pressing customer&#x27;s problem), and the foundation would be built last (after, 5 years into the future, a crack appeared in the house wall).<p>I also think what&#x27;s problematic is this debate goes too much around project management, rather than actual methodology of building software (software engineering). I have an idea how we can reframe the debate to this effect.<p>When software engineers plan and write software, they actually build two things - the product itself and the infrastructure&#x2F;tooling for it. From the project and product management perspective, the focus is on the first and the latter is often neglected as unimportant (and waste of time). This sort of jives with the human intuition, we want to build a house, while foundation and scaffolding are only incidental to it.<p>However, I can&#x27;t help but notice that the software products that are actually most &quot;agile&quot;, and easy to change, are written in a way where there is a huge amount of generic infrastructure and tooling with a tiny veneer of product (&quot;business logic&quot;) on top of them (and in fact this goes all the way down). For a canonical example, Emacs, to the point where people even joke about it.<p>But also, this is completely different to how the commercial industry builds software. The focus is on very specific features, pieces of business logic, instead of generic infrastructure, that makes delivering changing features easy. That is a very short-term view, which actually becomes more expensive as the time goes on.<p>I think the way we should build (commercial) software products is the first one, lots of infrastructure and tooling which only incidentally happens to create a product. See also functional core &#x2F; imperative shell, or Unix philosophy.<p>That&#x27;s why I think Agile failed in the industry, because the project management people who pushed for it, didn&#x27;t understand the above distinction. This distinction is also related to Eric Raymond&#x27;s the Cathedral and the Bazaar distinction, although his is more about social organization around building software than the actual software architecture. (The relation manifests in Conway&#x27;s law.)<p>To conclude, I think we need more Waterfall in the sense that we need to think (plan) more deeply about the infrastructure and tooling that we need to build our products. Jumping directly into building a product that solves customer problem will have detrimental effects on long-term productivity.<p>I would also add I see role of open source only as a relatively small part of all the infrastructure and tooling. As you get closer to the actual product, more and more infrastructure is specific to your business domain. The open source will not save you, because it&#x27;s too generic. Emacs is not just a generic OS or a Lisp interpreter. It has huge amount of infrastructure around text editing specifically.