Publishing with Apache Kafka at The New York Times
This is a flawed architecture. It will work at release, but it will be difficult to manoeuvre with, and they will grow to hate it.<p>As your business changes, your data changes. Imagine if on day one, they had one author per article. On day 1000, they change this to be a list of authors.<p>Kafka messages are immutable. Each of those green boxes on the right hand side of the first diagram will need to have special-case logic to unpack the kafka stream, with knowledge of its changes (up until 17 May 2017, treat the data like this, but between then and 19 May 2017 do x, and after that do y).<p>Document pipelines is a rare instance of a context where XML is the best choice. They should have defined normalised file formats for each of their data structures. Something like the gateway on the left of the first diagram would write files in that format.  (At some future time, they will need to modify the normalised formats. Files are good for that. You can change the gateway and your stored files in coordination.)<p>Secondly, they should have a gateway coming out of the file store. For each downstream consumer, they should have a distinct API.<p>These APIs might look the same on the first day of release. But they should be separate APIs so that you are free to refactor them independently.<p>When you have a one-to-one API relationship, you can negotiate significant refactors in a single phone call. When you have more than one codebase consuming, you need to have endless meetings and project managers. I call this, &quot;The Principle of Two.&quot;<p>Some of the other comments here say that they should have used databases. So far, they have not made the case for it. And databases are easily abused in settings like this one. People connect multiple codebases to them, and use SQL as a chainsaw. Again, you can&#x27;t negotiate changes.<p>When you create a system, your data structures are the centre of that system. You need to do everything you can to keep your options open to refactor them at a later time, and to do so in a way that respects APIs that you are offering your partners.<p>Kafka is a good tool. If used well, your deployment design will stop your system regularly (e.g. every day), nuke the channels, recreate them from scratch, and restart your system against these empty channels. You shouldn&#x27;t use it as a long-term data store.