Dgsh – Directed graph shell
I&#x27;ve worked with and looked at a lot of data processing helpers. Tools, that try to help you build data pipelines, for the sake of performance, reproducibility or simply code uniformity.<p>What I found so far: Most tools, that invent a new language or try to cram complex processes into lesser suited syntactical environments are not loved too much.<p>A few people like XSLT, most seem to dislike it, although it has a nice functional core hidden under a syntax that seems to come from a time, where the answer to everything was XML. There are big data orchestration frameworks, that use an XML as configuration language, which can be ok, if you have clear processing steps.<p>Every time a tool invents a DSL for data processing, I grab my list of ugly real world use cases and most of the tools fail soon, if not immediately. That&#x27;s a pity.<p>Programming languages can be effective as they are, and with the exceptions that unclean data brings, you want to have a programming language at your disposal anyway.<p>I&#x27;ll give dgsh a try. The tool reuse approach and the UNIX spirit seems nice. But my initial impression of the &quot;C code metrics&quot; example from the site is mixed: It reminds me of awk, about which one of the authors said, that it&#x27;s a beautiful language, but if your programs getting longer than hundred lines, you might want to switch to something else.<p>Two libraries which have a great grip at the plumbing aspect of data processing systems are airflow and luigi. They are python libraries and with it you have a concise syntax and basically all python libraries plus non-python tools with a command line interface at you fingertips.<p>I am curious, what kind of process orchestration tools people use and can recommend?