The evolution of the data engineer role
I&#x27;m a software dev who&#x27;s been bumping up against the data engineering field lately, and I&#x27;ve been dismayed as to how many popular tools shunt you towards unmaintainable, unevolvable system design.<p>- A predilection for SQL, yielding &quot;get the answer right once&quot; big-ball-of-SQL solutions which are infeasible to debug or modify without causing regressions.<p>- Poor support for unit testing.<p>- Poor support for version control.<p>- Frameworks over libraries (because the vendors want to lock you in).<p>&gt; data engineers have been increasingly adopting software engineering best practices<p>We can only hope.  I think it&#x27;s more likely that in the near term data engineers will get better and better at prototyping within low-code frameworks, and that transitioning from the prototype to an evolvable system will get harder.