The evolution of the data engineer role
Great article.<p>&gt; data engineers have been increasingly adopting software engineering best practices<p>I think the data engineering field is starting to adopt some software engineering best practices, but it&#x27;s still really early days.  I am the author of popular Spark testing libraries (spark-fast-tests, chispa) and they definitely have a large userbase, but could also grow a lot.<p>&gt; The way organizations structure data teams has changed over the years. Now we see a shift towards decentralized data teams, self-serve data platforms, and ways to store data beyond the data warehouse – such as the data lake, data lakehouse, or the previously mentioned data mesh – to better serve the needs of each data consumer.<p>I think the Lakehouse architecture is the real future of data engineering, see the paper: <a href="https:&#x2F;&#x2F;www.cidrdb.org&#x2F;cidr2021&#x2F;papers&#x2F;cidr2021_paper17.pdf" rel="nofollow">https:&#x2F;&#x2F;www.cidrdb.org&#x2F;cidr2021&#x2F;papers&#x2F;cidr2021_paper17.pdf</a><p>Disclosure: I am on the Delta Lake team, but joined because I believe in the Lakehouse architecture vision.<p>It will take a long time for folks to understand all the differences between data lakes, Lakehouses, data warehouses, etc.  Over time, I think mass adoption of the Lakehouse architecture is inevitable (benefits of open file formats, no lock in, separating compute from storage, cost management, scalability, etc.).