We need new data books, so we started one: Cloud Data Management
One of the experiences that stuck with me most going from doing software engineering stuff to data engineering stuff is the very difficult and sometimes complete lack of tooling around testing and debugging issues in SQL and pipelines. As software engineers, we have debuggers, unit testing, mocking frameworks, e2e tools, BDD languages like Cucumber, code quality tools, etc. But when you're working on a pipeline and you're wondering what will happen if you run it, sometimes your best bet is actually to just run it and wait 20 min for it to complete. Or run some portion of it ad hoc. Or if you want to know what a SQL query is going to do in the black box of your query engine, you might try to parse the esoteric language of a query explainer. The best tools seem to be available after deployment such as data quality tests, dashboards and alerts. I think there's a lot of opportunity to improve the ecosystem.