There is no data engineering roadmap
> All you need is l̶o̶v̶e̶ SQLI’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well.  I have mixed feelings about the field precisely because it’s so SQL dominated.Data engineering can be unsatisfying if you thrive on writing reliable systems.  There’s a whole lot of big-ball-of-inscrutable-SQL work out there which when it gets changed breaks in unpredictable ways.  And the cultural traditions of testing and validation are underdeveloped because SQL has no core testing story — people count on joins being logically correct, but don’t go to the trouble to prove that true.  Vector operations are just hard to get right in any ecosystem and if you don’t stay humble and apply skepticism to your work, it’s going to be wrong a lot.I try to do stuff in pandas or similar — which has its own problems, but at least allows for testable library code. But in the field of data engineering, that’s swimming against the tide.