We’re in the Middle of a Data Engineering Talent Shortage
We should rename this job position to Data Sanity Engineers.<p>I have been thrown these projects at work before, where I&#x27;m the frontend engineer and I need to make some cool D3 visualization, but low behold the data is shit, and I have to help the backend team make the data useable.  It&#x27;s a mind-numbing job, that nobody wants, because it sounds like a one month task to get a good REST API up and working, but it usually takes three months, because you have to go back and forth making sure the data is right, and there is always 10 tricky edge cases that you have to work some magic on. Not only that but you need to have smart people cleaning the data, so that you don&#x27;t make some big mistake down the line or your REST API is super slow, and you have to add another couple weeks or month to rework the data again. So that one month becomes three months, and most likely a year, because somebody will say that looks great but can we also add this, and it goes on and on.  It&#x27;s literally a mind-numbing job that most nobody wants.  I have found that products like Tableau are the best for this, you still have to clean the data, but it helps speed up the process.<p>Data cleaning is a super golden problem to solve.