Engineers Shouldn’t Write ETL
I&#x27;ve worked in a hedge fund in the past, where my role sounded a lot like what the author describes as a &quot;Data Engineer&quot;. I would have thinkers, ie people with a lot of financial experience, come up with ideas on which datasets we want to import from which vendors, and how we should handle the 80 different types of corporate actions that are contained within this dataset.<p>I sometimes gave my own suggestions on how to improve upon their ideas, but for the most part, I was happy to focus on implementing their ideas, in the most clean, elegant, robust and testable manner possible. I was happy to do the &quot;plumbing&quot; work of improving upon our tech stack and architecture, in order to make the entire system better functioning and easier to maintain.<p>According to the author, I&#x27;m supposed to resent the fact that I&#x27;m a &quot;doer&#x2F;plumber&quot;, and not a &quot;thinker&quot;. In reality, it was the opposite. Do I really want to spend my entire day reading the Bloomberg manual and figuring out which tables&#x2F;columns will give us the data we want, and the nuances of what this dataset does and does not cover? Sorry, I have zero interest in doing that.<p>I enjoy programming. I enjoy system design. I enjoy building stuff. I have zero interest in becoming an expert on how to interpret the Bloomberg symbology file. Besides, if I ever left the financial industry and joined a tech company, that knowledge will become completely useless.<p>Did I or anyone consider myself to be a &quot;menial&quot; plumber? I don&#x27;t think so. I was getting paid hundreds of thousands of dollars, because the &quot;thinkers&quot; recognized the value that I brought to the table. They appreciated that I could quickly and robustly implement the ideas that they had, and keep the system running smoothly without hiccups. They recognized anyone can do a &quot;good enough&quot; job, but it&#x27;s much much harder to find someone who can do a <i>great</i> job. And for my part, I was perfectly happy to be that guy.<p>If you&#x27;re someone who wants to expand your breadth and take on more &quot;thinker&quot; responsibilities, more power to you. But just don&#x27;t forget that there are people like me out there too. There&#x27;s no shame in being an excellent &quot;doer&quot;.