Engineers Shouldn’t Write ETL
<i>... a highly specialized team of dedicated engineers...If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”.</i><p>OMG, the author just described the last place I was at. Processed a few Tb of data and suddenly there&#x27;s this R. Goldbergesque system of MongoDb getting transformed into PostGres...oh, wait, I need Cassandra on my resume, so Mongo sux0r now...displayed with some of the worst bowl of spaghetti Android code I&#x27;ve witnessed. The technical debt hole was dug so deep you could hide an Abrams tank in there. To this day I could not tell you the confused thinking that led them to believe this was necessary rather than just slapping it all into PostGres and calling it a day.<p>All because they were processing data sets sooooo huge, that they would fit on my laptop.<p>I quit reading about the time the article turned into a pitch for Stitch Fix, but leading up to that point it made a good case for what happens when companies think they have &quot;big data&quot; when they really don&#x27;t. In summary, either a company hires skills they don&#x27;t really need and the hires end up bored, or you hire mediocre people that make the convoluted mess I worked with.