Why becoming a data scientist is not easier than you think
I was a math major in college, with a focus on pure math. I did a year or grad school (math PhD program) and left to work on Wall Street (and worked for a couple startups, and Google, in that mix). In all, I spent 6 years as a mix of quant, trader, software engineer, startup entrepreneur, data scientist.<p>The software engineering career is in somewhat of a mess right now. It comes down to the "bozo bit" problem. Being a software engineer (even with 10+ years of experience, because there are a lot of engineers who only do low-end work and don't learn much) is not enough to clear the bozo bit, and you won't be able to prove that you're good unless you have a major success, and it's hard to have that kind of success without people already trusting you with the autonomy to do something genuinely excellent.<p>It's not enough to write code, because LoC is a cost and source code is rarely actually read at the large scale. At least for backend developers, the only work-related (as opposed to political) way to establish that you're worth anything as a engineer is to have an architectural success, but it's very hard to have architectural successes unless you've established yourself as an "architect" to begin with. So there's a permission paradox: you can't do it until you've proven you can, and you can't prove you can do it until you've done it. Hence, the vicious politics that characterize software "architecture" in most companies.<p>Functional programming is one way to put yourself head-and-shoulders above the FactoryFactory hoipolloi. The problem is that most business people don't understand it. They just think Haskell's a weird language "that no one uses". Elite programmers get that you're elite if you know these languages, but most companies are run by people of mediocre engineering ability (and that's often just fine, from a business standpoint).<p>It is true that functional programming is superior to FactoryFactory business bullshit, but not well-enough known. Good luck making that case to someone who's been managing Java projects for 10 years. What is better known is that mathematics is hard. It's a barrier to entry. I doubt more than 5% of professional programmers could derive linear regression.<p>So I see the data science path (and yes, it takes a long time to learn all the components, including statistics, machine learning, and distributed systems) as a mechanism through which a genuinely competent software engineer can say, "I'm good at math, and I can code; therefore, I deserve the most interesting work your company can afford to fund." It's a way to keep getting the best work and avoid falling into that FactoryFactory hoipolloi who stop advancing at 25 and are unemployed by 40.