On moving from statistics to machine learning, the final stage of grief (2019)
The author's pie chart showing data science to be 60% data manipulation is accurate. The biggest gap between good and bad data scientist is their comfort level with data wrangling. When interviewing candidates for data science positions, one of the simplest questions is to have them sort a 1 GB tab-delimited file.<p>1. Poor candidates will try to open the file in Excel.<p>2a. Marginal candidates will use R or Stata.<p>2b. Okay candidates will use a scripting language like Python.<p>3. Good candidates will use Unix sort.<p>To my knowledge, there are no university courses teaching the Unix toolchain and it remains very much a skill learned through practice.