Ways Data Projects Fail
Good to see that data cleaning was #1 on that list. Whenever I do work on a side project, it takes way way more time to get and structure the data than it does running the algorithms. Granted, that's because I have to go out and get the data in the first place, and then make sure it's useable and in the correct format.Like the recent project I'm doing trying to classify country music songs based on their topic on the data blog I write on (https://bigishdata.com), the amount of time it's taking to scrape lyrics, remove duplicate / incorrect songs, and then do manual classification for training data is taking far longer than running the ml algorithms in the end one I've gone through that process.I've been looking for jobs recently, and I've seen only one job posting that mentions data cleaning as a necessity, whereas the rest only talk about data science and algorithm knowledge, or overall ETL design on the data engineering side. Seems like data set knowledge should be emphasized more.