Approaching Almost Any Machine Learning Problem
Interesting post, but there are several errors, or at least suggestions that don't make sense to me:<p>> Doing so will result in very good evaluation scores and make the user happy but instead he/she will be building a useless model with very high overfitting.<p>Testing on your training data doesn't in-and-of-itself lead to overfitting, but it will hide overfitting if it does exist (and is a terrible practice for that reason).<p>> At this stage only models you should go for should be ensemble tree based models.<p>Not sure why this should be the case. Many ensemble models are very memory hungry & slow, Random Forests being a good example. They are flexible and have minimal assumptions, sure, but that doesn't mean you shouldn't try any other modeling technique, especially if you have domain knowledge.<p>> We cannot apply linear models to the above features since they are not normalized.<p>Not true at all. You won't be able to meaningfully compare coefficient magnitudes, but you can certainly apply linear models.<p>> These normalization methods work only on dense features and don’t give very good results if applied on sparse features.<p>Since normalization is done feature-by-feature, there is no reason why this should be true.<p>> generally PCA is used decompose the data<p>If your latent features add linearly, ok, but do they?  Is it meaningful to have negative values for your features? If not, consider using something like sparse non-negative matrix factorization.<p>Lastly, using this approach means you have a HUGE model & parameter space to search through. Because of this you will need a ton of data to get meaninigful results.<p>He seems to be treating machine learning methods as a bag of tricks.  That's ok so far as it goes, but in my experience it's much more valuable to try and understand your data, and the process that generates it, and then build a model that tries to reflect that data generation process.