Do simpler machine learning models exist and how can we find them?
Yes, they exist, and they are called Linear Regression and Decision Tree. Not everything needs to be a neural network.<p>Anyway, residual connections in NNs as well as distillation being only a 1% hit to performance imply our models are way too big.