Do simpler machine learning models exist and how can we find them?
I just submitted an article about a paper by Deepmind whose main conclusion is that "data, not size, is the currently active constraint on language modeling performance" [0]. This means that even if we have bigger models, with billions and trillions of parameters, they are unlikely to be better than our current ones, because our amount of data is the bottleneck.TFA though also reminds me of the phenomenon in mathematical proofs where some long winded proof eventually comes up but then over time it becomes simplified as more mathematicians try to optimize it, in much the same way as programmers with technical debt ("make it work, make it right, make it fast"), such as with the four color theorem that was until now computer assisted but it seems there is a non computer assisted proof out [1].I wonder if the problem in TFA could itself be solved by machine learning, where models would create, train, and test other models, changing them along the way, similar to genetic programming but with "artificial selection" and not "natural selection" so to speak.[0] https://news.ycombinator.com/item?id=34098087[1] https://news.ycombinator.com/item?id=34082022