Tiny Language Models Come of Age
I'm a big believer that smaller models will take hold. The throwing paint at the wall strategy of having 100B+ models and random data won't scale. It's possible it will take new model architectures to get there. But having more fine-tuned control over model parameters is a pattern that will emerge. Then smaller, domain specific models can be joined together to form larger models, if full generalization is needed.I have an article on building micromodels that discusses some of this: https://neuml.hashnode.dev/train-a-language-model-from-scrat...