Tiny Language Models Come of Age
I'm a big believer that smaller models will take hold. The throwing paint at the wall strategy of having 100B+ models and random data won't scale. It's possible it will take new model architectures to get there. But having more fine-tuned control over model parameters is a pattern that will emerge. Then smaller, domain specific models can be joined together to form larger models, if full generalization is needed.<p>I have an article on building micromodels that discusses some of this: <a href="https://neuml.hashnode.dev/train-a-language-model-from-scratch" rel="nofollow noreferrer">https://neuml.hashnode.dev/train-a-language-model-from-scrat...</a>