How long can open-source LLMs truly promise on context length?
I think this is more of an issue of smaller models with larger context aren't a magic bullet and I'm not sure anyone really paying attention thought they would be.<p>Falcon 40BN, after extensive use, is the only model that has crossed the "I'd build a product on this" threshold in open source. I expect my tune won't change there until we see larger Open Source models trained on high quality data. These are the models that will benefit from the extended context lengths.<p>But longer context without a model capable of utilizing the context to actually do things, is yeah, unsurprisingly, not useful.<p>There's also not really an incentive to currently make that happen in Open Source. If you can train a big model, with a unique high quality dataset you've assembled, using some novel techniques on some performant serving infrastructure, you basically have a blank check right now.<p>Edit: On my last point, I'm going to say I was being very jaded. I think the community is doing a lot of awesome things and I know of a few efforts personally to train these larger models for open source consumption. I don't mean to be so overly negative here.