How long can open-source LLMs truly promise on context length?
I think this is more of an issue of smaller models with larger context aren't a magic bullet and I'm not sure anyone really paying attention thought they would be.Falcon 40BN, after extensive use, is the only model that has crossed the "I'd build a product on this" threshold in open source. I expect my tune won't change there until we see larger Open Source models trained on high quality data. These are the models that will benefit from the extended context lengths.But longer context without a model capable of utilizing the context to actually do things, is yeah, unsurprisingly, not useful.There's also not really an incentive to currently make that happen in Open Source. If you can train a big model, with a unique high quality dataset you've assembled, using some novel techniques on some performant serving infrastructure, you basically have a blank check right now.Edit: On my last point, I'm going to say I was being very jaded. I think the community is doing a lot of awesome things and I know of a few efforts personally to train these larger models for open source consumption. I don't mean to be so overly negative here.