Larger context version?
#3
by
smcleod
- opened
Howdy, just wondering if you're planning on releasing a larger context version of this model? I don't think 32k is usable for much other than some basic completion tasks these days.
Hello. After some experiments, we found that extending the context length to 64k achieves comparable or even better performance, while 128k may lead to some degradation. Therefore, we have set the context length to 64k. Further extension through training is also planned. Thanks