Persimmon-8B
Two important takeaways on the base model:<p>* scored 18.9 on HumanEval (coding) where Llama2 7B scored 12.2<p>* was trained from the beginning with a 16k context using a modified RoPe where many models are simply fine-tuned using RoPe to gain longer context windows after the base model has been trained at 4k.<p>Can anyone share ideas on how important the 2nd one is? Do LLMs benefit from large context windows using RoPe during pretraining?