Persimmon-8B
Two important takeaways on the base model:* scored 18.9 on HumanEval (coding) where Llama2 7B scored 12.2* was trained from the beginning with a 16k context using a modified RoPe where many models are simply fine-tuned using RoPe to gain longer context windows after the base model has been trained at 4k.Can anyone share ideas on how important the 2nd one is? Do LLMs benefit from large context windows using RoPe during pretraining?