Pre-training on Data with Higher Quality Lemur (Xu et al., 2023d) has verified a better mixture
of natural language data and code and induces the LLMs with stronger abilities on function calling,
automatic programming, and agent capabilities. Specifically, Lemur-70B-chat achieves significant
improvements over GPT-3.5-turbo on both HumanEval and GSM8K without task-specific fine-tuning.
Phi-1 and Phi-1.5 (Gunasekar et al., 2023; Li et al., 2023e) take a different road by using the textbook
as the main corpus for pre-training, which makes the strong abilities observable on much smaller
language models.

3.4 Modelling Long-context Capabilities

Benchmarks Processing long sequences remains one of the key technological bottlenecks of LLMs,
as all models are limited by a finite maximum context window, typically from 2k to 8k tokens
in length. Benchmarking long-context capability of LLMs involves evaluating on several tasks

7
ChatGPT’sOne-yearAnniversary-AreOpen-Source LargeLanguageModelsCatchingup