Difficulty in Continuous Improvements over Fundamental Abilities Reviewing the break-
throughs in fundamental abilities outlined in this paper reveals somewhat challenging scenarios: (1)
Considerable efforts have been invested in exploring improved data mixtures during pre-training to
enhance balance and robustness in constructing more potent foundation models. However, the associ-
ated exploration costs often render this approach impractical. (2) Models surpassing GPT-3.5-turbo or
GPT-4 are predominantly based on knowledge distillation from closed-source models and additional
expert annotation. While efficient, heavy reliance on knowledge distillation may mask potential
issues concerning the effectiveness of proposed approaches when being scaled to the teacher model.
Moreover, LLMs are anticipated to act as agents and provide reasonable interpretations to support
decisions, while annotating the agent-style data to make LLMs applicable to real-world scenarios
ChatGPT’sOne-yearAnniversary-AreOpen-Source LargeLanguageModelsCatchingup