Enhancing Chat Language Models by Scaling High-quality Instructional
  Conversations
Fine-tuning on instruction data has been widely validated as an effective
practice for implementing chat language models like ChatGPT. Scaling the
diversity and quality of such data, although straightforward, stands a great
chance of leading to improved performance. This paper aims to improve the upper
bound of open-source models further. We first provide a systematically
designed, diverse, informative, large-scale dataset of instructional
conversations, UltraChat, which does not involve human queries. Our objective
is to capture the breadth of interactions that a human might have with an AI
assistant and employs a comprehensive framework to generate multi-turn
conversation iteratively. UltraChat contains 1.5 million high-quality
multi-turn dialogues and covers a wide range of topics and instructions. Our
statistical analysis of UltraChat reveals its superiority in various key
metrics, including scale, average length, diversity, coherence, etc.,
solidifying its position as a leading open-source dataset. Building upon
UltraChat, we fine-tune a LLaMA model to create a powerful conversational
model, UltraLLaMA. Our evaluations indicate that UltraLLaMA consistently
outperforms other open-source models, including Vicuna, the previously
recognized state-of-the-art open-source model. The dataset and the model will
be publicly released\url{https://github.com/thunlp/UltraChat}.
646d6610eb9268aeebc436d3
Ning Ding
admin_assigned
2023-05-24T07:38:25.237Z
60cf4bcb1ce3775ebb86e5d5
/avatars/12bcd18d215abf91f297f93007733148.svg
Ning Ding
user
stingning
646d6610eb9268aeebc436d4
Yulin Chen
646d6610eb9268aeebc436d5
Bokai Xu
claimed_verified
2024-10-15T16:37:21.240Z
6415818a986557e8cac252bf
https://cdn-avatars.huggingface.co/v1/production/uploads/6415818a986557e8cac252bf/T4u9qjRt8P4clF4nOTA4W.jpeg
Bokai Xu
user
bokesyo
646d6610eb9268aeebc436d6
Yujia Qin
646d6610eb9268aeebc436d7
Zhi Zheng
646d6610eb9268aeebc436d8
Shengding Hu
admin_assigned
2023-05-24T07:43:48.835Z
6041bec6c15e823a685f2afc
/avatars/7d09fbe771d3c3312ac7eb6ef83ce053.svg
Shengding Hu
user
ShengdingHu
646d6610eb9268aeebc436d9
Zhiyuan Liu
claimed_verified
2025-06-02T08:03:48.916Z
6472f1f7485a7c8e1bcf507b
/avatars/79dca137583875191ea883b193b630b7.svg
Zhiyuan Liu
user
zibuyu9
646d6610eb9268aeebc436da
Maosong Sun
646d6610eb9268aeebc436db
Bowen Zhou
Enhancing Chat Language Models by Scaling High-quality Instructional
  Conversations
Fine-tuning on instruction data has been widely validated as an effective
practice for implementing chat language models like ChatGPT. Scaling the
diversity and quality of such data, although straightforward, stands a great
chance of leading to improved performance. This paper aims to improve the upper
bound of open-source models further. We first provide a systematically
designed, diverse, informative, large-scale dataset of instructional
conversations, UltraChat, which does not involve human queries. Our objective
is to capture the breadth of interactions that a human might have with an AI
assistant and employs a comprehensive framework to generate multi-turn
conversation iteratively. UltraChat contains 1.5 million high-quality
multi-turn dialogues and covers a wide range of topics and instructions. Our
statistical analysis of UltraChat reveals its superiority in various key
metrics, including scale, average length, diversity, coherence, etc.,
solidifying its position as a leading open-source dataset. Building upon
UltraChat, we fine-tune a LLaMA model to create a powerful conversational
model, UltraLLaMA. Our evaluations indicate that UltraLLaMA consistently
outperforms other open-source models, including Vicuna, the previously
recognized state-of-the-art open-source model. The dataset and the model will
be publicly released\url{https://github.com/thunlp/UltraChat}.
A large-scale dataset, UltraChat, and a fine-tuned model, UltraLLaMA, are created to improve open-source chat language models' performance.
Llama model Vicuna conversational model large-scale dataset instructional conversations multi-turn dialogues coherence diversity fine-tuning