Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for
  Language Models
We introduce Generalized Instruction Tuning (called GLAN), a general and
scalable method for instruction tuning of Large Language Models (LLMs). Unlike
prior work that relies on seed examples or existing datasets to construct
instruction tuning data, GLAN exclusively utilizes a pre-curated taxonomy of
human knowledge and capabilities as input and generates large-scale synthetic
instruction data across all disciplines. Specifically, inspired by the
systematic structure in human education system, we build the taxonomy by
decomposing human knowledge and capabilities to various fields, sub-fields and
ultimately, distinct disciplines semi-automatically, facilitated by LLMs.
Subsequently, we generate a comprehensive list of subjects for every discipline
and proceed to design a syllabus tailored to each subject, again utilizing
LLMs. With the fine-grained key concepts detailed in every class session of the
syllabus, we are able to generate diverse instructions with a broad coverage
across the entire spectrum of human knowledge and skills. Extensive experiments
on large language models (e.g., Mistral) demonstrate that GLAN excels in
multiple dimensions from mathematical reasoning, coding, academic exams,
logical reasoning to general instruction following without using task-specific
training data of these tasks. In addition, GLAN allows for easy customization
and new fields or skills can be added by simply incorporating a new node into
our taxonomy.
65d572a0b8c5655c03af4438
Haoran Li
65d572a0b8c5655c03af4439
Qingxiu Dong
claimed_verified
2024-10-10T08:03:12.860Z
670740744341dcee459fb990
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/66UkZvrAk7fQr5YCylEFk.png
Rosy24
user
Rsy24
65d572a0b8c5655c03af443a
Zhengyang Tang
claimed_verified
2024-03-06T10:10:03.260Z
64912976b95c3f0a1e6233cb
/avatars/c0615f8c6606073faffb419757d4e667.svg
Zhengyang Tang
user
tangzhy
65d572a0b8c5655c03af443b
Chaojun Wang
admin_assigned
2024-02-21T10:01:59.971Z
65012ca2ea5699b59ab7f44c
/avatars/526a5dd5d2289525a5b5fe70ed79c295.svg
Chaojun Wang
user
chaojwang
65d572a0b8c5655c03af443c
Xingxing Zhang
65d572a0b8c5655c03af443d
Haoyang Huang
admin_assigned
2024-02-21T10:04:40.250Z
63455a98ac1cb29fb2a55907
/avatars/1dfa775e2816725faf0133ecc7439c0a.svg
Huang Haoyang
user
Knighhy
65d572a0b8c5655c03af443e
Shaohan Huang
admin_assigned
2024-02-21T10:05:42.916Z
632bd2f72d6a805eeb4bc601
/avatars/6e1533e8a599f3068290aa69ac82cab7.svg
HUANG SHAOHAN
user
buaahsh
65d572a0b8c5655c03af443f
Xiaolong Huang
admin_assigned
2024-02-21T10:06:20.733Z
64149ba8e682c73521d46ecb
/avatars/c1859aa2d295077e798c8bdc83664a07.svg
Xiaolong Huang
user
DiamondH
65d572a0b8c5655c03af4440
Zeqiang Huang
65d572a0b8c5655c03af4441
Dongdong Zhang
admin_assigned
2024-02-21T10:07:10.504Z
640809b4656adc27c7733ee0
/avatars/e4f075e4927dae7845d17d396addd3a2.svg
Dongdong Zhang
user
zddfunseeker
65d572a0b8c5655c03af4442
Yuxian Gu
admin_assigned
2024-02-21T10:07:49.252Z
624ac662102fcdff87be51b9
https://cdn-avatars.huggingface.co/v1/production/uploads/624ac662102fcdff87be51b9/rzNahZFFkp194170tactJ.jpeg
Yuxian Gu
user
t1101675
65d572a0b8c5655c03af4443
Xin Cheng
65d572a0b8c5655c03af4444
Xun Wang
65d572a0b8c5655c03af4445
Si-Qing Chen
65d572a0b8c5655c03af4446
Li Dong
claimed_verified
2024-10-08T15:19:17.125Z
5df85abada6d0311fd3d5408
/avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
Li Dong
user
unilm
65d572a0b8c5655c03af4447
Wei Lu
admin_assigned
2024-02-21T10:10:10.045Z
6447f0ed19538c015b27ecb8
/avatars/aae591acfe36e3297d57103d1bda12cf.svg
Wei Lu
user
weilu
65d572a0b8c5655c03af4448
Zhifang Sui
65d572a0b8c5655c03af4449
Benyou Wang
admin_assigned
2024-02-21T10:11:03.115Z
637c6703ca8542a0ba900ccb
/avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
Wang
user
Benyou
65d572a0b8c5655c03af444a
Wai Lam
65d572a0b8c5655c03af444b
Furu Wei
admin_assigned
2024-02-21T10:11:17.172Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for
  Language Models
We introduce Generalized Instruction Tuning (called GLAN), a general and
scalable method for instruction tuning of Large Language Models (LLMs). Unlike
prior work that relies on seed examples or existing datasets to construct
instruction tuning data, GLAN exclusively utilizes a pre-curated taxonomy of
human knowledge and capabilities as input and generates large-scale synthetic
instruction data across all disciplines. Specifically, inspired by the
systematic structure in human education system, we build the taxonomy by
decomposing human knowledge and capabilities to various fields, sub-fields and
ultimately, distinct disciplines semi-automatically, facilitated by LLMs.
Subsequently, we generate a comprehensive list of subjects for every discipline
and proceed to design a syllabus tailored to each subject, again utilizing
LLMs. With the fine-grained key concepts detailed in every class session of the
syllabus, we are able to generate diverse instructions with a broad coverage
across the entire spectrum of human knowledge and skills. Extensive experiments
on large language models (e.g., Mistral) demonstrate that GLAN excels in
multiple dimensions from mathematical reasoning, coding, academic exams,
logical reasoning to general instruction following without using task-specific
training data of these tasks. In addition, GLAN allows for easy customization
and new fields or skills can be added by simply incorporating a new node into
our taxonomy.
GLAN, a general method for instruction tuning of LLMs, uses a taxonomy of human knowledge to generate synthetic instruction data, achieving strong performance across diverse tasks without task-specific training data.
Generalized Instruction Tuning GLAN Large Language Models LLMs pre-curated taxonomy human knowledge systematic structure human education system fields sub-fields disciplines key concepts class session syllabus mathematical reasoning coding academic exams logical reasoning general instruction following