title: Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for
    Language Models
summary: We introduce Generalized Instruction Tuning (called GLAN), a general and
  scalable method for instruction tuning of Large Language Models (LLMs). Unlike
  prior work that relies on seed examples or existing datasets to construct
  instruction tuning data, GLAN exclusively utilizes a pre-curated taxonomy of
  human knowledge and capabilities as input and generates large-scale synthetic
  instruction data across all disciplines. Specifically, inspired by the
  systematic structure in human education system, we build the taxonomy by
  decomposing human knowledge and capabilities to various fields, sub-fields and
  ultimately, distinct disciplines semi-automatically, facilitated by LLMs.
  Subsequently, we generate a comprehensive list of subjects for every discipline
  and proceed to design a syllabus tailored to each subject, again utilizing
  LLMs. With the fine-grained key concepts detailed in every class session of the
  syllabus, we are able to generate diverse instructions with a broad coverage
  across the entire spectrum of human knowledge and skills. Extensive experiments
  on large language models (e.g., Mistral) demonstrate that GLAN excels in
  multiple dimensions from mathematical reasoning, coding, academic exams,
  logical reasoning to general instruction following without using task-specific
  training data of these tasks. In addition, GLAN allows for easy customization
  and new fields or skills can be added by simply incorporating a new node into
  our taxonomy.
paper_authors._id: 65d572a0b8c5655c03af4438
paper_authors.name: Haoran Li
paper_authors._id: 65d572a0b8c5655c03af4439
paper_authors.name: Qingxiu Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:03:12.860Z
paper_authors._id: 670740744341dcee459fb990
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/66UkZvrAk7fQr5YCylEFk.png
paper_authors.fullname: Qingxiu Dong
paper_authors.type: user
paper_authors.user: Rsy24
paper_authors._id: 65d572a0b8c5655c03af443a
paper_authors.name: Zhengyang Tang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-06T10:10:03.260Z
paper_authors._id: 64912976b95c3f0a1e6233cb
paper_authors.avatarUrl: /avatars/c0615f8c6606073faffb419757d4e667.svg
paper_authors.fullname: Zhengyang Tang
paper_authors.type: user
paper_authors.user: tangzhy
paper_authors._id: 65d572a0b8c5655c03af443b
paper_authors.name: Chaojun Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T10:01:59.971Z
paper_authors._id: 65012ca2ea5699b59ab7f44c
paper_authors.avatarUrl: /avatars/526a5dd5d2289525a5b5fe70ed79c295.svg
paper_authors.fullname: Chaojun Wang
paper_authors.type: user
paper_authors.user: chaojwang
paper_authors._id: 65d572a0b8c5655c03af443c
paper_authors.name: Xingxing Zhang
paper_authors._id: 65d572a0b8c5655c03af443d
paper_authors.name: Haoyang Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T10:04:40.250Z
paper_authors._id: 63455a98ac1cb29fb2a55907
paper_authors.avatarUrl: /avatars/1dfa775e2816725faf0133ecc7439c0a.svg
paper_authors.fullname: Huang Haoyang
paper_authors.type: user
paper_authors.user: Knighhy
paper_authors._id: 65d572a0b8c5655c03af443e
paper_authors.name: Shaohan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T10:05:42.916Z
paper_authors._id: 632bd2f72d6a805eeb4bc601
paper_authors.avatarUrl: /avatars/6e1533e8a599f3068290aa69ac82cab7.svg
paper_authors.fullname: HUANG SHAOHAN
paper_authors.type: user
paper_authors.user: buaahsh
paper_authors._id: 65d572a0b8c5655c03af443f
paper_authors.name: Xiaolong Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T10:06:20.733Z
paper_authors._id: 64149ba8e682c73521d46ecb
paper_authors.avatarUrl: /avatars/c1859aa2d295077e798c8bdc83664a07.svg
paper_authors.fullname: Xiaolong Huang
paper_authors.type: user
paper_authors.user: DiamondH
paper_authors._id: 65d572a0b8c5655c03af4440
paper_authors.name: Zeqiang Huang
paper_authors._id: 65d572a0b8c5655c03af4441
paper_authors.name: Dongdong Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T10:07:10.504Z
paper_authors._id: 640809b4656adc27c7733ee0
paper_authors.avatarUrl: /avatars/e4f075e4927dae7845d17d396addd3a2.svg
paper_authors.fullname: Dongdong Zhang
paper_authors.type: user
paper_authors.user: zddfunseeker
paper_authors._id: 65d572a0b8c5655c03af4442
paper_authors.name: Yuxian Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T10:07:49.252Z
paper_authors._id: 624ac662102fcdff87be51b9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/624ac662102fcdff87be51b9/rzNahZFFkp194170tactJ.jpeg
paper_authors.fullname: Yuxian Gu
paper_authors.type: user
paper_authors.user: t1101675
paper_authors._id: 65d572a0b8c5655c03af4443
paper_authors.name: Xin Cheng
paper_authors._id: 65d572a0b8c5655c03af4444
paper_authors.name: Xun Wang
paper_authors._id: 65d572a0b8c5655c03af4445
paper_authors.name: Si-Qing Chen
paper_authors._id: 65d572a0b8c5655c03af4446
paper_authors.name: Li Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T15:19:17.125Z
paper_authors._id: 5df85abada6d0311fd3d5408
paper_authors.avatarUrl: /avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
paper_authors.fullname: Li Dong
paper_authors.type: user
paper_authors.user: unilm
paper_authors._id: 65d572a0b8c5655c03af4447
paper_authors.name: Wei Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T10:10:10.045Z
paper_authors._id: 6447f0ed19538c015b27ecb8
paper_authors.avatarUrl: /avatars/aae591acfe36e3297d57103d1bda12cf.svg
paper_authors.fullname: Wei Lu
paper_authors.type: user
paper_authors.user: weilu
paper_authors._id: 65d572a0b8c5655c03af4448
paper_authors.name: Zhifang Sui
paper_authors._id: 65d572a0b8c5655c03af4449
paper_authors.name: Benyou Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T10:11:03.115Z
paper_authors._id: 637c6703ca8542a0ba900ccb
paper_authors.avatarUrl: /avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
paper_authors.fullname: Wang
paper_authors.type: user
paper_authors.user: Benyou
paper_authors._id: 65d572a0b8c5655c03af444a
paper_authors.name: Wai Lam
paper_authors._id: 65d572a0b8c5655c03af444b
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T10:11:17.172Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for
    Language Models
paper_summary: We introduce Generalized Instruction Tuning (called GLAN), a general and
  scalable method for instruction tuning of Large Language Models (LLMs). Unlike
  prior work that relies on seed examples or existing datasets to construct
  instruction tuning data, GLAN exclusively utilizes a pre-curated taxonomy of
  human knowledge and capabilities as input and generates large-scale synthetic
  instruction data across all disciplines. Specifically, inspired by the
  systematic structure in human education system, we build the taxonomy by
  decomposing human knowledge and capabilities to various fields, sub-fields and
  ultimately, distinct disciplines semi-automatically, facilitated by LLMs.
  Subsequently, we generate a comprehensive list of subjects for every discipline
  and proceed to design a syllabus tailored to each subject, again utilizing
  LLMs. With the fine-grained key concepts detailed in every class session of the
  syllabus, we are able to generate diverse instructions with a broad coverage
  across the entire spectrum of human knowledge and skills. Extensive experiments
  on large language models (e.g., Mistral) demonstrate that GLAN excels in
  multiple dimensions from mathematical reasoning, coding, academic exams,
  logical reasoning to general instruction following without using task-specific
  training data of these tasks. In addition, GLAN allows for easy customization
  and new fields or skills can be added by simply incorporating a new node into
  our taxonomy.
paper_ai_summary: GLAN, a general method for instruction tuning of LLMs, uses a taxonomy of human knowledge to generate synthetic instruction data, achieving strong performance across diverse tasks without task-specific training data.
paper_ai_keywords: Generalized Instruction Tuning GLAN Large Language Models LLMs pre-curated taxonomy human knowledge systematic structure human education system fields sub-fields disciplines key concepts class session syllabus mathematical reasoning coding academic exams logical reasoning general instruction following