title: Leveraging Open Knowledge for Advancing Task Expertise in Large Language
    Models
summary: The cultivation of expertise for large language models (LLMs) to solve tasks
  of specific areas often requires special-purpose tuning with calibrated
  behaviors on the expected stable outputs. To avoid huge cost brought by manual
  preparation of instruction datasets and training resources up to hundreds of
  hours, the exploitation of open knowledge including a wealth of low rank
  adaptation (LoRA) models and instruction datasets serves as a good starting
  point. However, existing methods on model and data selection focus on the
  performance of general-purpose capabilities while neglecting the knowledge gap
  exposed in domain-specific deployment. In the present study, we propose to
  bridge such gap by introducing few human-annotated samples (i.e., K-shot) for
  advancing task expertise of LLMs with open knowledge. Specifically, we develop
  an efficient and scalable pipeline to cost-efficiently produce task experts
  where K-shot data intervene in selecting the most promising expert candidates
  and the task-relevant instructions. A mixture-of-expert (MoE) system is built
  to make the best use of individual-yet-complementary knowledge between multiple
  experts. We unveil the two keys to the success of a MoE system, 1) the abidance
  by K-shot, and 2) the insistence on diversity. For the former, we ensure that
  models that truly possess problem-solving abilities on K-shot are selected
  rather than those blind guessers. Besides, during data selection, instructions
  that share task-relevant contexts with K-shot are prioritized. For the latter,
  we highlight the diversity of constituting experts and that of the fine-tuning
  instructions throughout the model and data selection process. Extensive
  experimental results confirm the superiority of our approach over existing
  methods on utilization of open knowledge across various tasks. Codes and models
  will be released later.
paper_authors._id: 66cfd6e1fa19e148bab3f534
paper_authors.name: Yuncheng Yang
paper_authors._id: 66cfd6e1fa19e148bab3f535
paper_authors.name: Yulei Qin
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-08-29T02:03:46.581Z
paper_authors._id: 6390525c00fb8ec4a424e0ff
paper_authors.avatarUrl: /avatars/4302571e2ef4a9875491221aa630a329.svg
paper_authors.fullname: Yulei Qin
paper_authors.type: user
paper_authors.user: yolay
paper_authors._id: 66cfd6e1fa19e148bab3f536
paper_authors.name: Tong Wu
paper_authors._id: 66cfd6e1fa19e148bab3f537
paper_authors.name: Zihan Xu
paper_authors._id: 66cfd6e1fa19e148bab3f538
paper_authors.name: Gang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-29T07:23:15.001Z
paper_authors._id: 648c17fc17c7ceb9b4200d31
paper_authors.avatarUrl: /avatars/abc5b025dd682935056c33a9e1e362f0.svg
paper_authors.fullname: Gang Li
paper_authors.type: user
paper_authors.user: ligang-cs
paper_authors._id: 66cfd6e1fa19e148bab3f539
paper_authors.name: Pengcheng Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:38:42.185Z
paper_authors._id: 6177f7212a5804028ecc498a
paper_authors.avatarUrl: /avatars/a2a8b88b818d5b9e6869c52a970f9a62.svg
paper_authors.fullname: Pengcheng Guo
paper_authors.type: user
paper_authors.user: pcguo
paper_authors._id: 66cfd6e1fa19e148bab3f53a
paper_authors.name: Hang Shao
paper_authors._id: 66cfd6e1fa19e148bab3f53b
paper_authors.name: Yucheng Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T07:38:55.104Z
paper_authors._id: 64beb6b6140491ca9f803ebf
paper_authors.avatarUrl: /avatars/0daa2e813a13668b8b708cd8c12763d9.svg
paper_authors.fullname: Yucheng SHi
paper_authors.type: user
paper_authors.user: YuchengShi
paper_authors._id: 66cfd6e1fa19e148bab3f53c
paper_authors.name: Ke Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-09T10:11:41.474Z
paper_authors._id: 63280915eeee4dd858083092
paper_authors.avatarUrl: /avatars/78347af4af42527d53e88d9969c5c934.svg
paper_authors.fullname: Ke Li
paper_authors.type: user
paper_authors.user: tristanli
paper_authors._id: 66cfd6e1fa19e148bab3f53d
paper_authors.name: Xing Sun
paper_authors._id: 66cfd6e1fa19e148bab3f53e
paper_authors.name: Jie Yang
paper_authors._id: 66cfd6e1fa19e148bab3f53f
paper_authors.name: Yun Gu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-05-08T02:24:36.692Z
paper_authors._id: 681c15cb4e8877d583db8656
paper_authors.avatarUrl: /avatars/517f4c88a714a1fbea394decb253045b.svg
paper_authors.fullname: yun
paper_authors.type: user
paper_authors.user: geron762
paper_title: Leveraging Open Knowledge for Advancing Task Expertise in Large Language
    Models
paper_summary: The cultivation of expertise for large language models (LLMs) to solve tasks
  of specific areas often requires special-purpose tuning with calibrated
  behaviors on the expected stable outputs. To avoid huge cost brought by manual
  preparation of instruction datasets and training resources up to hundreds of
  hours, the exploitation of open knowledge including a wealth of low rank
  adaptation (LoRA) models and instruction datasets serves as a good starting
  point. However, existing methods on model and data selection focus on the
  performance of general-purpose capabilities while neglecting the knowledge gap
  exposed in domain-specific deployment. In the present study, we propose to
  bridge such gap by introducing few human-annotated samples (i.e., K-shot) for
  advancing task expertise of LLMs with open knowledge. Specifically, we develop
  an efficient and scalable pipeline to cost-efficiently produce task experts
  where K-shot data intervene in selecting the most promising expert candidates
  and the task-relevant instructions. A mixture-of-expert (MoE) system is built
  to make the best use of individual-yet-complementary knowledge between multiple
  experts. We unveil the two keys to the success of a MoE system, 1) the abidance
  by K-shot, and 2) the insistence on diversity. For the former, we ensure that
  models that truly possess problem-solving abilities on K-shot are selected
  rather than those blind guessers. Besides, during data selection, instructions
  that share task-relevant contexts with K-shot are prioritized. For the latter,
  we highlight the diversity of constituting experts and that of the fine-tuning
  instructions throughout the model and data selection process. Extensive
  experimental results confirm the superiority of our approach over existing
  methods on utilization of open knowledge across various tasks. Codes and models
  will be released later.
paper_ai_summary: An efficient pipeline using few human-annotated samples and a mixture-of-experts system enhances the domain-specific capabilities of large language models with open knowledge.
paper_ai_keywords: large language models (LLMs) low rank adaptation (LoRA) few-shot learning (K-shot) mixture-of-experts (MoE) task expertise open knowledge model selection data selection diversity