title: Unveiling Instruction-Specific Neurons & Experts: An Analytical
    Framework for LLM's Instruction-Following Capabilities
summary: The finetuning of Large Language Models (LLMs) has significantly advanced
  their instruction-following capabilities, yet the underlying computational
  mechanisms driving these improvements remain poorly understood. This study
  systematically examines how fine-tuning reconfigures LLM computations by
  isolating and analyzing instruction-specific sparse components, i.e., neurons
  in dense models and both neurons and experts in Mixture-of-Experts (MoE)
  architectures. In particular, we introduce HexaInst, a carefully curated and
  balanced instructional dataset spanning six distinct categories, and propose
  SPARCOM, a novel analytical framework comprising three key contributions: (1) a
  method for identifying these sparse components, (2) an evaluation of their
  functional generality and uniqueness, and (3) a systematic comparison of their
  alterations. Through experiments, we demonstrate functional generality,
  uniqueness, and the critical role of these components in instruction execution.
  By elucidating the relationship between fine-tuning-induced adaptations and
  sparse computational substrates, this work provides deeper insights into how
  LLMs internalize instruction-following behavior for the trustworthy LLM
  community.
paper_authors._id: 6837eb38f09a146728a4b80f
paper_authors.name: Junyan Zhang
paper_authors._id: 6837eb38f09a146728a4b810
paper_authors.name: Yubo Gao
paper_authors._id: 6837eb38f09a146728a4b811
paper_authors.name: Yibo Yan
paper_authors._id: 6837eb38f09a146728a4b812
paper_authors.name: Jungang Li
paper_authors._id: 6837eb38f09a146728a4b813
paper_authors.name: Zhaorui Hou
paper_authors._id: 6837eb38f09a146728a4b814
paper_authors.name: Sicheng Tao
paper_authors._id: 6837eb38f09a146728a4b815
paper_authors.name: Shuliang Liu
paper_authors._id: 6837eb38f09a146728a4b816
paper_authors.name: Song Dai
paper_authors._id: 6837eb38f09a146728a4b817
paper_authors.name: Yonghua Hei
paper_authors._id: 6837eb38f09a146728a4b818
paper_authors.name: Junzhuo Li
paper_authors._id: 6837eb38f09a146728a4b819
paper_authors.name: Xuming Hu
paper_title: Unveiling Instruction-Specific Neurons & Experts: An Analytical
    Framework for LLM's Instruction-Following Capabilities
paper_summary: The finetuning of Large Language Models (LLMs) has significantly advanced
  their instruction-following capabilities, yet the underlying computational
  mechanisms driving these improvements remain poorly understood. This study
  systematically examines how fine-tuning reconfigures LLM computations by
  isolating and analyzing instruction-specific sparse components, i.e., neurons
  in dense models and both neurons and experts in Mixture-of-Experts (MoE)
  architectures. In particular, we introduce HexaInst, a carefully curated and
  balanced instructional dataset spanning six distinct categories, and propose
  SPARCOM, a novel analytical framework comprising three key contributions: (1) a
  method for identifying these sparse components, (2) an evaluation of their
  functional generality and uniqueness, and (3) a systematic comparison of their
  alterations. Through experiments, we demonstrate functional generality,
  uniqueness, and the critical role of these components in instruction execution.
  By elucidating the relationship between fine-tuning-induced adaptations and
  sparse computational substrates, this work provides deeper insights into how
  LLMs internalize instruction-following behavior for the trustworthy LLM
  community.
paper_ai_summary: The study investigates the role of sparse computational components in the instruction-following capabilities of Large Language Models through systematic analysis and introduces HexaInst and SPARCOM for better understanding.
paper_ai_keywords: Large Language Models fine-tuning instruction-following HexaInst SPARCOM sparse components neurons Mixture-of-Experts instruction execution computational adaptations