Unveiling Instruction-Specific Neurons & Experts: An Analytical
  Framework for LLM's Instruction-Following Capabilities
The finetuning of Large Language Models (LLMs) has significantly advanced
their instruction-following capabilities, yet the underlying computational
mechanisms driving these improvements remain poorly understood. This study
systematically examines how fine-tuning reconfigures LLM computations by
isolating and analyzing instruction-specific sparse components, i.e., neurons
in dense models and both neurons and experts in Mixture-of-Experts (MoE)
architectures. In particular, we introduce HexaInst, a carefully curated and
balanced instructional dataset spanning six distinct categories, and propose
SPARCOM, a novel analytical framework comprising three key contributions: (1) a
method for identifying these sparse components, (2) an evaluation of their
functional generality and uniqueness, and (3) a systematic comparison of their
alterations. Through experiments, we demonstrate functional generality,
uniqueness, and the critical role of these components in instruction execution.
By elucidating the relationship between fine-tuning-induced adaptations and
sparse computational substrates, this work provides deeper insights into how
LLMs internalize instruction-following behavior for the trustworthy LLM
community.
6837eb38f09a146728a4b80f
Junyan Zhang
6837eb38f09a146728a4b810
Yubo Gao
6837eb38f09a146728a4b811
Yibo Yan
6837eb38f09a146728a4b812
Jungang Li
claimed_verified
2025-06-10T15:20:35.083Z
64b76528fdb702b3d8641514
/avatars/dcf7987e4c54f11fef0459c46f05ed62.svg
Jungang Li
user
Jungang
6837eb38f09a146728a4b813
Zhaorui Hou
6837eb38f09a146728a4b814
Sicheng Tao
6837eb38f09a146728a4b815
Shuliang Liu
6837eb38f09a146728a4b816
Song Dai
6837eb38f09a146728a4b817
Yonghua Hei
6837eb38f09a146728a4b818
Junzhuo Li
6837eb38f09a146728a4b819
Xuming Hu
Unveiling Instruction-Specific Neurons & Experts: An Analytical
  Framework for LLM's Instruction-Following Capabilities
The finetuning of Large Language Models (LLMs) has significantly advanced
their instruction-following capabilities, yet the underlying computational
mechanisms driving these improvements remain poorly understood. This study
systematically examines how fine-tuning reconfigures LLM computations by
isolating and analyzing instruction-specific sparse components, i.e., neurons
in dense models and both neurons and experts in Mixture-of-Experts (MoE)
architectures. In particular, we introduce HexaInst, a carefully curated and
balanced instructional dataset spanning six distinct categories, and propose
SPARCOM, a novel analytical framework comprising three key contributions: (1) a
method for identifying these sparse components, (2) an evaluation of their
functional generality and uniqueness, and (3) a systematic comparison of their
alterations. Through experiments, we demonstrate functional generality,
uniqueness, and the critical role of these components in instruction execution.
By elucidating the relationship between fine-tuning-induced adaptations and
sparse computational substrates, this work provides deeper insights into how
LLMs internalize instruction-following behavior for the trustworthy LLM
community.
The study investigates the role of sparse computational components in the instruction-following capabilities of Large Language Models through systematic analysis and introduces HexaInst and SPARCOM for better understanding.
Large Language Models fine-tuning instruction-following HexaInst SPARCOM sparse components neurons Mixture-of-Experts instruction execution computational adaptations