A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation
  through Low-Rank Clone
Training high-performing Small Language Models (SLMs) remains costly, even
with knowledge distillation and pruning from larger teacher models. Existing
work often faces three key challenges: (1) information loss from hard pruning,
(2) inefficient alignment of representations, and (3) underutilization of
informative activations, particularly from Feed-Forward Networks (FFNs). To
address these challenges, we introduce Low-Rank Clone (LRC), an efficient
pre-training method that constructs SLMs aspiring to behavioral equivalence
with strong teacher models. LRC trains a set of low-rank projection matrices
that jointly enable soft pruning by compressing teacher weights, and activation
clone by aligning student activations, including FFN signals, with those of the
teacher. This unified design maximizes knowledge transfer while removing the
need for explicit alignment modules. Extensive experiments with open-source
teachers (e.g., Llama-3.2-3B-Instruct, Qwen2.5-3B/7B-Instruct) show that LRC
matches or surpasses state-of-the-art models trained on trillions of
tokens--while using only 20B tokens, achieving over 1,000x training efficiency.
Our codes and model checkpoints are available at
https://github.com/CURRENTF/LowRankClone and
https://huggingface.co/collections/JitaiHao/low-rank-clone-lrc-6828389e96a93f1d4219dfaf.
682cac214252bc860663393e
Jitai Hao
extracted_confirmed
2025-05-20T16:23:06.957Z
634781fe0a9fa1ac4999be19
/avatars/bc4f5da2e15193cf7bd1ffde89810ee9.svg
Jitai Hao
user
JitaiHao
682cac214252bc860663393f
Qiang Huang
682cac214252bc8606633940
Hao Liu
682cac214252bc8606633941
Xinyan Xiao
682cac214252bc8606633942
Zhaochun Ren
682cac214252bc8606633943
Jun Yu
A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation
  through Low-Rank Clone
Training high-performing Small Language Models (SLMs) remains costly, even
with knowledge distillation and pruning from larger teacher models. Existing
work often faces three key challenges: (1) information loss from hard pruning,
(2) inefficient alignment of representations, and (3) underutilization of
informative activations, particularly from Feed-Forward Networks (FFNs). To
address these challenges, we introduce Low-Rank Clone (LRC), an efficient
pre-training method that constructs SLMs aspiring to behavioral equivalence
with strong teacher models. LRC trains a set of low-rank projection matrices
that jointly enable soft pruning by compressing teacher weights, and activation
clone by aligning student activations, including FFN signals, with those of the
teacher. This unified design maximizes knowledge transfer while removing the
need for explicit alignment modules. Extensive experiments with open-source
teachers (e.g., Llama-3.2-3B-Instruct, Qwen2.5-3B/7B-Instruct) show that LRC
matches or surpasses state-of-the-art models trained on trillions of
tokens--while using only 20B tokens, achieving over 1,000x training efficiency.
Our codes and model checkpoints are available at
https://github.com/CURRENTF/LowRankClone and
https://huggingface.co/collections/JitaiHao/low-rank-clone-lrc-6828389e96a93f1d4219dfaf.
Low-Rank Clone (LRC) is an efficient pre-training method that enhances Small Language Models (SLMs) by combining soft pruning and activation alignment, achieving high performance with minimal resources.
Small Language Models knowledge distillation pruning Feed-Forward Networks low-rank projection matrices soft pruning activation clone behavioral equivalence LRC training efficiency