title: A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation
    through Low-Rank Clone
summary: Training high-performing Small Language Models (SLMs) remains costly, even
  with knowledge distillation and pruning from larger teacher models. Existing
  work often faces three key challenges: (1) information loss from hard pruning,
  (2) inefficient alignment of representations, and (3) underutilization of
  informative activations, particularly from Feed-Forward Networks (FFNs). To
  address these challenges, we introduce Low-Rank Clone (LRC), an efficient
  pre-training method that constructs SLMs aspiring to behavioral equivalence
  with strong teacher models. LRC trains a set of low-rank projection matrices
  that jointly enable soft pruning by compressing teacher weights, and activation
  clone by aligning student activations, including FFN signals, with those of the
  teacher. This unified design maximizes knowledge transfer while removing the
  need for explicit alignment modules. Extensive experiments with open-source
  teachers (e.g., Llama-3.2-3B-Instruct, Qwen2.5-3B/7B-Instruct) show that LRC
  matches or surpasses state-of-the-art models trained on trillions of
  tokens--while using only 20B tokens, achieving over 1,000x training efficiency.
  Our codes and model checkpoints are available at
  https://github.com/CURRENTF/LowRankClone and
  https://huggingface.co/collections/JitaiHao/low-rank-clone-lrc-6828389e96a93f1d4219dfaf.
paper_authors._id: 682cac214252bc860663393e
paper_authors.name: Jitai Hao
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-20T16:23:06.957Z
paper_authors._id: 634781fe0a9fa1ac4999be19
paper_authors.avatarUrl: /avatars/bc4f5da2e15193cf7bd1ffde89810ee9.svg
paper_authors.fullname: Jitai Hao
paper_authors.type: user
paper_authors.user: JitaiHao
paper_authors._id: 682cac214252bc860663393f
paper_authors.name: Qiang Huang
paper_authors._id: 682cac214252bc8606633940
paper_authors.name: Hao Liu
paper_authors._id: 682cac214252bc8606633941
paper_authors.name: Xinyan Xiao
paper_authors._id: 682cac214252bc8606633942
paper_authors.name: Zhaochun Ren
paper_authors._id: 682cac214252bc8606633943
paper_authors.name: Jun Yu
paper_title: A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation
    through Low-Rank Clone
paper_summary: Training high-performing Small Language Models (SLMs) remains costly, even
  with knowledge distillation and pruning from larger teacher models. Existing
  work often faces three key challenges: (1) information loss from hard pruning,
  (2) inefficient alignment of representations, and (3) underutilization of
  informative activations, particularly from Feed-Forward Networks (FFNs). To
  address these challenges, we introduce Low-Rank Clone (LRC), an efficient
  pre-training method that constructs SLMs aspiring to behavioral equivalence
  with strong teacher models. LRC trains a set of low-rank projection matrices
  that jointly enable soft pruning by compressing teacher weights, and activation
  clone by aligning student activations, including FFN signals, with those of the
  teacher. This unified design maximizes knowledge transfer while removing the
  need for explicit alignment modules. Extensive experiments with open-source
  teachers (e.g., Llama-3.2-3B-Instruct, Qwen2.5-3B/7B-Instruct) show that LRC
  matches or surpasses state-of-the-art models trained on trillions of
  tokens--while using only 20B tokens, achieving over 1,000x training efficiency.
  Our codes and model checkpoints are available at
  https://github.com/CURRENTF/LowRankClone and
  https://huggingface.co/collections/JitaiHao/low-rank-clone-lrc-6828389e96a93f1d4219dfaf.
paper_ai_summary: Low-Rank Clone (LRC) is an efficient pre-training method that enhances Small Language Models (SLMs) by combining soft pruning and activation alignment, achieving high performance with minimal resources.
paper_ai_keywords: Small Language Models knowledge distillation pruning Feed-Forward Networks low-rank projection matrices soft pruning activation clone behavioral equivalence LRC training efficiency