title: Return of the Encoder: Maximizing Parameter Efficiency for SLMs
summary: The dominance of large decoder-only language models has overshadowed
  encoder-decoder architectures, despite their fundamental efficiency advantages
  in sequence processing. For small language models (SLMs) - those with 1 billion
  parameters or fewer - our systematic analysis across GPU, CPU, and NPU
  platforms reveals that encoder-decoder architectures achieve 47% lower
  first-token latency and 4.7x higher throughput compared to decoder-only models
  on edge devices. These gains may be attributed to encoder-decoder's one-time
  input processing and efficient separation of understanding and generation
  phases.
    We introduce a novel knowledge distillation framework that enables
  encoder-decoder models to leverage capabilities from large scalable
  decoder-only teachers while preserving their architectural advantages,
  achieving up to 6 average performance points improvement across diverse tasks,
  with significant gains in asymmetric sequence tasks where input and output
  distributions can benefit from different processing approaches.
    When combined with modern advances like Rotary Positional Embeddings (RoPE)
  and Vision encoders, our systematic investigation demonstrates that
  encoder-decoder architectures provide a more practical path toward deploying
  capable language models in resource-constrained environments. Our findings
  challenge the prevailing trend toward decoder-only scaling, showing that
  architectural choices become increasingly crucial as parameter budgets
  decrease, particularly for on-device and edge deployments where computational
  efficiency is paramount.
paper_authors._id: 67984addd46e4d88ee27f43f
paper_authors.name: Mohamed Elfeki
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-28T10:10:17.395Z
paper_authors._id: 67984a3a02ff123f680a15c6
paper_authors.avatarUrl: /avatars/3595c8962e1739325ba03ead8f76d2e9.svg
paper_authors.fullname: Mohamed Elfeki
paper_authors.type: user
paper_authors.user: melfeki11
paper_authors._id: 67984addd46e4d88ee27f440
paper_authors.name: Rui Liu
paper_authors._id: 67984addd46e4d88ee27f441
paper_authors.name: Chad Voegele
paper_title: Return of the Encoder: Maximizing Parameter Efficiency for SLMs
paper_summary: The dominance of large decoder-only language models has overshadowed
  encoder-decoder architectures, despite their fundamental efficiency advantages
  in sequence processing. For small language models (SLMs) - those with 1 billion
  parameters or fewer - our systematic analysis across GPU, CPU, and NPU
  platforms reveals that encoder-decoder architectures achieve 47% lower
  first-token latency and 4.7x higher throughput compared to decoder-only models
  on edge devices. These gains may be attributed to encoder-decoder's one-time
  input processing and efficient separation of understanding and generation
  phases.
    We introduce a novel knowledge distillation framework that enables
  encoder-decoder models to leverage capabilities from large scalable
  decoder-only teachers while preserving their architectural advantages,
  achieving up to 6 average performance points improvement across diverse tasks,
  with significant gains in asymmetric sequence tasks where input and output
  distributions can benefit from different processing approaches.
    When combined with modern advances like Rotary Positional Embeddings (RoPE)
  and Vision encoders, our systematic investigation demonstrates that
  encoder-decoder architectures provide a more practical path toward deploying
  capable language models in resource-constrained environments. Our findings
  challenge the prevailing trend toward decoder-only scaling, showing that
  architectural choices become increasingly crucial as parameter budgets
  decrease, particularly for on-device and edge deployments where computational
  efficiency is paramount.
paper_ai_summary: Encoder-decoder architectures offer superior efficiency and performance compared to decoder-only models for small language models and low-resource environments, especially when enhanced with knowledge distillation and modern embeddings.
paper_ai_keywords: encoder-decoder architectures decoder-only models small language models SLMs first-token latency throughput knowledge distillation Rotary Positional Embeddings (RoPE) Vision encoders asymmetric sequence tasks architectural advantages parameter budgets resource-constrained environments