title: Scaling Embedding Layers in Language Models
summary: We propose SCONE (Scalable, Contextualized,
  Offloaded, N-gram Embedding), a method for
  extending input embedding layers to enhance language model performance as layer
  size scales. To avoid increased decoding costs, SCONE retains the original
  vocabulary while introducing embeddings for a set of frequent n-grams. These
  embeddings provide contextualized representation for each input token and are
  learned with a separate model during training. During inference, they are
  precomputed and stored in off-accelerator memory with minimal impact on
  inference speed. SCONE enables two new scaling strategies: increasing the
  number of cached n-gram embeddings and scaling the model used to learn them,
  all while maintaining fixed inference-time FLOPS. We show that scaling both
  aspects allows SCONE to outperform a 1.9B parameter baseline across diverse
  corpora, while using only half the inference-time FLOPS.
paper_authors._id: 67a1a51e6aa8429da493d0b5
paper_authors.name: Da Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-05T10:15:26.901Z
paper_authors._id: 64b785384df206a3ed142dc0
paper_authors.avatarUrl: /avatars/501a90b2c80d9b3a2e0d1819a4211f84.svg
paper_authors.fullname: Da Yu
paper_authors.type: user
paper_authors.user: Jellyfish0538
paper_authors._id: 67a1a51e6aa8429da493d0b6
paper_authors.name: Edith Cohen
paper_authors._id: 67a1a51e6aa8429da493d0b7
paper_authors.name: Badih Ghazi
paper_authors._id: 67a1a51e6aa8429da493d0b8
paper_authors.name: Yangsibo Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T21:04:05.965Z
paper_authors._id: 645949e7ecf89b6a375ff129
paper_authors.avatarUrl: /avatars/93f167fe70c43328b95ed597b6dfa51b.svg
paper_authors.fullname: Yangsibo Huang
paper_authors.type: user
paper_authors.user: yangsibo
paper_authors._id: 67a1a51e6aa8429da493d0b9
paper_authors.name: Pritish Kamath
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T21:04:12.118Z
paper_authors._id: 64c0d2f8d76592ba8996036c
paper_authors.avatarUrl: /avatars/7b7720ad2060ac5c36651fee8d43ba69.svg
paper_authors.fullname: Pritish Kamath
paper_authors.type: user
paper_authors.user: pritishkamath
paper_authors._id: 67a1a51e6aa8429da493d0ba
paper_authors.name: Ravi Kumar
paper_authors._id: 67a1a51e6aa8429da493d0bb
paper_authors.name: Daogao Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T20:58:14.948Z
paper_authors._id: 667a19869f78b2d01bc402b1
paper_authors.avatarUrl: /avatars/b7fb00826a62e70d2dae5f978b7366f3.svg
paper_authors.fullname: Daogao Liu
paper_authors.type: user
paper_authors.user: ShyShowmaker
paper_authors._id: 67a1a51e6aa8429da493d0bc
paper_authors.name: Chiyuan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T20:58:08.569Z
paper_authors._id: 65c65a2ef2d7e1eb92256d1f
paper_authors.avatarUrl: /avatars/b02979bc1549a16515c880ce836c3023.svg
paper_authors.fullname: Chiyuan Zhang
paper_authors.type: user
paper_authors.user: pluspluskid
paper_title: Scaling Embedding Layers in Language Models
paper_summary: We propose SCONE (Scalable, Contextualized,
  Offloaded, N-gram Embedding), a method for
  extending input embedding layers to enhance language model performance as layer
  size scales. To avoid increased decoding costs, SCONE retains the original
  vocabulary while introducing embeddings for a set of frequent n-grams. These
  embeddings provide contextualized representation for each input token and are
  learned with a separate model during training. During inference, they are
  precomputed and stored in off-accelerator memory with minimal impact on
  inference speed. SCONE enables two new scaling strategies: increasing the
  number of cached n-gram embeddings and scaling the model used to learn them,
  all while maintaining fixed inference-time FLOPS. We show that scaling both
  aspects allows SCONE to outperform a 1.9B parameter baseline across diverse
  corpora, while using only half the inference-time FLOPS.
paper_ai_keywords: SCONE input embedding layers language model performance decoding costs vocabulary $n$-grams contextualized representation inference speed scaling strategies cached $n$-gram embeddings FLOPS