title: Exploring the Latent Capacity of LLMs for One-Step Text Generation
summary: A recent study showed that large language models (LLMs) can reconstruct
  surprisingly long texts - up to thousands of tokens - via autoregressive
  generation from just one specially trained input embedding. In this work, we
  explore whether such reconstruction is possible without autoregression. We show
  that frozen LLMs can generate hundreds of accurate tokens in just one forward
  pass, when provided with only two learned embeddings. This reveals a surprising
  and underexplored capability of LLMs - multi-token generation without iterative
  decoding. We investigate the behaviour of these embeddings and provide insight
  into the type of information they encode. We also empirically show that
  although these representations are not unique for a given text, they form
  connected and local regions in embedding space - a property that suggests the
  potential of learning a dedicated encoder into that space.
paper_authors._id: 6836babd75a4c5486bac4149
paper_authors.name: Gleb Mezentsev
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-28T07:43:05.521Z
paper_authors._id: 672e0638ee49faac3ad53af7
paper_authors.avatarUrl: /avatars/3a273b4beba8286309296a1e25bc34a9.svg
paper_authors.fullname: Gleb Mezentsev
paper_authors.type: user
paper_authors.user: glebzok
paper_authors._id: 6836babd75a4c5486bac414a
paper_authors.name: Ivan Oseledets
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:15:02.162Z
paper_authors._id: 6169a581d05945bfd8718dfa
paper_authors.avatarUrl: /avatars/1892ab06a7ddb557232777de3cbec470.svg
paper_authors.fullname: Ivan Oseledets
paper_authors.type: user
paper_authors.user: oseledets
paper_title: Exploring the Latent Capacity of LLMs for One-Step Text Generation
paper_summary: A recent study showed that large language models (LLMs) can reconstruct
  surprisingly long texts - up to thousands of tokens - via autoregressive
  generation from just one specially trained input embedding. In this work, we
  explore whether such reconstruction is possible without autoregression. We show
  that frozen LLMs can generate hundreds of accurate tokens in just one forward
  pass, when provided with only two learned embeddings. This reveals a surprising
  and underexplored capability of LLMs - multi-token generation without iterative
  decoding. We investigate the behaviour of these embeddings and provide insight
  into the type of information they encode. We also empirically show that
  although these representations are not unique for a given text, they form
  connected and local regions in embedding space - a property that suggests the
  potential of learning a dedicated encoder into that space.
paper_ai_summary: LLMs can generate long text segments in a single forward pass using learned embeddings, revealing a capability for multi-token generation without iterative decoding.
paper_ai_keywords: large language models autoregressive generation input embedding frozen LLMs multi-token generation iterative decoding learned embeddings embedding space dedicated encoder