title: Text Generation Beyond Discrete Token Sampling
summary: In standard autoregressive generation, an LLM predicts the next-token
  distribution, samples a discrete token, and then discards the distribution,
  passing only the sampled token as new input. To preserve this distribution's
  rich information, we propose Mixture of Inputs (MoI), a training-free method
  for autoregressive generation. After generating a token following the standard
  paradigm, we construct a new input that blends the generated discrete token
  with the previously discarded token distribution. Specifically, we employ a
  Bayesian estimation method that treats the token distribution as the prior, the
  sampled token as the observation, and replaces the conventional one-hot vector
  with the continuous posterior expectation as the new model input. MoI allows
  the model to maintain a richer internal representation throughout the
  generation process, resulting in improved text quality and reasoning
  capabilities. On mathematical reasoning, code generation, and PhD-level QA
  tasks, MoI consistently improves performance across multiple models including
  QwQ-32B, Nemotron-Super-49B, Gemma-3-27B, and DAPO-Qwen-32B, with no additional
  training and negligible computational overhead.
paper_authors._id: 682ea25b2c417303938ae8fc
paper_authors.name: Yufan Zhuang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:55:22.441Z
paper_authors._id: 6438ccbb3b46237de3d052e8
paper_authors.avatarUrl: /avatars/baa624d417b0b905e82127dc66346478.svg
paper_authors.fullname: Yufan Zhuang
paper_authors.type: user
paper_authors.user: yzhuang
paper_authors._id: 682ea25b2c417303938ae8fd
paper_authors.name: Liyuan Liu
paper_authors._id: 682ea25b2c417303938ae8fe
paper_authors.name: Chandan Singh
paper_authors._id: 682ea25b2c417303938ae8ff
paper_authors.name: Jingbo Shang
paper_authors._id: 682ea25b2c417303938ae900
paper_authors.name: Jianfeng Gao
paper_title: Text Generation Beyond Discrete Token Sampling
paper_summary: In standard autoregressive generation, an LLM predicts the next-token
  distribution, samples a discrete token, and then discards the distribution,
  passing only the sampled token as new input. To preserve this distribution's
  rich information, we propose Mixture of Inputs (MoI), a training-free method
  for autoregressive generation. After generating a token following the standard
  paradigm, we construct a new input that blends the generated discrete token
  with the previously discarded token distribution. Specifically, we employ a
  Bayesian estimation method that treats the token distribution as the prior, the
  sampled token as the observation, and replaces the conventional one-hot vector
  with the continuous posterior expectation as the new model input. MoI allows
  the model to maintain a richer internal representation throughout the
  generation process, resulting in improved text quality and reasoning
  capabilities. On mathematical reasoning, code generation, and PhD-level QA
  tasks, MoI consistently improves performance across multiple models including
  QwQ-32B, Nemotron-Super-49B, Gemma-3-27B, and DAPO-Qwen-32B, with no additional
  training and negligible computational overhead.
paper_ai_summary: Mixture of Inputs (MoI), a training-free method, enhances autoregressive generation by maintaining a richer internal representation, improving text quality and reasoning capabilities in mathematical reasoning, code generation, and PhD-level QA tasks.
paper_ai_keywords: Mixture of Inputs (MoI) autoregressive generation token distribution Bayesian estimation posterior expectation