title: Ideas in Inference-time Scaling can Benefit Generative Pre-training
    Algorithms
summary: Recent years have seen significant advancements in foundation models through
  generative pre-training, yet algorithmic innovation in this space has largely
  stagnated around autoregressive models for discrete signals and diffusion
  models for continuous signals. This stagnation creates a bottleneck that
  prevents us from fully unlocking the potential of rich multi-modal data, which
  in turn limits the progress on multimodal intelligence. We argue that an
  inference-first perspective, which prioritizes scaling efficiency during
  inference time across sequence length and refinement steps, can inspire novel
  generative pre-training algorithms. Using Inductive Moment Matching (IMM) as a
  concrete example, we demonstrate how addressing limitations in diffusion
  models' inference process through targeted modifications yields a stable,
  single-stage algorithm that achieves superior sample quality with over an order
  of magnitude greater inference efficiency.
paper_authors._id: 67d08abc873f56a3751f50e6
paper_authors.name: Jiaming Song
paper_authors._id: 67d08abc873f56a3751f50e7
paper_authors.name: Linqi Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T08:41:58.835Z
paper_authors._id: 64554c3c9d37c3fb332975ed
paper_authors.avatarUrl: /avatars/13001d48b56cca99471a7e75116b70cf.svg
paper_authors.fullname: Linqi Zhou
paper_authors.type: user
paper_authors.user: alexzhou907
paper_title: Ideas in Inference-time Scaling can Benefit Generative Pre-training
    Algorithms
paper_summary: Recent years have seen significant advancements in foundation models through
  generative pre-training, yet algorithmic innovation in this space has largely
  stagnated around autoregressive models for discrete signals and diffusion
  models for continuous signals. This stagnation creates a bottleneck that
  prevents us from fully unlocking the potential of rich multi-modal data, which
  in turn limits the progress on multimodal intelligence. We argue that an
  inference-first perspective, which prioritizes scaling efficiency during
  inference time across sequence length and refinement steps, can inspire novel
  generative pre-training algorithms. Using Inductive Moment Matching (IMM) as a
  concrete example, we demonstrate how addressing limitations in diffusion
  models' inference process through targeted modifications yields a stable,
  single-stage algorithm that achieves superior sample quality with over an order
  of magnitude greater inference efficiency.
paper_ai_summary: Inference-first perspective improves generative pre-training by enhancing diffusion models' inference efficiency and sample quality through modifications like Inductive Moment Matching.
paper_ai_keywords: generative pre-training autoregressive models diffusion models multimodal intelligence inference efficiency Inductive Moment Matching single-stage algorithm sample quality