Ideas in Inference-time Scaling can Benefit Generative Pre-training
  Algorithms
Recent years have seen significant advancements in foundation models through
generative pre-training, yet algorithmic innovation in this space has largely
stagnated around autoregressive models for discrete signals and diffusion
models for continuous signals. This stagnation creates a bottleneck that
prevents us from fully unlocking the potential of rich multi-modal data, which
in turn limits the progress on multimodal intelligence. We argue that an
inference-first perspective, which prioritizes scaling efficiency during
inference time across sequence length and refinement steps, can inspire novel
generative pre-training algorithms. Using Inductive Moment Matching (IMM) as a
concrete example, we demonstrate how addressing limitations in diffusion
models' inference process through targeted modifications yields a stable,
single-stage algorithm that achieves superior sample quality with over an order
of magnitude greater inference efficiency.
67d08abc873f56a3751f50e6
Jiaming Song
67d08abc873f56a3751f50e7
Linqi Zhou
claimed_verified
2025-03-12T08:41:58.835Z
64554c3c9d37c3fb332975ed
/avatars/13001d48b56cca99471a7e75116b70cf.svg
Linqi Zhou
user
alexzhou907
Ideas in Inference-time Scaling can Benefit Generative Pre-training
  Algorithms
Recent years have seen significant advancements in foundation models through
generative pre-training, yet algorithmic innovation in this space has largely
stagnated around autoregressive models for discrete signals and diffusion
models for continuous signals. This stagnation creates a bottleneck that
prevents us from fully unlocking the potential of rich multi-modal data, which
in turn limits the progress on multimodal intelligence. We argue that an
inference-first perspective, which prioritizes scaling efficiency during
inference time across sequence length and refinement steps, can inspire novel
generative pre-training algorithms. Using Inductive Moment Matching (IMM) as a
concrete example, we demonstrate how addressing limitations in diffusion
models' inference process through targeted modifications yields a stable,
single-stage algorithm that achieves superior sample quality with over an order
of magnitude greater inference efficiency.
Inference-first perspective improves generative pre-training by enhancing diffusion models' inference efficiency and sample quality through modifications like Inductive Moment Matching.
generative pre-training autoregressive models diffusion models multimodal intelligence inference efficiency Inductive Moment Matching single-stage algorithm sample quality