Ψ-Sampler: Initial Particle Sampling for SMC-Based Inference-Time
  Reward Alignment in Score Models
We introduce Psi-Sampler, an SMC-based framework incorporating pCNL-based
initial particle sampling for effective inference-time reward alignment with a
score-based generative model. Inference-time reward alignment with score-based
generative models has recently gained significant traction, following a broader
paradigm shift from pre-training to post-training optimization. At the core of
this trend is the application of Sequential Monte Carlo (SMC) to the denoising
process. However, existing methods typically initialize particles from the
Gaussian prior, which inadequately captures reward-relevant regions and results
in reduced sampling efficiency. We demonstrate that initializing from the
reward-aware posterior significantly improves alignment performance. To enable
posterior sampling in high-dimensional latent spaces, we introduce the
preconditioned Crank-Nicolson Langevin (pCNL) algorithm, which combines
dimension-robust proposals with gradient-informed dynamics. This approach
enables efficient and scalable posterior sampling and consistently improves
performance across various reward alignment tasks, including layout-to-image
generation, quantity-aware generation, and aesthetic-preference generation, as
demonstrated in our experiments.
684124368cb0edba3ab8f738
Taehoon Yoon
admin_assigned
2025-06-05T12:43:45.956Z
64df25a16466b911722ca1da
/avatars/da92419e4d8ba412f163f85e09055f49.svg
TaeHoon Yoon
user
sillsill777
684124368cb0edba3ab8f739
Yunhong Min
admin_assigned
2025-06-05T12:43:53.200Z
66ee81b676a8038cb42c8caa
/avatars/9b4c5ded9c94788c35ce7ffbc2f8d24b.svg
Yunhong Min
user
myhong
684124368cb0edba3ab8f73a
Kyeongmin Yeo
admin_assigned
2025-06-05T12:44:00.132Z
659e42cfb65ee9ee1fd11e61
/avatars/a9220d099f32800fc43ae79bb519c1e9.svg
Kyeongmin Yeo
user
32V
684124368cb0edba3ab8f73b
Minhyuk Sung
admin_assigned
2025-06-05T12:44:05.457Z
631f432b5ba8c026340a7890
https://cdn-avatars.huggingface.co/v1/production/uploads/631f432b5ba8c026340a7890/9PK7A_TRMpugwYjCsNBf1.jpeg
Minhyuk Sung
user
Minhyuk
Ψ-Sampler: Initial Particle Sampling for SMC-Based Inference-Time
  Reward Alignment in Score Models
We introduce Psi-Sampler, an SMC-based framework incorporating pCNL-based
initial particle sampling for effective inference-time reward alignment with a
score-based generative model. Inference-time reward alignment with score-based
generative models has recently gained significant traction, following a broader
paradigm shift from pre-training to post-training optimization. At the core of
this trend is the application of Sequential Monte Carlo (SMC) to the denoising
process. However, existing methods typically initialize particles from the
Gaussian prior, which inadequately captures reward-relevant regions and results
in reduced sampling efficiency. We demonstrate that initializing from the
reward-aware posterior significantly improves alignment performance. To enable
posterior sampling in high-dimensional latent spaces, we introduce the
preconditioned Crank-Nicolson Langevin (pCNL) algorithm, which combines
dimension-robust proposals with gradient-informed dynamics. This approach
enables efficient and scalable posterior sampling and consistently improves
performance across various reward alignment tasks, including layout-to-image
generation, quantity-aware generation, and aesthetic-preference generation, as
demonstrated in our experiments.
The framework $\Psi$-Sampler uses SMC with pCNL for efficient posterior sampling and reward alignment in score-based generative models, enhancing performance across various tasks.
SMC-based framework pCNL-based initial particle sampling inference-time reward alignment score-based generative model Sequential Monte Carlo denoising process Gaussian prior reward-aware posterior preconditioned Crank-Nicolson Langevin layout-to-image generation quantity-aware generation aesthetic-preference generation