DSO: Aligning 3D Generators with Simulation Feedback for Physical
  Soundness
Most 3D object generators focus on aesthetic quality, often neglecting
physical constraints necessary in applications. One such constraint is that the
3D object should be self-supporting, i.e., remains balanced under gravity.
Prior approaches to generating stable 3D objects used differentiable physics
simulators to optimize geometry at test-time, which is slow, unstable, and
prone to local optima. Inspired by the literature on aligning generative models
to external feedback, we propose Direct Simulation Optimization (DSO), a
framework to use the feedback from a (non-differentiable) simulator to increase
the likelihood that the 3D generator outputs stable 3D objects directly. We
construct a dataset of 3D objects labeled with a stability score obtained from
the physics simulator. We can then fine-tune the 3D generator using the
stability score as the alignment metric, via direct preference optimization
(DPO) or direct reward optimization (DRO), a novel objective, which we
introduce, to align diffusion models without requiring pairwise preferences.
Our experiments show that the fine-tuned feed-forward generator, using either
DPO or DRO objective, is much faster and more likely to produce stable objects
than test-time optimization. Notably, the DSO framework works even without any
ground-truth 3D objects for training, allowing the 3D generator to self-improve
by automatically collecting simulation feedback on its own outputs.
67ec5daf3dc5c42c327e161c
Ruining Li
claimed_verified
2025-04-02T08:23:03.470Z
6467cc17e7a6a374fd1a41e5
/avatars/3cbd4ad5bc17067ee38b13d0cbadd80c.svg
Ruining Li
user
rayli
67ec5daf3dc5c42c327e161d
Chuanxia Zheng
67ec5daf3dc5c42c327e161e
Christian Rupprecht
67ec5daf3dc5c42c327e161f
Andrea Vedaldi
DSO: Aligning 3D Generators with Simulation Feedback for Physical
  Soundness
Most 3D object generators focus on aesthetic quality, often neglecting
physical constraints necessary in applications. One such constraint is that the
3D object should be self-supporting, i.e., remains balanced under gravity.
Prior approaches to generating stable 3D objects used differentiable physics
simulators to optimize geometry at test-time, which is slow, unstable, and
prone to local optima. Inspired by the literature on aligning generative models
to external feedback, we propose Direct Simulation Optimization (DSO), a
framework to use the feedback from a (non-differentiable) simulator to increase
the likelihood that the 3D generator outputs stable 3D objects directly. We
construct a dataset of 3D objects labeled with a stability score obtained from
the physics simulator. We can then fine-tune the 3D generator using the
stability score as the alignment metric, via direct preference optimization
(DPO) or direct reward optimization (DRO), a novel objective, which we
introduce, to align diffusion models without requiring pairwise preferences.
Our experiments show that the fine-tuned feed-forward generator, using either
DPO or DRO objective, is much faster and more likely to produce stable objects
than test-time optimization. Notably, the DSO framework works even without any
ground-truth 3D objects for training, allowing the 3D generator to self-improve
by automatically collecting simulation feedback on its own outputs.
The Direct Simulation Optimization (DSO) framework integrates physics simulation feedback to enhance the stability of 3D objects generated by diffusion models, achieving faster and more reliable results without ground-truth data.
Differentiable physics simulators Direct Simulation Optimization (DSO) stability score direct preference optimization (DPO) direct reward optimization (DRO) diffusion models 3D generator self-supporting feed-forward generator