title: DSO: Aligning 3D Generators with Simulation Feedback for Physical
    Soundness
summary: Most 3D object generators focus on aesthetic quality, often neglecting
  physical constraints necessary in applications. One such constraint is that the
  3D object should be self-supporting, i.e., remains balanced under gravity.
  Prior approaches to generating stable 3D objects used differentiable physics
  simulators to optimize geometry at test-time, which is slow, unstable, and
  prone to local optima. Inspired by the literature on aligning generative models
  to external feedback, we propose Direct Simulation Optimization (DSO), a
  framework to use the feedback from a (non-differentiable) simulator to increase
  the likelihood that the 3D generator outputs stable 3D objects directly. We
  construct a dataset of 3D objects labeled with a stability score obtained from
  the physics simulator. We can then fine-tune the 3D generator using the
  stability score as the alignment metric, via direct preference optimization
  (DPO) or direct reward optimization (DRO), a novel objective, which we
  introduce, to align diffusion models without requiring pairwise preferences.
  Our experiments show that the fine-tuned feed-forward generator, using either
  DPO or DRO objective, is much faster and more likely to produce stable objects
  than test-time optimization. Notably, the DSO framework works even without any
  ground-truth 3D objects for training, allowing the 3D generator to self-improve
  by automatically collecting simulation feedback on its own outputs.
paper_authors._id: 67ec5daf3dc5c42c327e161c
paper_authors.name: Ruining Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-02T08:23:03.470Z
paper_authors._id: 6467cc17e7a6a374fd1a41e5
paper_authors.avatarUrl: /avatars/3cbd4ad5bc17067ee38b13d0cbadd80c.svg
paper_authors.fullname: Ruining Li
paper_authors.type: user
paper_authors.user: rayli
paper_authors._id: 67ec5daf3dc5c42c327e161d
paper_authors.name: Chuanxia Zheng
paper_authors._id: 67ec5daf3dc5c42c327e161e
paper_authors.name: Christian Rupprecht
paper_authors._id: 67ec5daf3dc5c42c327e161f
paper_authors.name: Andrea Vedaldi
paper_title: DSO: Aligning 3D Generators with Simulation Feedback for Physical
    Soundness
paper_summary: Most 3D object generators focus on aesthetic quality, often neglecting
  physical constraints necessary in applications. One such constraint is that the
  3D object should be self-supporting, i.e., remains balanced under gravity.
  Prior approaches to generating stable 3D objects used differentiable physics
  simulators to optimize geometry at test-time, which is slow, unstable, and
  prone to local optima. Inspired by the literature on aligning generative models
  to external feedback, we propose Direct Simulation Optimization (DSO), a
  framework to use the feedback from a (non-differentiable) simulator to increase
  the likelihood that the 3D generator outputs stable 3D objects directly. We
  construct a dataset of 3D objects labeled with a stability score obtained from
  the physics simulator. We can then fine-tune the 3D generator using the
  stability score as the alignment metric, via direct preference optimization
  (DPO) or direct reward optimization (DRO), a novel objective, which we
  introduce, to align diffusion models without requiring pairwise preferences.
  Our experiments show that the fine-tuned feed-forward generator, using either
  DPO or DRO objective, is much faster and more likely to produce stable objects
  than test-time optimization. Notably, the DSO framework works even without any
  ground-truth 3D objects for training, allowing the 3D generator to self-improve
  by automatically collecting simulation feedback on its own outputs.
paper_ai_summary: The Direct Simulation Optimization (DSO) framework integrates physics simulation feedback to enhance the stability of 3D objects generated by diffusion models, achieving faster and more reliable results without ground-truth data.
paper_ai_keywords: Differentiable physics simulators Direct Simulation Optimization (DSO) stability score direct preference optimization (DPO) direct reward optimization (DRO) diffusion models 3D generator self-supporting feed-forward generator