Dreamer XL: Towards High-Resolution Text-to-3D Generation via Trajectory
  Score Matching
In this work, we propose a novel Trajectory Score Matching (TSM) method that
aims to solve the pseudo ground truth inconsistency problem caused by the
accumulated error in Interval Score Matching (ISM) when using the Denoising
Diffusion Implicit Models (DDIM) inversion process. Unlike ISM which adopts the
inversion process of DDIM to calculate on a single path, our TSM method
leverages the inversion process of DDIM to generate two paths from the same
starting point for calculation. Since both paths start from the same starting
point, TSM can reduce the accumulated error compared to ISM, thus alleviating
the problem of pseudo ground truth inconsistency. TSM enhances the stability
and consistency of the model's generated paths during the distillation process.
We demonstrate this experimentally and further show that ISM is a special case
of TSM. Furthermore, to optimize the current multi-stage optimization process
from high-resolution text to 3D generation, we adopt Stable Diffusion XL for
guidance. In response to the issues of abnormal replication and splitting
caused by unstable gradients during the 3D Gaussian splatting process when
using Stable Diffusion XL, we propose a pixel-by-pixel gradient clipping
method. Extensive experiments show that our model significantly surpasses the
state-of-the-art models in terms of visual quality and performance. Code:
https://github.com/xingy038/Dreamer-XL.
664c09bf877f8871f4422023
Xingyu Miao
664c09bf877f8871f4422024
Haoran Duan
664c09bf877f8871f4422025
Varun Ojha
664c09bf877f8871f4422026
Jun Song
664c09bf877f8871f4422027
Tejal Shah
664c09bf877f8871f4422028
Yang Long
664c09bf877f8871f4422029
Rajiv Ranjan
Dreamer XL: Towards High-Resolution Text-to-3D Generation via Trajectory
  Score Matching
In this work, we propose a novel Trajectory Score Matching (TSM) method that
aims to solve the pseudo ground truth inconsistency problem caused by the
accumulated error in Interval Score Matching (ISM) when using the Denoising
Diffusion Implicit Models (DDIM) inversion process. Unlike ISM which adopts the
inversion process of DDIM to calculate on a single path, our TSM method
leverages the inversion process of DDIM to generate two paths from the same
starting point for calculation. Since both paths start from the same starting
point, TSM can reduce the accumulated error compared to ISM, thus alleviating
the problem of pseudo ground truth inconsistency. TSM enhances the stability
and consistency of the model's generated paths during the distillation process.
We demonstrate this experimentally and further show that ISM is a special case
of TSM. Furthermore, to optimize the current multi-stage optimization process
from high-resolution text to 3D generation, we adopt Stable Diffusion XL for
guidance. In response to the issues of abnormal replication and splitting
caused by unstable gradients during the 3D Gaussian splatting process when
using Stable Diffusion XL, we propose a pixel-by-pixel gradient clipping
method. Extensive experiments show that our model significantly surpasses the
state-of-the-art models in terms of visual quality and performance. Code:
https://github.com/xingy038/Dreamer-XL.
TSM, a novel method reducing accumulated error in DDIM inversion, enhances model stability and achieves high visual quality and performance in 3D generation with gradient clipping.
Trajectory Score Matching (TSM) Interval Score Matching (ISM) Denoising Diffusion Implicit Models (DDIM) inversion process 3D generation Stable Diffusion XL Gaussian splatting pixel-by-pixel gradient clipping