Score Distillation Sampling with Learned Manifold Corrective
Score Distillation Sampling (SDS) is a recent but already widely popular
method that relies on an image diffusion model to control optimization problems
using text prompts. In this paper, we conduct an in-depth analysis of the SDS
loss function, identify an inherent problem with its formulation, and propose a
surprisingly easy but effective fix. Specifically, we decompose the loss into
different factors and isolate the component responsible for noisy gradients. In
the original formulation, high text guidance is used to account for the noise,
leading to unwanted side effects. Instead, we train a shallow network mimicking
the timestep-dependent denoising deficiency of the image diffusion model in
order to effectively factor it out. We demonstrate the versatility and the
effectiveness of our novel loss formulation through several qualitative and
quantitative experiments, including optimization-based image synthesis and
editing, zero-shot image translation network training, and text-to-3D
synthesis.
659f6862874e583fed675471
Thiemo Alldieck
admin_assigned
2024-01-11T11:20:20.428Z
648c964d39d2584ee47af19c
https://cdn-avatars.huggingface.co/v1/production/uploads/648c964d39d2584ee47af19c/5UEkzDTMY8I3svKjR7kxN.jpeg
Thiemo Alldieck
user
thiemoall
659f6862874e583fed675472
Nikos Kolotouros
admin_assigned
2024-01-11T11:20:28.010Z
648cd29f8f46d7284544083b
/avatars/53163398f1a629b9838548b808af53f7.svg
Nikos Kolotouros
user
kolotouros
659f6862874e583fed675473
Cristian Sminchisescu
Score Distillation Sampling with Learned Manifold Corrective
Score Distillation Sampling (SDS) is a recent but already widely popular
method that relies on an image diffusion model to control optimization problems
using text prompts. In this paper, we conduct an in-depth analysis of the SDS
loss function, identify an inherent problem with its formulation, and propose a
surprisingly easy but effective fix. Specifically, we decompose the loss into
different factors and isolate the component responsible for noisy gradients. In
the original formulation, high text guidance is used to account for the noise,
leading to unwanted side effects. Instead, we train a shallow network mimicking
the timestep-dependent denoising deficiency of the image diffusion model in
order to effectively factor it out. We demonstrate the versatility and the
effectiveness of our novel loss formulation through several qualitative and
quantitative experiments, including optimization-based image synthesis and
editing, zero-shot image translation network training, and text-to-3D
synthesis.
The paper addresses issues in the Score Distillation Sampling loss function by training a shallow network to remove noisy gradients introduced by high text guidance, demonstrating improvements in various applications like image synthesis, network training, and 3D synthesis.
image diffusion model Score Distillation Sampling (SDS) loss function noisy gradients timestep-dependent denoising deficiency optimization-based image synthesis zero-shot image translation text-to-3D synthesis