title: Enhancing High-Resolution 3D Generation through Pixel-wise Gradient
    Clipping
summary: High-resolution 3D object generation remains a challenging task primarily due
  to the limited availability of comprehensive annotated training data. Recent
  advancements have aimed to overcome this constraint by harnessing image
  generative models, pretrained on extensive curated web datasets, using
  knowledge transfer techniques like Score Distillation Sampling (SDS).
  Efficiently addressing the requirements of high-resolution rendering often
  necessitates the adoption of latent representation-based models, such as the
  Latent Diffusion Model (LDM). In this framework, a significant challenge
  arises: To compute gradients for individual image pixels, it is necessary to
  backpropagate gradients from the designated latent space through the frozen
  components of the image model, such as the VAE encoder used within LDM.
  However, this gradient propagation pathway has never been optimized, remaining
  uncontrolled during training. We find that the unregulated gradients adversely
  affect the 3D model's capacity in acquiring texture-related information from
  the image generative model, leading to poor quality appearance synthesis. To
  address this overarching challenge, we propose an innovative operation termed
  Pixel-wise Gradient Clipping (PGC) designed for seamless integration into
  existing 3D generative models, thereby enhancing their synthesis quality.
  Specifically, we control the magnitude of stochastic gradients by clipping the
  pixel-wise gradients efficiently, while preserving crucial texture-related
  gradient directions. Despite this simplicity and minimal extra cost, extensive
  experiments demonstrate the efficacy of our PGC in enhancing the performance of
  existing 3D generative models for high-resolution object rendering.
paper_authors._id: 653213a94d00778c4d85c990
paper_authors.name: Zijie Pan
paper_authors._id: 653213a94d00778c4d85c991
paper_authors.name: Jiachen Lu
paper_authors._id: 653213a94d00778c4d85c992
paper_authors.name: Xiatian Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-20T08:45:14.385Z
paper_authors._id: 647ef9e7aa8c04bbf9362a8a
paper_authors.avatarUrl: /avatars/dc23e15acf7ee7f3b9f1a68c2716a66d.svg
paper_authors.fullname: Xiatian Zhu
paper_authors.type: user
paper_authors.user: Xiatian-Zhu
paper_authors._id: 653213a94d00778c4d85c993
paper_authors.name: Li Zhang
paper_title: Enhancing High-Resolution 3D Generation through Pixel-wise Gradient
    Clipping
paper_summary: High-resolution 3D object generation remains a challenging task primarily due
  to the limited availability of comprehensive annotated training data. Recent
  advancements have aimed to overcome this constraint by harnessing image
  generative models, pretrained on extensive curated web datasets, using
  knowledge transfer techniques like Score Distillation Sampling (SDS).
  Efficiently addressing the requirements of high-resolution rendering often
  necessitates the adoption of latent representation-based models, such as the
  Latent Diffusion Model (LDM). In this framework, a significant challenge
  arises: To compute gradients for individual image pixels, it is necessary to
  backpropagate gradients from the designated latent space through the frozen
  components of the image model, such as the VAE encoder used within LDM.
  However, this gradient propagation pathway has never been optimized, remaining
  uncontrolled during training. We find that the unregulated gradients adversely
  affect the 3D model's capacity in acquiring texture-related information from
  the image generative model, leading to poor quality appearance synthesis. To
  address this overarching challenge, we propose an innovative operation termed
  Pixel-wise Gradient Clipping (PGC) designed for seamless integration into
  existing 3D generative models, thereby enhancing their synthesis quality.
  Specifically, we control the magnitude of stochastic gradients by clipping the
  pixel-wise gradients efficiently, while preserving crucial texture-related
  gradient directions. Despite this simplicity and minimal extra cost, extensive
  experiments demonstrate the efficacy of our PGC in enhancing the performance of
  existing 3D generative models for high-resolution object rendering.
paper_ai_keywords: Latent Diffusion Model (LDM) VAE encoder backpropagation latent space gradient propagation texture-related information appearance synthesis Pixel-wise Gradient Clipping (PGC) stochastic gradients pixel-wise gradients high-resolution rendering 3D generative models