title: MARBLE: Material Recomposition and Blending in CLIP-Space
summary: Editing materials of objects in images based on exemplar images is an active
  area of research in computer vision and graphics. We propose MARBLE, a method
  for performing material blending and recomposing fine-grained material
  properties by finding material embeddings in CLIP-space and using that to
  control pre-trained text-to-image models. We improve exemplar-based material
  editing by finding a block in the denoising UNet responsible for material
  attribution. Given two material exemplar-images, we find directions in the
  CLIP-space for blending the materials. Further, we can achieve parametric
  control over fine-grained material attributes such as roughness, metallic,
  transparency, and glow using a shallow network to predict the direction for the
  desired material attribute change. We perform qualitative and quantitative
  analysis to demonstrate the efficacy of our proposed method. We also present
  the ability of our method to perform multiple edits in a single forward pass
  and applicability to painting.
    Project Page: https://marblecontrol.github.io/
paper_authors._id: 684297e21cb038a2bda2b024
paper_authors.name: Ta-Ying Cheng
paper_authors._id: 684297e21cb038a2bda2b025
paper_authors.name: Prafull Sharma
paper_authors._id: 684297e21cb038a2bda2b026
paper_authors.name: Mark Boss
paper_authors._id: 684297e21cb038a2bda2b027
paper_authors.name: Varun Jampani
paper_title: MARBLE: Material Recomposition and Blending in CLIP-Space
paper_summary: Editing materials of objects in images based on exemplar images is an active
  area of research in computer vision and graphics. We propose MARBLE, a method
  for performing material blending and recomposing fine-grained material
  properties by finding material embeddings in CLIP-space and using that to
  control pre-trained text-to-image models. We improve exemplar-based material
  editing by finding a block in the denoising UNet responsible for material
  attribution. Given two material exemplar-images, we find directions in the
  CLIP-space for blending the materials. Further, we can achieve parametric
  control over fine-grained material attributes such as roughness, metallic,
  transparency, and glow using a shallow network to predict the direction for the
  desired material attribute change. We perform qualitative and quantitative
  analysis to demonstrate the efficacy of our proposed method. We also present
  the ability of our method to perform multiple edits in a single forward pass
  and applicability to painting.
    Project Page: https://marblecontrol.github.io/
paper_ai_summary: MARBLE utilizes material embeddings in CLIP-space to control pre-trained text-to-image models for blending and recomposing material properties in images with parametric control over attributes.
paper_ai_keywords: material embeddings CLIP-space text-to-image models denoising UNet material attribution parametric control roughness metallic transparency glow shallow network multiple edits