title: Consolidating Attention Features for Multi-view Image Editing
summary: Large-scale text-to-image models enable a wide range of image editing
  techniques, using text prompts or even spatial controls. However, applying
  these editing methods to multi-view images depicting a single scene leads to
  3D-inconsistent results. In this work, we focus on spatial control-based
  geometric manipulations and introduce a method to consolidate the editing
  process across various views. We build on two insights: (1) maintaining
  consistent features throughout the generative process helps attain consistency
  in multi-view editing, and (2) the queries in self-attention layers
  significantly influence the image structure. Hence, we propose to improve the
  geometric consistency of the edited images by enforcing the consistency of the
  queries. To do so, we introduce QNeRF, a neural radiance field trained on the
  internal query features of the edited images. Once trained, QNeRF can render
  3D-consistent queries, which are then softly injected back into the
  self-attention layers during generation, greatly improving multi-view
  consistency. We refine the process through a progressive, iterative method that
  better consolidates queries across the diffusion timesteps. We compare our
  method to a range of existing techniques and demonstrate that it can achieve
  better multi-view consistency and higher fidelity to the input scene. These
  advantages allow us to train NeRFs with fewer visual artifacts, that are better
  aligned with the target geometry.
paper_authors._id: 65d81ceef5d21ab1937144f8
paper_authors.name: Or Patashnik
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-23T13:20:54.813Z
paper_authors._id: 62853516e483e0d37b354ce1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62853516e483e0d37b354ce1/t5Tyd3E07w26B9Z3XpZWI.jpeg
paper_authors.fullname: Or Patashnik
paper_authors.type: user
paper_authors.user: orpatashnik
paper_authors._id: 65d81ceef5d21ab1937144f9
paper_authors.name: Rinon Gal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-23T13:21:01.320Z
paper_authors._id: 627c1360f19c5eb46d55ba05
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1652712871747-627c1360f19c5eb46d55ba05.jpeg
paper_authors.fullname: Rinon Gal
paper_authors.type: user
paper_authors.user: rinong
paper_authors._id: 65d81ceef5d21ab1937144fa
paper_authors.name: Daniel Cohen-Or
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-23T13:21:07.874Z
paper_authors._id: 628507161949ebcae8e24ec3
paper_authors.avatarUrl: /avatars/008ecb3daa4c8187b5f339f1176b3c39.svg
paper_authors.fullname: Daniel Cohen-Or
paper_authors.type: user
paper_authors.user: cohenor
paper_authors._id: 65d81ceef5d21ab1937144fb
paper_authors.name: Jun-Yan Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-23T13:21:13.995Z
paper_authors._id: 63a0acc32fabbbb899952a2b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671474335794-noauth.jpeg
paper_authors.fullname: Jun-Yan Zhu
paper_authors.type: user
paper_authors.user: junyanz
paper_authors._id: 65d81ceef5d21ab1937144fc
paper_authors.name: Fernando De la Torre
paper_title: Consolidating Attention Features for Multi-view Image Editing
paper_summary: Large-scale text-to-image models enable a wide range of image editing
  techniques, using text prompts or even spatial controls. However, applying
  these editing methods to multi-view images depicting a single scene leads to
  3D-inconsistent results. In this work, we focus on spatial control-based
  geometric manipulations and introduce a method to consolidate the editing
  process across various views. We build on two insights: (1) maintaining
  consistent features throughout the generative process helps attain consistency
  in multi-view editing, and (2) the queries in self-attention layers
  significantly influence the image structure. Hence, we propose to improve the
  geometric consistency of the edited images by enforcing the consistency of the
  queries. To do so, we introduce QNeRF, a neural radiance field trained on the
  internal query features of the edited images. Once trained, QNeRF can render
  3D-consistent queries, which are then softly injected back into the
  self-attention layers during generation, greatly improving multi-view
  consistency. We refine the process through a progressive, iterative method that
  better consolidates queries across the diffusion timesteps. We compare our
  method to a range of existing techniques and demonstrate that it can achieve
  better multi-view consistency and higher fidelity to the input scene. These
  advantages allow us to train NeRFs with fewer visual artifacts, that are better
  aligned with the target geometry.
paper_ai_summary: A method enforces geometric consistency in multi-view image editing by injecting 3D-consistent queries into self-attention layers, improving fidelity and reducing visual artifacts when training NeRFs.
paper_ai_keywords: neural radiance field QNeRF self-attention layers diffusion timesteps multi-view consistency geometric manipulation query features