title: Drag View: Generalizable Novel View Synthesis with Unposed Imagery
summary: We introduce DragView, a novel and interactive framework for generating novel
  views of unseen scenes. DragView initializes the new view from a single source
  image, and the rendering is supported by a sparse set of unposed multi-view
  images, all seamlessly executed within a single feed-forward pass. Our approach
  begins with users dragging a source view through a local relative coordinate
  system. Pixel-aligned features are obtained by projecting the sampled 3D points
  along the target ray onto the source view. We then incorporate a view-dependent
  modulation layer to effectively handle occlusion during the projection.
  Additionally, we broaden the epipolar attention mechanism to encompass all
  source pixels, facilitating the aggregation of initialized coordinate-aligned
  point features from other unposed views. Finally, we employ another transformer
  to decode ray features into final pixel intensities. Crucially, our framework
  does not rely on either 2D prior models or the explicit estimation of camera
  poses. During testing, DragView showcases the capability to generalize to new
  scenes unseen during training, also utilizing only unposed support images,
  enabling the generation of photo-realistic new views characterized by flexible
  camera trajectories. In our experiments, we conduct a comprehensive comparison
  of the performance of DragView with recent scene representation networks
  operating under pose-free conditions, as well as with generalizable NeRFs
  subject to noisy test camera poses. DragView consistently demonstrates its
  superior performance in view synthesis quality, while also being more
  user-friendly. Project page: https://zhiwenfan.github.io/DragView/.
paper_authors._id: 651f6ab9630a846432fbdcc2
paper_authors.name: Zhiwen Fan
paper_authors._id: 651f6ab9630a846432fbdcc3
paper_authors.name: Panwang Pan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-07T11:20:55.777Z
paper_authors._id: 654866e8cd0a5621395f8287
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/654866e8cd0a5621395f8287/4Bccwd1ehn-Ee4T1rId5S.jpeg
paper_authors.fullname: Panwang Pan
paper_authors.type: user
paper_authors.user: paulpanwang
paper_authors._id: 651f6ab9630a846432fbdcc4
paper_authors.name: Peihao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-10T08:38:34.953Z
paper_authors._id: 62c8f1773e28b8ee4cddca09
paper_authors.avatarUrl: /avatars/9edbcc84e4ab2b335011daab8acfdbb8.svg
paper_authors.fullname: Peihao Wang
paper_authors.type: user
paper_authors.user: peihaowang
paper_authors._id: 651f6ab9630a846432fbdcc5
paper_authors.name: Yifan Jiang
paper_authors._id: 651f6ab9630a846432fbdcc6
paper_authors.name: Hanwen Jiang
paper_authors._id: 651f6ab9630a846432fbdcc7
paper_authors.name: Dejia Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-10T08:40:06.828Z
paper_authors._id: 62f687a0c58915315c4ff75d
paper_authors.avatarUrl: /avatars/b657180c7666735062782edd4f6a69c9.svg
paper_authors.fullname: Dejia Xu
paper_authors.type: user
paper_authors.user: ir1d
paper_authors._id: 651f6ab9630a846432fbdcc8
paper_authors.name: Zehao Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-10T08:40:19.933Z
paper_authors._id: 63611b5b6bd72c97d0fe5dc8
paper_authors.avatarUrl: /avatars/8905d4c0a930c7ba6e057c8b01f22a2e.svg
paper_authors.fullname: Zehao Zhu
paper_authors.type: user
paper_authors.user: zzhzzh
paper_authors._id: 651f6ab9630a846432fbdcc9
paper_authors.name: Dilin Wang
paper_authors._id: 651f6ab9630a846432fbdcca
paper_authors.name: Zhangyang Wang
paper_title: Drag View: Generalizable Novel View Synthesis with Unposed Imagery
paper_summary: We introduce DragView, a novel and interactive framework for generating novel
  views of unseen scenes. DragView initializes the new view from a single source
  image, and the rendering is supported by a sparse set of unposed multi-view
  images, all seamlessly executed within a single feed-forward pass. Our approach
  begins with users dragging a source view through a local relative coordinate
  system. Pixel-aligned features are obtained by projecting the sampled 3D points
  along the target ray onto the source view. We then incorporate a view-dependent
  modulation layer to effectively handle occlusion during the projection.
  Additionally, we broaden the epipolar attention mechanism to encompass all
  source pixels, facilitating the aggregation of initialized coordinate-aligned
  point features from other unposed views. Finally, we employ another transformer
  to decode ray features into final pixel intensities. Crucially, our framework
  does not rely on either 2D prior models or the explicit estimation of camera
  poses. During testing, DragView showcases the capability to generalize to new
  scenes unseen during training, also utilizing only unposed support images,
  enabling the generation of photo-realistic new views characterized by flexible
  camera trajectories. In our experiments, we conduct a comprehensive comparison
  of the performance of DragView with recent scene representation networks
  operating under pose-free conditions, as well as with generalizable NeRFs
  subject to noisy test camera poses. DragView consistently demonstrates its
  superior performance in view synthesis quality, while also being more
  user-friendly. Project page: https://zhiwenfan.github.io/DragView/.
paper_ai_summary: DragView is an interactive framework that generates novel views of unseen scenes using a single source image and sparse unposed multi-view images, utilizing an epipolar attention mechanism and transformer-based ray decoding without estimating camera poses.
paper_ai_keywords: interactive framework DragView view synthesis source image unposed multi-view images feed-forward pass pixel-aligned features view-dependent modulation epipolar attention mechanism transformer ray features photo-realistic views scene representation networks pose-free conditions NeRF