title: EscherNet: A Generative Model for Scalable View Synthesis
summary: We introduce EscherNet, a multi-view conditioned diffusion model for view
  synthesis. EscherNet learns implicit and generative 3D representations coupled
  with a specialised camera positional encoding, allowing precise and continuous
  relative control of the camera transformation between an arbitrary number of
  reference and target views. EscherNet offers exceptional generality,
  flexibility, and scalability in view synthesis -- it can generate more than 100
  consistent target views simultaneously on a single consumer-grade GPU, despite
  being trained with a fixed number of 3 reference views to 3 target views. As a
  result, EscherNet not only addresses zero-shot novel view synthesis, but also
  naturally unifies single- and multi-image 3D reconstruction, combining these
  diverse tasks into a single, cohesive framework. Our extensive experiments
  demonstrate that EscherNet achieves state-of-the-art performance in multiple
  benchmarks, even when compared to methods specifically tailored for each
  individual problem. This remarkable versatility opens up new directions for
  designing scalable neural architectures for 3D vision. Project page:
  https://kxhit.github.io/EscherNet.
paper_authors._id: 65c306fe400d860a3eae7e9a
paper_authors.name: Xin Kong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-07T20:41:00.936Z
paper_authors._id: 633ef0bfac395449287b3ed9
paper_authors.avatarUrl: /avatars/7e2cc27da7761f4e7af547b7d65528db.svg
paper_authors.fullname: Xin Kong
paper_authors.type: user
paper_authors.user: kxic
paper_authors._id: 65c306fe400d860a3eae7e9b
paper_authors.name: Shikun Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-08T11:52:52.000Z
paper_authors._id: 633ef2d6905131c8f52cfb95
paper_authors.avatarUrl: /avatars/415f61f80feaffca4e35e029b84ba5d2.svg
paper_authors.fullname: Shikun Liu
paper_authors.type: user
paper_authors.user: shikunl
paper_authors._id: 65c306fe400d860a3eae7e9c
paper_authors.name: Xiaoyang Lyu
paper_authors._id: 65c306fe400d860a3eae7e9d
paper_authors.name: Marwan Taher
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-06T06:56:23.960Z
paper_authors._id: 633732015f0025d728b830fe
paper_authors.avatarUrl: /avatars/59a38b26604e08bc3e6509600d002bfe.svg
paper_authors.fullname: Marwan Taher
paper_authors.type: user
paper_authors.user: marwanPtr
paper_authors._id: 65c306fe400d860a3eae7e9e
paper_authors.name: Xiaojuan Qi
paper_authors._id: 65c306fe400d860a3eae7e9f
paper_authors.name: Andrew J. Davison
paper_title: EscherNet: A Generative Model for Scalable View Synthesis
paper_summary: We introduce EscherNet, a multi-view conditioned diffusion model for view
  synthesis. EscherNet learns implicit and generative 3D representations coupled
  with a specialised camera positional encoding, allowing precise and continuous
  relative control of the camera transformation between an arbitrary number of
  reference and target views. EscherNet offers exceptional generality,
  flexibility, and scalability in view synthesis -- it can generate more than 100
  consistent target views simultaneously on a single consumer-grade GPU, despite
  being trained with a fixed number of 3 reference views to 3 target views. As a
  result, EscherNet not only addresses zero-shot novel view synthesis, but also
  naturally unifies single- and multi-image 3D reconstruction, combining these
  diverse tasks into a single, cohesive framework. Our extensive experiments
  demonstrate that EscherNet achieves state-of-the-art performance in multiple
  benchmarks, even when compared to methods specifically tailored for each
  individual problem. This remarkable versatility opens up new directions for
  designing scalable neural architectures for 3D vision. Project page:
  https://kxhit.github.io/EscherNet.
paper_ai_keywords: multi-view conditioned diffusion model view synthesis implicit and generative 3D representations specialised camera positional encoding camera transformation reference views target views view synthesis zero-shot novel view synthesis single- and multi-image 3D reconstruction