title: 3D Reconstruction with Spatial Memory
summary: We present Spann3R, a novel approach for dense 3D reconstruction from ordered
  or unordered image collections. Built on the DUSt3R paradigm, Spann3R uses a
  transformer-based architecture to directly regress pointmaps from images
  without any prior knowledge of the scene or camera parameters. Unlike DUSt3R,
  which predicts per image-pair pointmaps each expressed in its local coordinate
  frame, Spann3R can predict per-image pointmaps expressed in a global coordinate
  system, thus eliminating the need for optimization-based global alignment. The
  key idea of Spann3R is to manage an external spatial memory that learns to keep
  track of all previous relevant 3D information. Spann3R then queries this
  spatial memory to predict the 3D structure of the next frame in a global
  coordinate system. Taking advantage of DUSt3R's pre-trained weights, and
  further fine-tuning on a subset of datasets, Spann3R shows competitive
  performance and generalization ability on various unseen datasets and can
  process ordered image collections in real time. Project page:
  https://hengyiwang.github.io/projects/spanner
paper_authors._id: 66d1246f332325ec078e997e
paper_authors.name: Hengyi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-30T12:46:10.208Z
paper_authors._id: 65f475985528c0b13a7bdcce
paper_authors.avatarUrl: /avatars/5cac4d77028fe64886396aeff9db3974.svg
paper_authors.fullname: Hengyi Wang
paper_authors.type: user
paper_authors.user: HengyiWang
paper_authors._id: 66d1246f332325ec078e997f
paper_authors.name: Lourdes Agapito
paper_title: 3D Reconstruction with Spatial Memory
paper_summary: We present Spann3R, a novel approach for dense 3D reconstruction from ordered
  or unordered image collections. Built on the DUSt3R paradigm, Spann3R uses a
  transformer-based architecture to directly regress pointmaps from images
  without any prior knowledge of the scene or camera parameters. Unlike DUSt3R,
  which predicts per image-pair pointmaps each expressed in its local coordinate
  frame, Spann3R can predict per-image pointmaps expressed in a global coordinate
  system, thus eliminating the need for optimization-based global alignment. The
  key idea of Spann3R is to manage an external spatial memory that learns to keep
  track of all previous relevant 3D information. Spann3R then queries this
  spatial memory to predict the 3D structure of the next frame in a global
  coordinate system. Taking advantage of DUSt3R's pre-trained weights, and
  further fine-tuning on a subset of datasets, Spann3R shows competitive
  performance and generalization ability on various unseen datasets and can
  process ordered image collections in real time. Project page:
  https://hengyiwang.github.io/projects/spanner
paper_ai_summary: Spann3R is a transformer-based method for dense 3D reconstruction from image collections, using an external spatial memory to predict global pointmaps and demonstrating real-time performance and generalization.
paper_ai_keywords: transformer-based architecture pointmaps DUSt3R paradigm global coordinate system spatial memory pre-trained weights