title: Continuous 3D Perception Model with Persistent State
summary: We present a unified framework capable of solving a broad range of 3D tasks.
  Our approach features a stateful recurrent model that continuously updates its
  state representation with each new observation. Given a stream of images, this
  evolving state can be used to generate metric-scale pointmaps (per-pixel 3D
  points) for each new input in an online fashion. These pointmaps reside within
  a common coordinate system, and can be accumulated into a coherent, dense scene
  reconstruction that updates as new images arrive. Our model, called CUT3R
  (Continuous Updating Transformer for 3D Reconstruction), captures rich priors
  of real-world scenes: not only can it predict accurate pointmaps from image
  observations, but it can also infer unseen regions of the scene by probing at
  virtual, unobserved views. Our method is simple yet highly flexible, naturally
  accepting varying lengths of images that may be either video streams or
  unordered photo collections, containing both static and dynamic content. We
  evaluate our method on various 3D/4D tasks and demonstrate competitive or
  state-of-the-art performance in each. Project Page: https://cut3r.github.io/
paper_authors._id: 67a1596a167bea74d5057f25
paper_authors.name: Qianqian Wang
paper_authors._id: 67a1596a167bea74d5057f26
paper_authors.name: Yifei Zhang
paper_authors._id: 67a1596a167bea74d5057f27
paper_authors.name: Aleksander Holynski
paper_authors._id: 67a1596a167bea74d5057f28
paper_authors.name: Alexei A. Efros
paper_authors._id: 67a1596a167bea74d5057f29
paper_authors.name: Angjoo Kanazawa
paper_title: Continuous 3D Perception Model with Persistent State
paper_summary: We present a unified framework capable of solving a broad range of 3D tasks.
  Our approach features a stateful recurrent model that continuously updates its
  state representation with each new observation. Given a stream of images, this
  evolving state can be used to generate metric-scale pointmaps (per-pixel 3D
  points) for each new input in an online fashion. These pointmaps reside within
  a common coordinate system, and can be accumulated into a coherent, dense scene
  reconstruction that updates as new images arrive. Our model, called CUT3R
  (Continuous Updating Transformer for 3D Reconstruction), captures rich priors
  of real-world scenes: not only can it predict accurate pointmaps from image
  observations, but it can also infer unseen regions of the scene by probing at
  virtual, unobserved views. Our method is simple yet highly flexible, naturally
  accepting varying lengths of images that may be either video streams or
  unordered photo collections, containing both static and dynamic content. We
  evaluate our method on various 3D/4D tasks and demonstrate competitive or
  state-of-the-art performance in each. Project Page: https://cut3r.github.io/
paper_ai_summary: A recurrent transformer model generates metric-scale pointmaps in an online fashion to create a continuously updating 3D reconstruction from image streams.
paper_ai_keywords: recurrent model state representation pointmaps 3D reconstruction continuous updating virtual views state-of-the-art