title: GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting
summary: We propose GS-LRM, a scalable large reconstruction model that can predict
  high-quality 3D Gaussian primitives from 2-4 posed sparse images in 0.23
  seconds on single A100 GPU. Our model features a very simple transformer-based
  architecture; we patchify input posed images, pass the concatenated multi-view
  image tokens through a sequence of transformer blocks, and decode final
  per-pixel Gaussian parameters directly from these tokens for differentiable
  rendering. In contrast to previous LRMs that can only reconstruct objects, by
  predicting per-pixel Gaussians, GS-LRM naturally handles scenes with large
  variations in scale and complexity. We show that our model can work on both
  object and scene captures by training it on Objaverse and RealEstate10K
  respectively. In both scenarios, the models outperform state-of-the-art
  baselines by a wide margin. We also demonstrate applications of our model in
  downstream 3D generation tasks. Our project webpage is available at:
  https://sai-bi.github.io/project/gs-lrm/ .
paper_authors._id: 6631a053472898cdc32742c7
paper_authors.name: Kai Zhang
paper_authors._id: 6631a053472898cdc32742c8
paper_authors.name: Sai Bi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T08:25:47.138Z
paper_authors._id: 66303560597dc7c535168f07
paper_authors.avatarUrl: /avatars/a69d52a4a198c6c8451c1d137031bf07.svg
paper_authors.fullname: Sai Bi
paper_authors.type: user
paper_authors.user: saibi
paper_authors._id: 6631a053472898cdc32742c9
paper_authors.name: Hao Tan
paper_authors._id: 6631a053472898cdc32742ca
paper_authors.name: Yuanbo Xiangli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T08:25:26.072Z
paper_authors._id: 651edd0faa888068acb5d680
paper_authors.avatarUrl: /avatars/d5e03e8bc1d22bc67a3e684b52dc5598.svg
paper_authors.fullname: Yuanbo Xiangli
paper_authors.type: user
paper_authors.user: amberkk
paper_authors._id: 6631a053472898cdc32742cb
paper_authors.name: Nanxuan Zhao
paper_authors._id: 6631a053472898cdc32742cc
paper_authors.name: Kalyan Sunkavalli
paper_authors._id: 6631a053472898cdc32742cd
paper_authors.name: Zexiang Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T08:25:08.627Z
paper_authors._id: 65565b1b8616d8d740abb976
paper_authors.avatarUrl: /avatars/c245b19ed05292bf1bf4a9d8cf2a409c.svg
paper_authors.fullname: Zexiang Xu
paper_authors.type: user
paper_authors.user: zexiangxu
paper_title: GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting
paper_summary: We propose GS-LRM, a scalable large reconstruction model that can predict
  high-quality 3D Gaussian primitives from 2-4 posed sparse images in 0.23
  seconds on single A100 GPU. Our model features a very simple transformer-based
  architecture; we patchify input posed images, pass the concatenated multi-view
  image tokens through a sequence of transformer blocks, and decode final
  per-pixel Gaussian parameters directly from these tokens for differentiable
  rendering. In contrast to previous LRMs that can only reconstruct objects, by
  predicting per-pixel Gaussians, GS-LRM naturally handles scenes with large
  variations in scale and complexity. We show that our model can work on both
  object and scene captures by training it on Objaverse and RealEstate10K
  respectively. In both scenarios, the models outperform state-of-the-art
  baselines by a wide margin. We also demonstrate applications of our model in
  downstream 3D generation tasks. Our project webpage is available at:
  https://sai-bi.github.io/project/gs-lrm/ .
paper_ai_keywords: transformer-based architecture patchify image tokens sequence of transformer blocks per-pixel Gaussian differentiable rendering scale and complexity scene-level downstream 3D generation tasks