title: LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias
summary: We propose the Large View Synthesis Model (LVSM), a novel transformer-based
  approach for scalable and generalizable novel view synthesis from sparse-view
  inputs. We introduce two architectures: (1) an encoder-decoder LVSM, which
  encodes input image tokens into a fixed number of 1D latent tokens, functioning
  as a fully learned scene representation, and decodes novel-view images from
  them; and (2) a decoder-only LVSM, which directly maps input images to
  novel-view outputs, completely eliminating intermediate scene representations.
  Both models bypass the 3D inductive biases used in previous methods -- from 3D
  representations (e.g., NeRF, 3DGS) to network designs (e.g., epipolar
  projections, plane sweeps) -- addressing novel view synthesis with a fully
  data-driven approach. While the encoder-decoder model offers faster inference
  due to its independent latent representation, the decoder-only LVSM achieves
  superior quality, scalability, and zero-shot generalization, outperforming
  previous state-of-the-art methods by 1.5 to 3.5 dB PSNR. Comprehensive
  evaluations across multiple datasets demonstrate that both LVSM variants
  achieve state-of-the-art novel view synthesis quality. Notably, our models
  surpass all previous methods even with reduced computational resources (1-2
  GPUs). Please see our website for more details:
  https://haian-jin.github.io/projects/LVSM/ .
paper_authors._id: 67191986f14003cd774531ca
paper_authors.name: Haian Jin
paper_authors._id: 67191986f14003cd774531cb
paper_authors.name: Hanwen Jiang
paper_authors._id: 67191986f14003cd774531cc
paper_authors.name: Hao Tan
paper_authors._id: 67191986f14003cd774531cd
paper_authors.name: Kai Zhang
paper_authors._id: 67191986f14003cd774531ce
paper_authors.name: Sai Bi
paper_authors._id: 67191986f14003cd774531cf
paper_authors.name: Tianyuan Zhang
paper_authors._id: 67191986f14003cd774531d0
paper_authors.name: Fujun Luan
paper_authors._id: 67191986f14003cd774531d1
paper_authors.name: Noah Snavely
paper_authors._id: 67191986f14003cd774531d2
paper_authors.name: Zexiang Xu
paper_title: LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias
paper_summary: We propose the Large View Synthesis Model (LVSM), a novel transformer-based
  approach for scalable and generalizable novel view synthesis from sparse-view
  inputs. We introduce two architectures: (1) an encoder-decoder LVSM, which
  encodes input image tokens into a fixed number of 1D latent tokens, functioning
  as a fully learned scene representation, and decodes novel-view images from
  them; and (2) a decoder-only LVSM, which directly maps input images to
  novel-view outputs, completely eliminating intermediate scene representations.
  Both models bypass the 3D inductive biases used in previous methods -- from 3D
  representations (e.g., NeRF, 3DGS) to network designs (e.g., epipolar
  projections, plane sweeps) -- addressing novel view synthesis with a fully
  data-driven approach. While the encoder-decoder model offers faster inference
  due to its independent latent representation, the decoder-only LVSM achieves
  superior quality, scalability, and zero-shot generalization, outperforming
  previous state-of-the-art methods by 1.5 to 3.5 dB PSNR. Comprehensive
  evaluations across multiple datasets demonstrate that both LVSM variants
  achieve state-of-the-art novel view synthesis quality. Notably, our models
  surpass all previous methods even with reduced computational resources (1-2
  GPUs). Please see our website for more details:
  https://haian-jin.github.io/projects/LVSM/ .
paper_ai_summary: A transformer-based synthesizer for novel views from sparse inputs using encoder-decoder and decoder-only architectures, bypassing 3D biases and achieving state-of-the-art quality with reduced resources.
paper_ai_keywords: transformer-based novel view synthesis sparse-view inputs encoder-decoder latent tokens scene representation decoder-only data-driven epipolar projections plane sweeps PSNR zero-shot generalization