title: L4GM: Large 4D Gaussian Reconstruction Model
summary: We present L4GM, the first 4D Large Reconstruction Model that produces
  animated objects from a single-view video input -- in a single feed-forward
  pass that takes only a second. Key to our success is a novel dataset of
  multiview videos containing curated, rendered animated objects from Objaverse.
  This dataset depicts 44K diverse objects with 110K animations rendered in 48
  viewpoints, resulting in 12M videos with a total of 300M frames. We keep our
  L4GM simple for scalability and build directly on top of LGM, a pretrained 3D
  Large Reconstruction Model that outputs 3D Gaussian ellipsoids from multiview
  image input. L4GM outputs a per-frame 3D Gaussian Splatting representation from
  video frames sampled at a low fps and then upsamples the representation to a
  higher fps to achieve temporal smoothness. We add temporal self-attention
  layers to the base LGM to help it learn consistency across time, and utilize a
  per-timestep multiview rendering loss to train the model. The representation is
  upsampled to a higher framerate by training an interpolation model which
  produces intermediate 3D Gaussian representations. We showcase that L4GM that
  is only trained on synthetic data generalizes extremely well on in-the-wild
  videos, producing high quality animated 3D assets.
paper_authors._id: 6670fa1cc718469ee73cac89
paper_authors.name: Jiawei Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T21:36:20.287Z
paper_authors._id: 64bf93139a69e8da48146cce
paper_authors.avatarUrl: /avatars/0be0719e77312b12b4b3520f98ed808b.svg
paper_authors.fullname: Jiawei Ren
paper_authors.type: user
paper_authors.user: Jiawei1992
paper_authors._id: 6670fa1cc718469ee73cac8a
paper_authors.name: Kevin Xie
paper_authors._id: 6670fa1cc718469ee73cac8b
paper_authors.name: Ashkan Mirzaei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T21:36:37.220Z
paper_authors._id: 634db15dd00bb5d92c3bd94f
paper_authors.avatarUrl: /avatars/9a6a1231bc5205911272d83527593f1a.svg
paper_authors.fullname: Ashkan Mirzaei
paper_authors.type: user
paper_authors.user: ashmrz
paper_authors._id: 6670fa1cc718469ee73cac8c
paper_authors.name: Hanxue Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T21:36:46.070Z
paper_authors._id: 64499e6e6fa5e1ba98a309e2
paper_authors.avatarUrl: /avatars/17ac2b15e42ea927de775d183d57359d.svg
paper_authors.fullname: liang
paper_authors.type: user
paper_authors.user: hanxue
paper_authors._id: 6670fa1cc718469ee73cac8d
paper_authors.name: Xiaohui Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-16T19:47:22.615Z
paper_authors._id: 63363cb8a8048332fdc268cd
paper_authors.avatarUrl: /avatars/a9bf6ce3394a8b76eade1a7517b37fa1.svg
paper_authors.fullname: xiaohui zeng
paper_authors.type: user
paper_authors.user: xiaohui2022
paper_authors._id: 6670fa1cc718469ee73cac8e
paper_authors.name: Karsten Kreis
paper_authors._id: 6670fa1cc718469ee73cac8f
paper_authors.name: Ziwei Liu
paper_authors._id: 6670fa1cc718469ee73cac90
paper_authors.name: Antonio Torralba
paper_authors._id: 6670fa1cc718469ee73cac91
paper_authors.name: Sanja Fidler
paper_authors._id: 6670fa1cc718469ee73cac92
paper_authors.name: Seung Wook Kim
paper_authors._id: 6670fa1cc718469ee73cac93
paper_authors.name: Huan Ling
paper_title: L4GM: Large 4D Gaussian Reconstruction Model
paper_summary: We present L4GM, the first 4D Large Reconstruction Model that produces
  animated objects from a single-view video input -- in a single feed-forward
  pass that takes only a second. Key to our success is a novel dataset of
  multiview videos containing curated, rendered animated objects from Objaverse.
  This dataset depicts 44K diverse objects with 110K animations rendered in 48
  viewpoints, resulting in 12M videos with a total of 300M frames. We keep our
  L4GM simple for scalability and build directly on top of LGM, a pretrained 3D
  Large Reconstruction Model that outputs 3D Gaussian ellipsoids from multiview
  image input. L4GM outputs a per-frame 3D Gaussian Splatting representation from
  video frames sampled at a low fps and then upsamples the representation to a
  higher fps to achieve temporal smoothness. We add temporal self-attention
  layers to the base LGM to help it learn consistency across time, and utilize a
  per-timestep multiview rendering loss to train the model. The representation is
  upsampled to a higher framerate by training an interpolation model which
  produces intermediate 3D Gaussian representations. We showcase that L4GM that
  is only trained on synthetic data generalizes extremely well on in-the-wild
  videos, producing high quality animated 3D assets.
paper_ai_keywords: Large Reconstruction Model animated objects single-feed-forward pass multiview videos curated rendered animated objects Objaverse Gaussian ellipsoids Gaussian Splatting temporal smoothness temporal self-attention layers per-timestep multiview rendering loss interpolation model 3D assets