title: GST: Precise 3D Human Body from a Single Image with Gaussian Splatting
    Transformers
summary: Reconstructing realistic 3D human models from monocular images has
  significant applications in creative industries, human-computer interfaces, and
  healthcare. We base our work on 3D Gaussian Splatting (3DGS), a scene
  representation composed of a mixture of Gaussians. Predicting such mixtures for
  a human from a single input image is challenging, as it is a non-uniform
  density (with a many-to-one relationship with input pixels) with strict
  physical constraints. At the same time, it needs to be flexible to accommodate
  a variety of clothes and poses. Our key observation is that the vertices of
  standardized human meshes (such as SMPL) can provide an adequate density and
  approximate initial position for Gaussians. We can then train a transformer
  model to jointly predict comparatively small adjustments to these positions, as
  well as the other Gaussians' attributes and the SMPL parameters. We show
  empirically that this combination (using only multi-view supervision) can
  achieve fast inference of 3D human models from a single image without test-time
  optimization, expensive diffusion models, or 3D points supervision. We also
  show that it can improve 3D pose estimation by better fitting human models that
  account for clothes and other variations. The code is available on the project
  website https://abdullahamdi.com/gst/ .
paper_authors._id: 66decbb4ac5e6632f9067cba
paper_authors.name: Lorenza Prospero
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-10T08:43:47.840Z
paper_authors._id: 66e0061d47e4afabbb0a35a0
paper_authors.avatarUrl: /avatars/3c3fbcf4aacf6341f55ce8211c02df70.svg
paper_authors.fullname: Lorenza Prospero
paper_authors.type: user
paper_authors.user: prosperolo
paper_authors._id: 66decbb4ac5e6632f9067cbb
paper_authors.name: Abdullah Hamdi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-09T10:57:55.944Z
paper_authors._id: 62fe3442e9061c0170d06e0b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1660827186084-62fe3442e9061c0170d06e0b.png
paper_authors.fullname: Abdullah Hamdi
paper_authors.type: user
paper_authors.user: ajhamdi
paper_authors._id: 66decbb4ac5e6632f9067cbc
paper_authors.name: Joao F. Henriques
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T08:07:33.879Z
paper_authors._id: 64bd4213891751ce9b17f4c3
paper_authors.avatarUrl: /avatars/3a20ba60382a2b6fe5a576d4274ffc24.svg
paper_authors.fullname: Joao Henriques
paper_authors.type: user
paper_authors.user: JMHenry
paper_authors._id: 66decbb4ac5e6632f9067cbd
paper_authors.name: Christian Rupprecht
paper_title: GST: Precise 3D Human Body from a Single Image with Gaussian Splatting
    Transformers
paper_summary: Reconstructing realistic 3D human models from monocular images has
  significant applications in creative industries, human-computer interfaces, and
  healthcare. We base our work on 3D Gaussian Splatting (3DGS), a scene
  representation composed of a mixture of Gaussians. Predicting such mixtures for
  a human from a single input image is challenging, as it is a non-uniform
  density (with a many-to-one relationship with input pixels) with strict
  physical constraints. At the same time, it needs to be flexible to accommodate
  a variety of clothes and poses. Our key observation is that the vertices of
  standardized human meshes (such as SMPL) can provide an adequate density and
  approximate initial position for Gaussians. We can then train a transformer
  model to jointly predict comparatively small adjustments to these positions, as
  well as the other Gaussians' attributes and the SMPL parameters. We show
  empirically that this combination (using only multi-view supervision) can
  achieve fast inference of 3D human models from a single image without test-time
  optimization, expensive diffusion models, or 3D points supervision. We also
  show that it can improve 3D pose estimation by better fitting human models that
  account for clothes and other variations. The code is available on the project
  website https://abdullahamdi.com/gst/ .
paper_ai_summary: A transformer model predicts adjustments to Gaussian-splatted 3D human models from a single image, improving 3D pose estimation without test-time optimization or 3D point supervision.
paper_ai_keywords: 3D Gaussian Splatting Gaussians SMPL transformer model multi-view supervision 3D human models 3D pose estimation