title: HyperHuman: Hyper-Realistic Human Generation with Latent Structural
    Diffusion
summary: Despite significant advances in large-scale text-to-image models, achieving
  hyper-realistic human image generation remains a desirable yet unsolved task.
  Existing models like Stable Diffusion and DALL-E 2 tend to generate human
  images with incoherent parts or unnatural poses. To tackle these challenges,
  our key insight is that human image is inherently structural over multiple
  granularities, from the coarse-level body skeleton to fine-grained spatial
  geometry. Therefore, capturing such correlations between the explicit
  appearance and latent structure in one model is essential to generate coherent
  and natural human images. To this end, we propose a unified framework,
  HyperHuman, that generates in-the-wild human images of high realism and diverse
  layouts. Specifically, 1) we first build a large-scale human-centric dataset,
  named HumanVerse, which consists of 340M images with comprehensive annotations
  like human pose, depth, and surface normal. 2) Next, we propose a Latent
  Structural Diffusion Model that simultaneously denoises the depth and surface
  normal along with the synthesized RGB image. Our model enforces the joint
  learning of image appearance, spatial relationship, and geometry in a unified
  network, where each branch in the model complements to each other with both
  structural awareness and textural richness. 3) Finally, to further boost the
  visual quality, we propose a Structure-Guided Refiner to compose the predicted
  conditions for more detailed generation of higher resolution. Extensive
  experiments demonstrate that our framework yields the state-of-the-art
  performance, generating hyper-realistic human images under diverse scenarios.
  Project Page: https://snap-research.github.io/HyperHuman/
paper_authors._id: 65294d8a0b44378dcc6862a9
paper_authors.name: Xian Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-13T15:50:41.663Z
paper_authors._id: 6423f10a3fa01ecba6fca861
paper_authors.avatarUrl: /avatars/e4b4c7dddc5cf962ade45d8bdfa10b1a.svg
paper_authors.fullname: Xian Liu
paper_authors.type: user
paper_authors.user: alvinliu0
paper_authors._id: 65294d8a0b44378dcc6862aa
paper_authors.name: Jian Ren
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-10-26T03:40:22.879Z
paper_authors._id: 61f19829233c91cbd2f79e70
paper_authors.avatarUrl: /avatars/a0735a94542b4f7cda5aed8bc4be0538.svg
paper_authors.fullname: Jian Ren
paper_authors.type: user
paper_authors.user: alanspike
paper_authors._id: 65294d8a0b44378dcc6862ab
paper_authors.name: Aliaksandr Siarohin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-13T15:50:49.640Z
paper_authors._id: 64276311eb9a0ed86180715b
paper_authors.avatarUrl: /avatars/76f933cd549f10e5e2db379de235d304.svg
paper_authors.fullname: Aliaksandr Siarohin
paper_authors.type: user
paper_authors.user: aliaksandr-siarohin
paper_authors._id: 65294d8a0b44378dcc6862ac
paper_authors.name: Ivan Skorokhodov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-13T15:50:58.308Z
paper_authors._id: 63610db13c7147fae7de88e3
paper_authors.avatarUrl: /avatars/d7e97a16cfee39e1e50d7a5b747876f1.svg
paper_authors.fullname: Ivan Skorokhodov
paper_authors.type: user
paper_authors.user: universome
paper_authors._id: 65294d8a0b44378dcc6862ad
paper_authors.name: Yanyu Li
paper_authors._id: 65294d8a0b44378dcc6862ae
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-13T15:51:16.567Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_authors._id: 65294d8a0b44378dcc6862af
paper_authors.name: Xihui Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-03T18:30:36.148Z
paper_authors._id: 65d5ec74cd05bc1eaa125040
paper_authors.avatarUrl: /avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
paper_authors.fullname: Xihui Liu
paper_authors.type: user
paper_authors.user: XihuiLiu
paper_authors._id: 65294d8a0b44378dcc6862b0
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-13T15:51:36.113Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_authors._id: 65294d8a0b44378dcc6862b1
paper_authors.name: Sergey Tulyakov
paper_title: HyperHuman: Hyper-Realistic Human Generation with Latent Structural
    Diffusion
paper_summary: Despite significant advances in large-scale text-to-image models, achieving
  hyper-realistic human image generation remains a desirable yet unsolved task.
  Existing models like Stable Diffusion and DALL-E 2 tend to generate human
  images with incoherent parts or unnatural poses. To tackle these challenges,
  our key insight is that human image is inherently structural over multiple
  granularities, from the coarse-level body skeleton to fine-grained spatial
  geometry. Therefore, capturing such correlations between the explicit
  appearance and latent structure in one model is essential to generate coherent
  and natural human images. To this end, we propose a unified framework,
  HyperHuman, that generates in-the-wild human images of high realism and diverse
  layouts. Specifically, 1) we first build a large-scale human-centric dataset,
  named HumanVerse, which consists of 340M images with comprehensive annotations
  like human pose, depth, and surface normal. 2) Next, we propose a Latent
  Structural Diffusion Model that simultaneously denoises the depth and surface
  normal along with the synthesized RGB image. Our model enforces the joint
  learning of image appearance, spatial relationship, and geometry in a unified
  network, where each branch in the model complements to each other with both
  structural awareness and textural richness. 3) Finally, to further boost the
  visual quality, we propose a Structure-Guided Refiner to compose the predicted
  conditions for more detailed generation of higher resolution. Extensive
  experiments demonstrate that our framework yields the state-of-the-art
  performance, generating hyper-realistic human images under diverse scenarios.
  Project Page: https://snap-research.github.io/HyperHuman/
paper_ai_keywords: Latent Structural Diffusion Model denoises depth surface normal synthesized RGB image joint learning image appearance spatial relationship geometry unified network Structure-Guided Refiner hyper-realistic human images diverse scenarios HumanVerse human pose surface normal