title: Sapiens: Foundation for Human Vision Models
summary: We present Sapiens, a family of models for four fundamental human-centric
  vision tasks - 2D pose estimation, body-part segmentation, depth estimation,
  and surface normal prediction. Our models natively support 1K high-resolution
  inference and are extremely easy to adapt for individual tasks by simply
  fine-tuning models pretrained on over 300 million in-the-wild human images. We
  observe that, given the same computational budget, self-supervised pretraining
  on a curated dataset of human images significantly boosts the performance for a
  diverse set of human-centric tasks. The resulting models exhibit remarkable
  generalization to in-the-wild data, even when labeled data is scarce or
  entirely synthetic. Our simple model design also brings scalability - model
  performance across tasks improves as we scale the number of parameters from 0.3
  to 2 billion. Sapiens consistently surpasses existing baselines across various
  human-centric benchmarks. We achieve significant improvements over the prior
  state-of-the-art on Humans-5K (pose) by 7.6 mAP, Humans-2K (part-seg) by 17.1
  mIoU, Hi4D (depth) by 22.4% relative RMSE, and THuman2 (normal) by 53.5%
  relative angular error.
paper_authors._id: 66c7f3b3ab23d3d7df974707
paper_authors.name: Rawal Khirodkar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-30T13:10:09.902Z
paper_authors._id: 6413702057d68cf865da8314
paper_authors.avatarUrl: /avatars/332e72602a4c6ac0a997b295bb9b2f7d.svg
paper_authors.fullname: Rawal Khirodkar
paper_authors.type: user
paper_authors.user: rawalkhirodkar
paper_authors._id: 66c7f3b3ab23d3d7df974708
paper_authors.name: Timur Bagautdinov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-30T13:10:16.104Z
paper_authors._id: 63be0fe19a15a3e9419d3d7d
paper_authors.avatarUrl: /avatars/145760938083ec031e30d20d321d9185.svg
paper_authors.fullname: Timur Bagautdinov
paper_authors.type: user
paper_authors.user: psycharo
paper_authors._id: 66c7f3b3ab23d3d7df974709
paper_authors.name: Julieta Martinez
paper_authors._id: 66c7f3b3ab23d3d7df97470a
paper_authors.name: Su Zhaoen
paper_authors._id: 66c7f3b3ab23d3d7df97470b
paper_authors.name: Austin James
paper_authors._id: 66c7f3b3ab23d3d7df97470c
paper_authors.name: Peter Selednik
paper_authors._id: 66c7f3b3ab23d3d7df97470d
paper_authors.name: Stuart Anderson
paper_authors._id: 66c7f3b3ab23d3d7df97470e
paper_authors.name: Shunsuke Saito
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-26T07:15:45.623Z
paper_authors._id: 630688edd37ce67e0e50e548
paper_authors.avatarUrl: /avatars/1681e465d7649f67f94e1b69d236cb1e.svg
paper_authors.fullname: Shunsuke Saito
paper_authors.type: user
paper_authors.user: psyth
paper_title: Sapiens: Foundation for Human Vision Models
paper_summary: We present Sapiens, a family of models for four fundamental human-centric
  vision tasks - 2D pose estimation, body-part segmentation, depth estimation,
  and surface normal prediction. Our models natively support 1K high-resolution
  inference and are extremely easy to adapt for individual tasks by simply
  fine-tuning models pretrained on over 300 million in-the-wild human images. We
  observe that, given the same computational budget, self-supervised pretraining
  on a curated dataset of human images significantly boosts the performance for a
  diverse set of human-centric tasks. The resulting models exhibit remarkable
  generalization to in-the-wild data, even when labeled data is scarce or
  entirely synthetic. Our simple model design also brings scalability - model
  performance across tasks improves as we scale the number of parameters from 0.3
  to 2 billion. Sapiens consistently surpasses existing baselines across various
  human-centric benchmarks. We achieve significant improvements over the prior
  state-of-the-art on Humans-5K (pose) by 7.6 mAP, Humans-2K (part-seg) by 17.1
  mIoU, Hi4D (depth) by 22.4% relative RMSE, and THuman2 (normal) by 53.5%
  relative angular error.
paper_ai_summary: Sapiens, a family of vision models for human-centric tasks, achieves superior performance with self-supervised pretraining and can be easily fine-tuned for 2D pose estimation, body-part segmentation, depth estimation, and surface normal prediction.
paper_ai_keywords: self-supervised pretraining model adaptation fine-tuning high-resolution inference generalization parameter scaling human-centric tasks 2D pose estimation body-part segmentation depth estimation surface normal prediction Humans-5K Humans-2K Hi4D THuman2