title: BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation
summary: The systematic evaluation and understanding of computer vision models under
  varying conditions require large amounts of data with comprehensive and
  customized labels, which real-world vision datasets rarely satisfy. While
  current synthetic data generators offer a promising alternative, particularly
  for embodied AI tasks, they often fall short for computer vision tasks due to
  low asset and rendering quality, limited diversity, and unrealistic physical
  properties. We introduce the BEHAVIOR Vision Suite (BVS), a set of tools and
  assets to generate fully customized synthetic data for systematic evaluation of
  computer vision models, based on the newly developed embodied AI benchmark,
  BEHAVIOR-1K. BVS supports a large number of adjustable parameters at the scene
  level (e.g., lighting, object placement), the object level (e.g., joint
  configuration, attributes such as "filled" and "folded"), and the camera level
  (e.g., field of view, focal length). Researchers can arbitrarily vary these
  parameters during data generation to perform controlled experiments. We
  showcase three example application scenarios: systematically evaluating the
  robustness of models across different continuous axes of domain shift,
  evaluating scene understanding models on the same set of images, and training
  and evaluating simulation-to-real transfer for a novel vision task: unary and
  binary state prediction. Project website:
  https://behavior-vision-suite.github.io/
paper_authors._id: 66456bea22a71d1eb4ab995f
paper_authors.name: Yunhao Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-16T12:53:56.429Z
paper_authors._id: 64370de87cbe1e781fc5c7a2
paper_authors.avatarUrl: /avatars/98101739726ed797958e14f3d666933b.svg
paper_authors.fullname: Yunhao Ge
paper_authors.type: user
paper_authors.user: andyge
paper_authors._id: 66456bea22a71d1eb4ab9960
paper_authors.name: Yihe Tang
paper_authors._id: 66456bea22a71d1eb4ab9961
paper_authors.name: Jiashu Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-16T12:54:31.096Z
paper_authors._id: 62da681ff014388f9086a578
paper_authors.avatarUrl: /avatars/a33b0c94c907c8a1179332a4bb1a0165.svg
paper_authors.fullname: XUJIASHU
paper_authors.type: user
paper_authors.user: JSX
paper_authors._id: 66456bea22a71d1eb4ab9962
paper_authors.name: Cem Gokmen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-16T12:54:37.236Z
paper_authors._id: 6487a3ccaa6b952c784eab16
paper_authors.avatarUrl: /avatars/6d543110f169fbc0142b8485e2919f5f.svg
paper_authors.fullname: Cem Gokmen
paper_authors.type: user
paper_authors.user: cgokmen
paper_authors._id: 66456bea22a71d1eb4ab9963
paper_authors.name: Chengshu Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-22T07:15:28.719Z
paper_authors._id: 664d3a3079f09472c1193e06
paper_authors.avatarUrl: /avatars/3cf4591f36230137bb059e7f2de8bbc3.svg
paper_authors.fullname: ChengshuLi
paper_authors.type: user
paper_authors.user: chengshu
paper_authors._id: 66456bea22a71d1eb4ab9964
paper_authors.name: Wensi Ai
paper_authors._id: 66456bea22a71d1eb4ab9965
paper_authors.name: Benjamin Jose Martinez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-16T12:55:01.062Z
paper_authors._id: 63c9d40cbedad7e2bf1a7b79
paper_authors.avatarUrl: /avatars/b0a631685cac7aec1252918226e753f4.svg
paper_authors.fullname: Benjamin Martinez
paper_authors.type: user
paper_authors.user: benjaminmartinez
paper_authors._id: 66456bea22a71d1eb4ab9966
paper_authors.name: Arman Aydin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-16T12:55:14.973Z
paper_authors._id: 62241a3dbe89d35b37211aa0
paper_authors.avatarUrl: /avatars/0a77e25cd3f7504c672adf59af61783c.svg
paper_authors.fullname: Arman Aydin
paper_authors.type: user
paper_authors.user: goldenJester
paper_authors._id: 66456bea22a71d1eb4ab9967
paper_authors.name: Mona Anvari
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-16T12:55:21.833Z
paper_authors._id: 63af3f505fe9db73f67b1c4d
paper_authors.avatarUrl: /avatars/430ef0638353698b4a08bf6c4aa43f5e.svg
paper_authors.fullname: Mona Anvari
paper_authors.type: user
paper_authors.user: monaavr
paper_authors._id: 66456bea22a71d1eb4ab9968
paper_authors.name: Ayush K Chakravarthy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-16T12:55:31.341Z
paper_authors._id: 624f9e3d07bd004fb855f5e9
paper_authors.avatarUrl: /avatars/86a349cd4053bc0317e27e75a51c69fa.svg
paper_authors.fullname: Ayush Chakravarthy
paper_authors.type: user
paper_authors.user: ayushchakravarthy
paper_authors._id: 66456bea22a71d1eb4ab9969
paper_authors.name: Hong-Xing Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-16T13:24:41.700Z
paper_authors._id: 64ca8dbdbb5d195b99febe28
paper_authors.avatarUrl: /avatars/e6eb3ba76a5ced56c8c41506d707d05b.svg
paper_authors.fullname: Koven Yu
paper_authors.type: user
paper_authors.user: KovenYu
paper_authors._id: 66456bea22a71d1eb4ab996a
paper_authors.name: Josiah Wong
paper_authors._id: 66456bea22a71d1eb4ab996b
paper_authors.name: Sanjana Srivastava
paper_authors._id: 66456bea22a71d1eb4ab996c
paper_authors.name: Sharon Lee
paper_authors._id: 66456bea22a71d1eb4ab996d
paper_authors.name: Shengxin Zha
paper_authors._id: 66456bea22a71d1eb4ab996e
paper_authors.name: Laurent Itti
paper_authors._id: 66456bea22a71d1eb4ab996f
paper_authors.name: Yunzhu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-16T13:19:13.938Z
paper_authors._id: 64a46562c641afb468a3dc04
paper_authors.avatarUrl: /avatars/bc54e5c13f657251b05bf2725086e56d.svg
paper_authors.fullname: Yunzhu Li
paper_authors.type: user
paper_authors.user: yunzhuli
paper_authors._id: 66456bea22a71d1eb4ab9970
paper_authors.name: Roberto Martín-Martín
paper_authors._id: 66456bea22a71d1eb4ab9971
paper_authors.name: Miao Liu
paper_authors._id: 66456bea22a71d1eb4ab9972
paper_authors.name: Pengchuan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-16T13:22:10.518Z
paper_authors._id: 640056f5261cfa61f39b4c35
paper_authors.avatarUrl: /avatars/25aa5dee42caa69327d158a669f6bf3a.svg
paper_authors.fullname: Pengchuan Zhang
paper_authors.type: user
paper_authors.user: pzzhang
paper_authors._id: 66456bea22a71d1eb4ab9973
paper_authors.name: Ruohan Zhang
paper_authors._id: 66456bea22a71d1eb4ab9974
paper_authors.name: Li Fei-Fei
paper_authors._id: 66456bea22a71d1eb4ab9975
paper_authors.name: Jiajun Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-16T13:21:34.303Z
paper_authors._id: 647b7b8345616c3de3627863
paper_authors.avatarUrl: /avatars/8196364bffed71086ba9a0a4a62027d3.svg
paper_authors.fullname: Jiajun Wu
paper_authors.type: user
paper_authors.user: jiajunwu
paper_title: BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation
paper_summary: The systematic evaluation and understanding of computer vision models under
  varying conditions require large amounts of data with comprehensive and
  customized labels, which real-world vision datasets rarely satisfy. While
  current synthetic data generators offer a promising alternative, particularly
  for embodied AI tasks, they often fall short for computer vision tasks due to
  low asset and rendering quality, limited diversity, and unrealistic physical
  properties. We introduce the BEHAVIOR Vision Suite (BVS), a set of tools and
  assets to generate fully customized synthetic data for systematic evaluation of
  computer vision models, based on the newly developed embodied AI benchmark,
  BEHAVIOR-1K. BVS supports a large number of adjustable parameters at the scene
  level (e.g., lighting, object placement), the object level (e.g., joint
  configuration, attributes such as "filled" and "folded"), and the camera level
  (e.g., field of view, focal length). Researchers can arbitrarily vary these
  parameters during data generation to perform controlled experiments. We
  showcase three example application scenarios: systematically evaluating the
  robustness of models across different continuous axes of domain shift,
  evaluating scene understanding models on the same set of images, and training
  and evaluating simulation-to-real transfer for a novel vision task: unary and
  binary state prediction. Project website:
  https://behavior-vision-suite.github.io/
paper_ai_keywords: BEHAVIOR Vision Suite (BVS) BEHAVIOR-1K domain shift scene understanding models simulation-to-real transfer unary and binary state prediction