title: FACET: Fairness in Computer Vision Evaluation Benchmark
summary: Computer vision models have known performance disparities across attributes
  such as gender and skin tone. This means during tasks such as classification
  and detection, model performance differs for certain classes based on the
  demographics of the people in the image. These disparities have been shown to
  exist, but until now there has not been a unified approach to measure these
  differences for common use-cases of computer vision models. We present a new
  benchmark named FACET (FAirness in Computer Vision EvaluaTion), a large,
  publicly available evaluation set of 32k images for some of the most common
  vision tasks - image classification, object detection and segmentation. For
  every image in FACET, we hired expert reviewers to manually annotate
  person-related attributes such as perceived skin tone and hair type, manually
  draw bounding boxes and label fine-grained person-related classes such as disk
  jockey or guitarist. In addition, we use FACET to benchmark state-of-the-art
  vision models and present a deeper understanding of potential performance
  disparities and challenges across sensitive demographic attributes. With the
  exhaustive annotations collected, we probe models using single demographics
  attributes as well as multiple attributes using an intersectional approach
  (e.g. hair color and perceived skin tone). Our results show that
  classification, detection, segmentation, and visual grounding models exhibit
  performance disparities across demographic attributes and intersections of
  attributes. These harms suggest that not all people represented in datasets
  receive fair and equitable treatment in these vision tasks. We hope current and
  future results using our benchmark will contribute to fairer, more robust
  vision models. FACET is available publicly at https://facet.metademolab.com/
paper_authors._id: 64f531ba4937e64ca402e3ee
paper_authors.name: Laura Gustafson
paper_authors._id: 64f531ba4937e64ca402e3ef
paper_authors.name: Chloe Rolland
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-04T08:19:34.205Z
paper_authors._id: 6463d03f38083255f6754152
paper_authors.avatarUrl: /avatars/a158e36e0a24ece399b8ec26a2084444.svg
paper_authors.fullname: Chloe Rolland
paper_authors.type: user
paper_authors.user: crolland
paper_authors._id: 64f531ba4937e64ca402e3f0
paper_authors.name: Nikhila Ravi
paper_authors._id: 64f531ba4937e64ca402e3f1
paper_authors.name: Quentin Duval
paper_authors._id: 64f531ba4937e64ca402e3f2
paper_authors.name: Aaron Adcock
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-04T08:21:16.597Z
paper_authors._id: 645bd1631c24cd669dd8f2b3
paper_authors.avatarUrl: /avatars/f592e8b66a31ed70a745512b63926946.svg
paper_authors.fullname: Aaron Adcock
paper_authors.type: user
paper_authors.user: aadcock
paper_authors._id: 64f531ba4937e64ca402e3f3
paper_authors.name: Cheng-Yang Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-04T08:23:03.028Z
paper_authors._id: 64cc01bd87bfb0fc2124b112
paper_authors.avatarUrl: /avatars/32c20e200360e2bfd7bfe29ed37ea242.svg
paper_authors.fullname: Cheng-Yang Fu
paper_authors.type: user
paper_authors.user: chengyangfu1985
paper_authors._id: 64f531ba4937e64ca402e3f4
paper_authors.name: Melissa Hall
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:11:08.621Z
paper_authors._id: 6697bfb4acbced03ebd133f8
paper_authors.avatarUrl: /avatars/a7d41367c6f7974303feb5af2a387ce4.svg
paper_authors.fullname: Melissa Hall
paper_authors.type: user
paper_authors.user: melissah14
paper_authors._id: 64f531ba4937e64ca402e3f5
paper_authors.name: Candace Ross
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-04T08:23:30.473Z
paper_authors._id: 628709d57cc3b7bc1b95fcf0
paper_authors.avatarUrl: /avatars/98e0d9695527125a0409d1098cdceaa4.svg
paper_authors.fullname: Candace Ross
paper_authors.type: user
paper_authors.user: candaceross
paper_title: FACET: Fairness in Computer Vision Evaluation Benchmark
paper_summary: Computer vision models have known performance disparities across attributes
  such as gender and skin tone. This means during tasks such as classification
  and detection, model performance differs for certain classes based on the
  demographics of the people in the image. These disparities have been shown to
  exist, but until now there has not been a unified approach to measure these
  differences for common use-cases of computer vision models. We present a new
  benchmark named FACET (FAirness in Computer Vision EvaluaTion), a large,
  publicly available evaluation set of 32k images for some of the most common
  vision tasks - image classification, object detection and segmentation. For
  every image in FACET, we hired expert reviewers to manually annotate
  person-related attributes such as perceived skin tone and hair type, manually
  draw bounding boxes and label fine-grained person-related classes such as disk
  jockey or guitarist. In addition, we use FACET to benchmark state-of-the-art
  vision models and present a deeper understanding of potential performance
  disparities and challenges across sensitive demographic attributes. With the
  exhaustive annotations collected, we probe models using single demographics
  attributes as well as multiple attributes using an intersectional approach
  (e.g. hair color and perceived skin tone). Our results show that
  classification, detection, segmentation, and visual grounding models exhibit
  performance disparities across demographic attributes and intersections of
  attributes. These harms suggest that not all people represented in datasets
  receive fair and equitable treatment in these vision tasks. We hope current and
  future results using our benchmark will contribute to fairer, more robust
  vision models. FACET is available publicly at https://facet.metademolab.com/
paper_ai_keywords: image classification object detection segmentation bounding boxes disk jockey guitarist visual grounding performance disparities demographic attributes intersectional approach hair color blurred skin tone