title: Sightation Counts: Leveraging Sighted User Feedback in Building a
    BLV-aligned Dataset of Diagram Descriptions
summary: Often, the needs and visual abilities differ between the annotator group and
  the end user group. Generating detailed diagram descriptions for blind and
  low-vision (BLV) users is one such challenging domain. Sighted annotators could
  describe visuals with ease, but existing studies have shown that direct
  generations by them are costly, bias-prone, and somewhat lacking by BLV
  standards. In this study, we ask sighted individuals to assess -- rather than
  produce -- diagram descriptions generated by vision-language models (VLM) that
  have been guided with latent supervision via a multi-pass inference. The
  sighted assessments prove effective and useful to professional educators who
  are themselves BLV and teach visually impaired learners. We release Sightation,
  a collection of diagram description datasets spanning 5k diagrams and 137k
  samples for completion, preference, retrieval, question answering, and
  reasoning training purposes and demonstrate their fine-tuning potential in
  various downstream tasks.
paper_authors._id: 67d8e38e5fde3b1be16874e4
paper_authors.name: Wan Ju Kang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-18T08:08:01.967Z
paper_authors._id: 64b214c4f4361a032002cdcf
paper_authors.avatarUrl: /avatars/3188a4edec4d6c945a0ed9f36050a03c.svg
paper_authors.fullname: Andrew Wan Ju Kang
paper_authors.type: user
paper_authors.user: soarhigh
paper_authors._id: 67d8e38e5fde3b1be16874e5
paper_authors.name: Eunki Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-18T09:36:36.532Z
paper_authors._id: 628e3b87a2cb9819d4391ba6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1653488512816-noauth.jpeg
paper_authors.fullname: Eunki Kim
paper_authors.type: user
paper_authors.user: eunkey
paper_authors._id: 67d8e38e5fde3b1be16874e6
paper_authors.name: Na Min An
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-18T09:36:43.821Z
paper_authors._id: 65cccd8c80d3c4b865d3b262
paper_authors.avatarUrl: /avatars/e6f6d8f06dd54e1e7b6d686835a9c075.svg
paper_authors.fullname: Na Min An
paper_authors.type: user
paper_authors.user: namin0202
paper_authors._id: 67d8e38e5fde3b1be16874e7
paper_authors.name: Sangryul Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-18T09:36:49.867Z
paper_authors._id: 62f2638d04674e28535d40f8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1672818467177-62f2638d04674e28535d40f8.png
paper_authors.fullname: Sangryul Kim
paper_authors.type: user
paper_authors.user: sangryul
paper_authors._id: 67d8e38e5fde3b1be16874e8
paper_authors.name: Haemin Choi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-18T11:34:30.468Z
paper_authors._id: 67a86f66c6f66e2fa5888b41
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/67a86f66c6f66e2fa5888b41/P2d1vJ_iGn5QWvOYxn7pc.jpeg
paper_authors.fullname: Haemin Choi
paper_authors.type: user
paper_authors.user: Jaime-Choi
paper_authors._id: 67d8e38e5fde3b1be16874e9
paper_authors.name: Ki Hoon Kwak
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-19T09:46:08.722Z
paper_authors._id: 669c9a4ddac1eb34c084f9b5
paper_authors.avatarUrl: /avatars/6bd176e10e5ac551218ac83bece96f65.svg
paper_authors.fullname: Kwak Ki Hoon
paper_authors.type: user
paper_authors.user: kwakk
paper_authors._id: 67d8e38e5fde3b1be16874ea
paper_authors.name: James Thorne
paper_title: Sightation Counts: Leveraging Sighted User Feedback in Building a
    BLV-aligned Dataset of Diagram Descriptions
paper_summary: Often, the needs and visual abilities differ between the annotator group and
  the end user group. Generating detailed diagram descriptions for blind and
  low-vision (BLV) users is one such challenging domain. Sighted annotators could
  describe visuals with ease, but existing studies have shown that direct
  generations by them are costly, bias-prone, and somewhat lacking by BLV
  standards. In this study, we ask sighted individuals to assess -- rather than
  produce -- diagram descriptions generated by vision-language models (VLM) that
  have been guided with latent supervision via a multi-pass inference. The
  sighted assessments prove effective and useful to professional educators who
  are themselves BLV and teach visually impaired learners. We release Sightation,
  a collection of diagram description datasets spanning 5k diagrams and 137k
  samples for completion, preference, retrieval, question answering, and
  reasoning training purposes and demonstrate their fine-tuning potential in
  various downstream tasks.
paper_ai_summary: Sighted individuals assess descriptions of visuals generated by vision-language models with latent supervision, aiding professional educators for visually impaired learners, and releasing a dataset for model fine-tuning.
paper_ai_keywords: vision-language models latent supervision multi-pass inference diagram description datasets