Sightation Counts: Leveraging Sighted User Feedback in Building a
  BLV-aligned Dataset of Diagram Descriptions
Often, the needs and visual abilities differ between the annotator group and
the end user group. Generating detailed diagram descriptions for blind and
low-vision (BLV) users is one such challenging domain. Sighted annotators could
describe visuals with ease, but existing studies have shown that direct
generations by them are costly, bias-prone, and somewhat lacking by BLV
standards. In this study, we ask sighted individuals to assess -- rather than
produce -- diagram descriptions generated by vision-language models (VLM) that
have been guided with latent supervision via a multi-pass inference. The
sighted assessments prove effective and useful to professional educators who
are themselves BLV and teach visually impaired learners. We release Sightation,
a collection of diagram description datasets spanning 5k diagrams and 137k
samples for completion, preference, retrieval, question answering, and
reasoning training purposes and demonstrate their fine-tuning potential in
various downstream tasks.
67d8e38e5fde3b1be16874e4
Wan Ju Kang
claimed_verified
2025-03-18T08:08:01.967Z
64b214c4f4361a032002cdcf
/avatars/3188a4edec4d6c945a0ed9f36050a03c.svg
Andrew Wan Ju Kang
user
soarhigh
67d8e38e5fde3b1be16874e5
Eunki Kim
admin_assigned
2025-03-18T09:36:36.532Z
628e3b87a2cb9819d4391ba6
https://cdn-avatars.huggingface.co/v1/production/uploads/1653488512816-noauth.jpeg
Eunki Kim
user
eunkey
67d8e38e5fde3b1be16874e6
Na Min An
admin_assigned
2025-03-18T09:36:43.821Z
65cccd8c80d3c4b865d3b262
/avatars/e6f6d8f06dd54e1e7b6d686835a9c075.svg
Na Min An
user
namin0202
67d8e38e5fde3b1be16874e7
Sangryul Kim
admin_assigned
2025-03-18T09:36:49.867Z
62f2638d04674e28535d40f8
https://cdn-avatars.huggingface.co/v1/production/uploads/1672818467177-62f2638d04674e28535d40f8.png
Sangryul Kim
user
sangryul
67d8e38e5fde3b1be16874e8
Haemin Choi
claimed_verified
2025-03-18T11:34:30.468Z
67a86f66c6f66e2fa5888b41
https://cdn-avatars.huggingface.co/v1/production/uploads/67a86f66c6f66e2fa5888b41/P2d1vJ_iGn5QWvOYxn7pc.jpeg
Haemin Choi
user
Jaime-Choi
67d8e38e5fde3b1be16874e9
Ki Hoon Kwak
claimed_verified
2025-03-19T09:46:08.722Z
669c9a4ddac1eb34c084f9b5
/avatars/6bd176e10e5ac551218ac83bece96f65.svg
Kwak Ki Hoon
user
kwakk
67d8e38e5fde3b1be16874ea
James Thorne
Sightation Counts: Leveraging Sighted User Feedback in Building a
  BLV-aligned Dataset of Diagram Descriptions
Often, the needs and visual abilities differ between the annotator group and
the end user group. Generating detailed diagram descriptions for blind and
low-vision (BLV) users is one such challenging domain. Sighted annotators could
describe visuals with ease, but existing studies have shown that direct
generations by them are costly, bias-prone, and somewhat lacking by BLV
standards. In this study, we ask sighted individuals to assess -- rather than
produce -- diagram descriptions generated by vision-language models (VLM) that
have been guided with latent supervision via a multi-pass inference. The
sighted assessments prove effective and useful to professional educators who
are themselves BLV and teach visually impaired learners. We release Sightation,
a collection of diagram description datasets spanning 5k diagrams and 137k
samples for completion, preference, retrieval, question answering, and
reasoning training purposes and demonstrate their fine-tuning potential in
various downstream tasks.
Sighted individuals assess descriptions of visuals generated by vision-language models with latent supervision, aiding professional educators for visually impaired learners, and releasing a dataset for model fine-tuning.
vision-language models latent supervision multi-pass inference diagram description datasets