DOCCI: Descriptions of Connected and Contrasting Images
Vision-language datasets are vital for both text-to-image (T2I) and
image-to-text (I2T) research. However, current datasets lack descriptions with
fine-grained detail that would allow for richer associations to be learned by
models. To fill the gap, we introduce Descriptions of Connected and Contrasting
Images (DOCCI), a dataset with long, human-annotated English descriptions for
15k images that were taken, curated and donated by a single researcher intent
on capturing key challenges such as spatial relations, counting, text
rendering, world knowledge, and more. We instruct human annotators to create
comprehensive descriptions for each image; these average 136 words in length
and are crafted to clearly distinguish each image from those that are related
or similar. Each description is highly compositional and typically encompasses
multiple challenges. Through both quantitative and qualitative analyses, we
demonstrate that DOCCI serves as an effective training resource for
image-to-text generation -- a PaLI 5B model finetuned on DOCCI shows equal or
superior results compared to highly-performant larger models like LLaVA-1.5 7B
and InstructBLIP 7B. Furthermore, we show that DOCCI is a useful testbed for
text-to-image generation, highlighting the limitations of current text-to-image
models in capturing long descriptions and fine details.
66319f01351231c428faf20e
Yasumasa Onoe
admin_assigned
2024-05-02T08:33:46.503Z
60591867ce97b2b478a3dc4d
/avatars/bc8fedbca5f60f99fe66b9ac37ab9a1a.svg
Yasumasa Onoe
user
yasumasaonoe
66319f01351231c428faf20f
Sunayana Rane
66319f01351231c428faf210
Zachary Berger
66319f01351231c428faf211
Yonatan Bitton
admin_assigned
2024-05-02T08:34:01.549Z
632e0771ae0a7b1fc95630bf
https://cdn-avatars.huggingface.co/v1/production/uploads/1663961181981-632e0771ae0a7b1fc95630bf.jpeg
Yonatan
user
yonatanbitton
66319f01351231c428faf212
Jaemin Cho
claimed_verified
2024-05-02T07:11:24.721Z
5ffe32d8942cf3533d364449
https://cdn-avatars.huggingface.co/v1/production/uploads/1654821969191-5ffe32d8942cf3533d364449.jpeg
Jaemin Cho
user
j-min
66319f01351231c428faf213
Roopal Garg
admin_assigned
2024-05-02T08:34:08.126Z
6490ffd4f049c475bdb6a40f
https://cdn-avatars.huggingface.co/v1/production/uploads/6490ffd4f049c475bdb6a40f/EQQQamUGoBsoq0w_BFflM.jpeg
Roopal Garg
user
roopalgarg
66319f01351231c428faf214
Alexander Ku
66319f01351231c428faf215
Zarana Parekh
66319f01351231c428faf216
Jordi Pont-Tuset
claimed_verified
2024-05-02T09:24:27.825Z
66335b4b07f864a2f94b403d
/avatars/0a290942c4a282b48f870500cc6424b5.svg
Jordi Pont Tuset
user
jponttuset
66319f01351231c428faf217
Garrett Tanzer
66319f01351231c428faf218
Su Wang
66319f01351231c428faf219
Jason Baldridge
admin_assigned
2024-05-02T08:34:31.222Z
630c9f648e3ff0c72327e8ee
/avatars/1a5262c6575daf92f62f5cc6d9f1f00f.svg
Jason Baldridge
user
jasonbaldridge
DOCCI: Descriptions of Connected and Contrasting Images
Vision-language datasets are vital for both text-to-image (T2I) and
image-to-text (I2T) research. However, current datasets lack descriptions with
fine-grained detail that would allow for richer associations to be learned by
models. To fill the gap, we introduce Descriptions of Connected and Contrasting
Images (DOCCI), a dataset with long, human-annotated English descriptions for
15k images that were taken, curated and donated by a single researcher intent
on capturing key challenges such as spatial relations, counting, text
rendering, world knowledge, and more. We instruct human annotators to create
comprehensive descriptions for each image; these average 136 words in length
and are crafted to clearly distinguish each image from those that are related
or similar. Each description is highly compositional and typically encompasses
multiple challenges. Through both quantitative and qualitative analyses, we
demonstrate that DOCCI serves as an effective training resource for
image-to-text generation -- a PaLI 5B model finetuned on DOCCI shows equal or
superior results compared to highly-performant larger models like LLaVA-1.5 7B
and InstructBLIP 7B. Furthermore, we show that DOCCI is a useful testbed for
text-to-image generation, highlighting the limitations of current text-to-image
models in capturing long descriptions and fine details.
The Descriptions of Connected and Contrasting Images (DOCCI) dataset provides detailed human-annotated descriptions for image-to-text and text-to-image generation, demonstrating effectiveness in training and highlighting limitations in current models.
vision-language datasets text-to-image image-to-text Descriptions of Connected and Contrasting Images human-annotated spatial relations counting text rendering world knowledge PaLI 5B LLaVA-1.5 InstructBLIP image-to-text generation text-to-image generation