title: DOCCI: Descriptions of Connected and Contrasting Images
summary: Vision-language datasets are vital for both text-to-image (T2I) and
  image-to-text (I2T) research. However, current datasets lack descriptions with
  fine-grained detail that would allow for richer associations to be learned by
  models. To fill the gap, we introduce Descriptions of Connected and Contrasting
  Images (DOCCI), a dataset with long, human-annotated English descriptions for
  15k images that were taken, curated and donated by a single researcher intent
  on capturing key challenges such as spatial relations, counting, text
  rendering, world knowledge, and more. We instruct human annotators to create
  comprehensive descriptions for each image; these average 136 words in length
  and are crafted to clearly distinguish each image from those that are related
  or similar. Each description is highly compositional and typically encompasses
  multiple challenges. Through both quantitative and qualitative analyses, we
  demonstrate that DOCCI serves as an effective training resource for
  image-to-text generation -- a PaLI 5B model finetuned on DOCCI shows equal or
  superior results compared to highly-performant larger models like LLaVA-1.5 7B
  and InstructBLIP 7B. Furthermore, we show that DOCCI is a useful testbed for
  text-to-image generation, highlighting the limitations of current text-to-image
  models in capturing long descriptions and fine details.
paper_authors._id: 66319f01351231c428faf20e
paper_authors.name: Yasumasa Onoe
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T08:33:46.503Z
paper_authors._id: 60591867ce97b2b478a3dc4d
paper_authors.avatarUrl: /avatars/bc8fedbca5f60f99fe66b9ac37ab9a1a.svg
paper_authors.fullname: Yasumasa Onoe
paper_authors.type: user
paper_authors.user: yasumasaonoe
paper_authors._id: 66319f01351231c428faf20f
paper_authors.name: Sunayana Rane
paper_authors._id: 66319f01351231c428faf210
paper_authors.name: Zachary Berger
paper_authors._id: 66319f01351231c428faf211
paper_authors.name: Yonatan Bitton
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T08:34:01.549Z
paper_authors._id: 632e0771ae0a7b1fc95630bf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1663961181981-632e0771ae0a7b1fc95630bf.jpeg
paper_authors.fullname: Yonatan
paper_authors.type: user
paper_authors.user: yonatanbitton
paper_authors._id: 66319f01351231c428faf212
paper_authors.name: Jaemin Cho
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-02T07:11:24.721Z
paper_authors._id: 5ffe32d8942cf3533d364449
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654821969191-5ffe32d8942cf3533d364449.jpeg
paper_authors.fullname: Jaemin Cho
paper_authors.type: user
paper_authors.user: j-min
paper_authors._id: 66319f01351231c428faf213
paper_authors.name: Roopal Garg
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T08:34:08.126Z
paper_authors._id: 6490ffd4f049c475bdb6a40f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6490ffd4f049c475bdb6a40f/EQQQamUGoBsoq0w_BFflM.jpeg
paper_authors.fullname: Roopal Garg
paper_authors.type: user
paper_authors.user: roopalgarg
paper_authors._id: 66319f01351231c428faf214
paper_authors.name: Alexander Ku
paper_authors._id: 66319f01351231c428faf215
paper_authors.name: Zarana Parekh
paper_authors._id: 66319f01351231c428faf216
paper_authors.name: Jordi Pont-Tuset
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-02T09:24:27.825Z
paper_authors._id: 66335b4b07f864a2f94b403d
paper_authors.avatarUrl: /avatars/0a290942c4a282b48f870500cc6424b5.svg
paper_authors.fullname: Jordi Pont Tuset
paper_authors.type: user
paper_authors.user: jponttuset
paper_authors._id: 66319f01351231c428faf217
paper_authors.name: Garrett Tanzer
paper_authors._id: 66319f01351231c428faf218
paper_authors.name: Su Wang
paper_authors._id: 66319f01351231c428faf219
paper_authors.name: Jason Baldridge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T08:34:31.222Z
paper_authors._id: 630c9f648e3ff0c72327e8ee
paper_authors.avatarUrl: /avatars/1a5262c6575daf92f62f5cc6d9f1f00f.svg
paper_authors.fullname: Jason Baldridge
paper_authors.type: user
paper_authors.user: jasonbaldridge
paper_title: DOCCI: Descriptions of Connected and Contrasting Images
paper_summary: Vision-language datasets are vital for both text-to-image (T2I) and
  image-to-text (I2T) research. However, current datasets lack descriptions with
  fine-grained detail that would allow for richer associations to be learned by
  models. To fill the gap, we introduce Descriptions of Connected and Contrasting
  Images (DOCCI), a dataset with long, human-annotated English descriptions for
  15k images that were taken, curated and donated by a single researcher intent
  on capturing key challenges such as spatial relations, counting, text
  rendering, world knowledge, and more. We instruct human annotators to create
  comprehensive descriptions for each image; these average 136 words in length
  and are crafted to clearly distinguish each image from those that are related
  or similar. Each description is highly compositional and typically encompasses
  multiple challenges. Through both quantitative and qualitative analyses, we
  demonstrate that DOCCI serves as an effective training resource for
  image-to-text generation -- a PaLI 5B model finetuned on DOCCI shows equal or
  superior results compared to highly-performant larger models like LLaVA-1.5 7B
  and InstructBLIP 7B. Furthermore, we show that DOCCI is a useful testbed for
  text-to-image generation, highlighting the limitations of current text-to-image
  models in capturing long descriptions and fine details.
paper_ai_summary: The Descriptions of Connected and Contrasting Images (DOCCI) dataset provides detailed human-annotated descriptions for image-to-text and text-to-image generation, demonstrating effectiveness in training and highlighting limitations in current models.
paper_ai_keywords: vision-language datasets text-to-image image-to-text Descriptions of Connected and Contrasting Images human-annotated spatial relations counting text rendering world knowledge PaLI 5B LLaVA-1.5 InstructBLIP image-to-text generation text-to-image generation