title: Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting
    Region Captions
summary: Humans describe complex scenes with compositionality, using simple text
  descriptions enriched with links and relationships. While vision-language
  research has aimed to develop models with compositional understanding
  capabilities, this is not reflected yet in existing datasets which, for the
  most part, still use plain text to describe images. In this work, we propose a
  new annotation strategy, graph-based captioning (GBC) that describes an image
  using a labelled graph structure, with nodes of various types. The nodes in GBC
  are created using, in a first stage, object detection and dense captioning
  tools nested recursively to uncover and describe entity nodes, further linked
  together in a second stage by highlighting, using new types of nodes,
  compositions and relations among entities. Since all GBC nodes hold plain text
  descriptions, GBC retains the flexibility found in natural language, but can
  also encode hierarchical information in its edges. We demonstrate that GBC can
  be produced automatically, using off-the-shelf multimodal LLMs and
  open-vocabulary detection models, by building a new dataset, GBC10M, gathering
  GBC annotations for about 10M images of the CC12M dataset. We use GBC10M to
  showcase the wealth of node captions uncovered by GBC, as measured with CLIP
  training. We show that using GBC nodes' annotations -- notably those stored in
  composition and relation nodes -- results in significant performance boost on
  downstream models when compared to other dataset formats. To further explore
  the opportunities provided by GBC, we also propose a new attention mechanism
  that can leverage the entire GBC graph, with encouraging experimental results
  that show the extra benefits of incorporating the graph structure. Our datasets
  are released at https://huggingface.co/graph-based-captions.
paper_authors._id: 668e4386d8804369fef280ef
paper_authors.name: Yu-Guan Hsieh
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-07-10T08:51:47.062Z
paper_authors._id: 6660ab1b0911c0a2a1f77c8c
paper_authors.avatarUrl: /avatars/13c8657b2e06900896de0edc3136bd24.svg
paper_authors.fullname: Yu-Guan Hsieh
paper_authors.type: user
paper_authors.user: yhsieh
paper_authors._id: 668e4386d8804369fef280f0
paper_authors.name: Cheng-Yu Hsieh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-10T10:20:20.405Z
paper_authors._id: 6660ab1b0911c0a2a1f77c8c
paper_authors.avatarUrl: /avatars/13c8657b2e06900896de0edc3136bd24.svg
paper_authors.fullname: Yu-Guan Hsieh
paper_authors.type: user
paper_authors.user: yhsieh
paper_authors._id: 668e4386d8804369fef280f1
paper_authors.name: Shih-Ying Yeh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-10T11:57:08.963Z
paper_authors._id: 630593e2fca1d8d92b81d2a1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/630593e2fca1d8d92b81d2a1/PBNjstYFGAoyx59n3eYOf.png
paper_authors.fullname: Shih-Ying Yeh
paper_authors.type: user
paper_authors.user: KBlueLeaf
paper_authors._id: 668e4386d8804369fef280f2
paper_authors.name: Louis Béthune
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T12:22:12.840Z
paper_authors._id: 662a8d7faa7781f8ea521f16
paper_authors.avatarUrl: /avatars/c05f9a7885b89c8818d52b98b52ecc7b.svg
paper_authors.fullname: Louis Bethune
paper_authors.type: user
paper_authors.user: AlgueRythme
paper_authors._id: 668e4386d8804369fef280f3
paper_authors.name: Hadi Pour Ansari
paper_authors._id: 668e4386d8804369fef280f4
paper_authors.name: Pavan Kumar Anasosalu Vasu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-26T07:41:03.300Z
paper_authors._id: 65e000cce1823ce0df17a3d4
paper_authors.avatarUrl: /avatars/b50c2d2c1f3ad1114867e199fd578a29.svg
paper_authors.fullname: Pavan Kumar Vasu
paper_authors.type: user
paper_authors.user: pavankumarvasu
paper_authors._id: 668e4386d8804369fef280f5
paper_authors.name: Chun-Liang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T12:22:28.792Z
paper_authors._id: 65de6ef104b9352580ce23bc
paper_authors.avatarUrl: /avatars/aace4a7c8ee9a591f1103a2b38f9b9ff.svg
paper_authors.fullname: Chun-Liang Li
paper_authors.type: user
paper_authors.user: chunliangli
paper_authors._id: 668e4386d8804369fef280f6
paper_authors.name: Ranjay Krishna
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T12:22:34.483Z
paper_authors._id: 66429868ab89e3a3a85668b0
paper_authors.avatarUrl: /avatars/170e0daa454838deee2bf946f7118651.svg
paper_authors.fullname: Ranjay Krishna
paper_authors.type: user
paper_authors.user: ranjaykrishna
paper_authors._id: 668e4386d8804369fef280f7
paper_authors.name: Oncel Tuzel
paper_authors._id: 668e4386d8804369fef280f8
paper_authors.name: Marco Cuturi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T12:22:44.066Z
paper_authors._id: 65d88dfda5269668b735a39b
paper_authors.avatarUrl: /avatars/cf46727be075a51ab894efbe906ee25a.svg
paper_authors.fullname: marco cuturi
paper_authors.type: user
paper_authors.user: marcocuturi
paper_title: Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting
    Region Captions
paper_summary: Humans describe complex scenes with compositionality, using simple text
  descriptions enriched with links and relationships. While vision-language
  research has aimed to develop models with compositional understanding
  capabilities, this is not reflected yet in existing datasets which, for the
  most part, still use plain text to describe images. In this work, we propose a
  new annotation strategy, graph-based captioning (GBC) that describes an image
  using a labelled graph structure, with nodes of various types. The nodes in GBC
  are created using, in a first stage, object detection and dense captioning
  tools nested recursively to uncover and describe entity nodes, further linked
  together in a second stage by highlighting, using new types of nodes,
  compositions and relations among entities. Since all GBC nodes hold plain text
  descriptions, GBC retains the flexibility found in natural language, but can
  also encode hierarchical information in its edges. We demonstrate that GBC can
  be produced automatically, using off-the-shelf multimodal LLMs and
  open-vocabulary detection models, by building a new dataset, GBC10M, gathering
  GBC annotations for about 10M images of the CC12M dataset. We use GBC10M to
  showcase the wealth of node captions uncovered by GBC, as measured with CLIP
  training. We show that using GBC nodes' annotations -- notably those stored in
  composition and relation nodes -- results in significant performance boost on
  downstream models when compared to other dataset formats. To further explore
  the opportunities provided by GBC, we also propose a new attention mechanism
  that can leverage the entire GBC graph, with encouraging experimental results
  that show the extra benefits of incorporating the graph structure. Our datasets
  are released at https://huggingface.co/graph-based-captions.
paper_ai_keywords: graph-based captioning (GBC) object detection dense captioning entity nodes composition nodes relation nodes off-the-shelf multimodal LLMs open-vocabulary models GBC10M CLIP training downstream models new attention mechanism