Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting
  Region Captions
Humans describe complex scenes with compositionality, using simple text
descriptions enriched with links and relationships. While vision-language
research has aimed to develop models with compositional understanding
capabilities, this is not reflected yet in existing datasets which, for the
most part, still use plain text to describe images. In this work, we propose a
new annotation strategy, graph-based captioning (GBC) that describes an image
using a labelled graph structure, with nodes of various types. The nodes in GBC
are created using, in a first stage, object detection and dense captioning
tools nested recursively to uncover and describe entity nodes, further linked
together in a second stage by highlighting, using new types of nodes,
compositions and relations among entities. Since all GBC nodes hold plain text
descriptions, GBC retains the flexibility found in natural language, but can
also encode hierarchical information in its edges. We demonstrate that GBC can
be produced automatically, using off-the-shelf multimodal LLMs and
open-vocabulary detection models, by building a new dataset, GBC10M, gathering
GBC annotations for about 10M images of the CC12M dataset. We use GBC10M to
showcase the wealth of node captions uncovered by GBC, as measured with CLIP
training. We show that using GBC nodes' annotations -- notably those stored in
composition and relation nodes -- results in significant performance boost on
downstream models when compared to other dataset formats. To further explore
the opportunities provided by GBC, we also propose a new attention mechanism
that can leverage the entire GBC graph, with encouraging experimental results
that show the extra benefits of incorporating the graph structure. Our datasets
are released at https://huggingface.co/graph-based-captions.
668e4386d8804369fef280ef
Yu-Guan Hsieh
extracted_confirmed
2024-07-10T08:51:47.062Z
6660ab1b0911c0a2a1f77c8c
/avatars/13c8657b2e06900896de0edc3136bd24.svg
Yu-Guan Hsieh
user
yhsieh
668e4386d8804369fef280f0
Cheng-Yu Hsieh
claimed_verified
2024-07-10T10:20:20.405Z
6660ab1b0911c0a2a1f77c8c
/avatars/13c8657b2e06900896de0edc3136bd24.svg
Yu-Guan Hsieh
user
yhsieh
668e4386d8804369fef280f1
Shih-Ying Yeh
claimed_verified
2024-07-10T11:57:08.963Z
630593e2fca1d8d92b81d2a1
https://cdn-avatars.huggingface.co/v1/production/uploads/630593e2fca1d8d92b81d2a1/PBNjstYFGAoyx59n3eYOf.png
Shih-Ying Yeh
user
KBlueLeaf
668e4386d8804369fef280f2
Louis Béthune
admin_assigned
2024-07-10T12:22:12.840Z
662a8d7faa7781f8ea521f16
/avatars/c05f9a7885b89c8818d52b98b52ecc7b.svg
Louis Bethune
user
AlgueRythme
668e4386d8804369fef280f3
Hadi Pour Ansari
668e4386d8804369fef280f4
Pavan Kumar Anasosalu Vasu
claimed_verified
2024-07-26T07:41:03.300Z
65e000cce1823ce0df17a3d4
/avatars/b50c2d2c1f3ad1114867e199fd578a29.svg
Pavan Kumar Vasu
user
pavankumarvasu
668e4386d8804369fef280f5
Chun-Liang Li
admin_assigned
2024-07-10T12:22:28.792Z
65de6ef104b9352580ce23bc
/avatars/aace4a7c8ee9a591f1103a2b38f9b9ff.svg
Chun-Liang Li
user
chunliangli
668e4386d8804369fef280f6
Ranjay Krishna
admin_assigned
2024-07-10T12:22:34.483Z
66429868ab89e3a3a85668b0
/avatars/170e0daa454838deee2bf946f7118651.svg
Ranjay Krishna
user
ranjaykrishna
668e4386d8804369fef280f7
Oncel Tuzel
668e4386d8804369fef280f8
Marco Cuturi
admin_assigned
2024-07-10T12:22:44.066Z
65d88dfda5269668b735a39b
/avatars/cf46727be075a51ab894efbe906ee25a.svg
marco cuturi
user
marcocuturi
Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting
  Region Captions
Humans describe complex scenes with compositionality, using simple text
descriptions enriched with links and relationships. While vision-language
research has aimed to develop models with compositional understanding
capabilities, this is not reflected yet in existing datasets which, for the
most part, still use plain text to describe images. In this work, we propose a
new annotation strategy, graph-based captioning (GBC) that describes an image
using a labelled graph structure, with nodes of various types. The nodes in GBC
are created using, in a first stage, object detection and dense captioning
tools nested recursively to uncover and describe entity nodes, further linked
together in a second stage by highlighting, using new types of nodes,
compositions and relations among entities. Since all GBC nodes hold plain text
descriptions, GBC retains the flexibility found in natural language, but can
also encode hierarchical information in its edges. We demonstrate that GBC can
be produced automatically, using off-the-shelf multimodal LLMs and
open-vocabulary detection models, by building a new dataset, GBC10M, gathering
GBC annotations for about 10M images of the CC12M dataset. We use GBC10M to
showcase the wealth of node captions uncovered by GBC, as measured with CLIP
training. We show that using GBC nodes' annotations -- notably those stored in
composition and relation nodes -- results in significant performance boost on
downstream models when compared to other dataset formats. To further explore
the opportunities provided by GBC, we also propose a new attention mechanism
that can leverage the entire GBC graph, with encouraging experimental results
that show the extra benefits of incorporating the graph structure. Our datasets
are released at https://huggingface.co/graph-based-captions.
A new graph-based captioning (GBC) annotation strategy enhances vision-language models by encoding hierarchical image descriptions with nodes representing entities, compositions, and relations, resulting in improved performance using multimodal LLMs and an attention mechanism.
graph-based captioning GBC object detection dense captioning CLIP attention mechanism multimodal LLMs open-vocabulary detection models