title: CoVLM: Composing Visual Entities and Relationships in Large Language
    Models Via Communicative Decoding
summary: A remarkable ability of human beings resides in compositional reasoning,
  i.e., the capacity to make "infinite use of finite means". However, current
  large vision-language foundation models (VLMs) fall short of such compositional
  abilities due to their "bag-of-words" behaviors and inability to construct
  words that correctly represent visual entities and the relations among the
  entities. To this end, we propose CoVLM, which can guide the LLM to explicitly
  compose visual entities and relationships among the text and dynamically
  communicate with the vision encoder and detection network to achieve
  vision-language communicative decoding. Specifically, we first devise a set of
  novel communication tokens for the LLM, for dynamic communication between the
  visual detection system and the language system. A communication token is
  generated by the LLM following a visual entity or a relation, to inform the
  detection network to propose regions that are relevant to the sentence
  generated so far. The proposed regions-of-interests (ROIs) are then fed back
  into the LLM for better language generation contingent on the relevant regions.
  The LLM is thus able to compose the visual entities and relationships through
  the communication tokens. The vision-to-language and language-to-vision
  communication are iteratively performed until the entire sentence is generated.
  Our framework seamlessly bridges the gap between visual perception and LLMs and
  outperforms previous VLMs by a large margin on compositional reasoning
  benchmarks (e.g., ~20% in HICO-DET mAP, ~14% in Cola top-1 accuracy, and ~3% on
  ARO top-1 accuracy). We also achieve state-of-the-art performances on
  traditional vision-language tasks such as referring expression comprehension
  and visual question answering.
paper_authors._id: 6549ca71c26b9e805de70681
paper_authors.name: Junyan Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-07T20:23:33.070Z
paper_authors._id: 62d09eb86a61a88ea0d83918
paper_authors.avatarUrl: /avatars/81b511d94cced304ffca058caff662d4.svg
paper_authors.fullname: Junyan Li
paper_authors.type: user
paper_authors.user: senfu
paper_authors._id: 6549ca71c26b9e805de70682
paper_authors.name: Delin Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T17:13:21.901Z
paper_authors._id: 62189e34523d13b0992f67ac
paper_authors.avatarUrl: /avatars/b5153ff75b20185b195eb38c4786fd34.svg
paper_authors.fullname: Delin Chen
paper_authors.type: user
paper_authors.user: chendl
paper_authors._id: 6549ca71c26b9e805de70683
paper_authors.name: Yining Hong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T17:13:14.644Z
paper_authors._id: 6431b64df76c34519e93d1ba
paper_authors.avatarUrl: /avatars/ea577762b6b4798f87a7a3f1d53d082c.svg
paper_authors.fullname: Yining Hong
paper_authors.type: user
paper_authors.user: evelynhong
paper_authors._id: 6549ca71c26b9e805de70684
paper_authors.name: Zhenfang Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T17:14:51.749Z
paper_authors._id: 6216f8a3a805e5a4d418f4cb
paper_authors.avatarUrl: /avatars/8860471b3e397ca61505ee2edf73e4a6.svg
paper_authors.fullname: Zhenfang Chen
paper_authors.type: user
paper_authors.user: zfchen
paper_authors._id: 6549ca71c26b9e805de70685
paper_authors.name: Peihao Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T17:15:23.682Z
paper_authors._id: 64bfe895c874dda773303b44
paper_authors.avatarUrl: /avatars/a54ddb6d69a73f63f9d848f28abcbff3.svg
paper_authors.fullname: PeihaoChen
paper_authors.type: user
paper_authors.user: PeihaoChen
paper_authors._id: 6549ca71c26b9e805de70686
paper_authors.name: Yikang Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T17:16:05.485Z
paper_authors._id: 6463995939359568c63d740c
paper_authors.avatarUrl: /avatars/11caacefc68c1259dbc8764cc4340481.svg
paper_authors.fullname: Yikang Shen
paper_authors.type: user
paper_authors.user: YikangS
paper_authors._id: 6549ca71c26b9e805de70687
paper_authors.name: Chuang Gan
paper_title: CoVLM: Composing Visual Entities and Relationships in Large Language
    Models Via Communicative Decoding
paper_summary: A remarkable ability of human beings resides in compositional reasoning,
  i.e., the capacity to make "infinite use of finite means". However, current
  large vision-language foundation models (VLMs) fall short of such compositional
  abilities due to their "bag-of-words" behaviors and inability to construct
  words that correctly represent visual entities and the relations among the
  entities. To this end, we propose CoVLM, which can guide the LLM to explicitly
  compose visual entities and relationships among the text and dynamically
  communicate with the vision encoder and detection network to achieve
  vision-language communicative decoding. Specifically, we first devise a set of
  novel communication tokens for the LLM, for dynamic communication between the
  visual detection system and the language system. A communication token is
  generated by the LLM following a visual entity or a relation, to inform the
  detection network to propose regions that are relevant to the sentence
  generated so far. The proposed regions-of-interests (ROIs) are then fed back
  into the LLM for better language generation contingent on the relevant regions.
  The LLM is thus able to compose the visual entities and relationships through
  the communication tokens. The vision-to-language and language-to-vision
  communication are iteratively performed until the entire sentence is generated.
  Our framework seamlessly bridges the gap between visual perception and LLMs and
  outperforms previous VLMs by a large margin on compositional reasoning
  benchmarks (e.g., ~20% in HICO-DET mAP, ~14% in Cola top-1 accuracy, and ~3% on
  ARO top-1 accuracy). We also achieve state-of-the-art performances on
  traditional vision-language tasks such as referring expression comprehension
  and visual question answering.
paper_ai_summary: CoVLM enhances compositional reasoning in vision-language models by introducing communication tokens for iterative interaction between vision and language systems.
paper_ai_keywords: large vision-language foundation models compositional abilities bag-of-words communication tokens regions-of-interest visual-to-language language-to-vision referring expression comprehension visual question answering