title: CogCoM: Train Large Vision-Language Models Diving into Details through
    Chain of Manipulations
summary: Vision-Language Models (VLMs) have demonstrated their widespread viability
  thanks to extensive training in aligning visual instructions to answers.
  However, this conclusive alignment leads models to ignore critical visual
  reasoning, and further result in failures on meticulous visual problems and
  unfaithful responses. In this paper, we propose Chain of Manipulations, a
  mechanism that enables VLMs to solve problems with a series of manipulations,
  where each manipulation refers to an operation on the visual input, either from
  intrinsic abilities (e.g., grounding) acquired through prior training or from
  imitating human-like behaviors (e.g., zoom in). This mechanism encourages VLMs
  to generate faithful responses with evidential visual reasoning, and permits
  users to trace error causes in the interpretable paths. We thus train CogCoM, a
  general 17B VLM with a memory-based compatible architecture endowed this
  reasoning mechanism. Experiments show that our model achieves the
  state-of-the-art performance across 8 benchmarks from 3 categories, and a
  limited number of training steps with the data swiftly gains a competitive
  performance. The code and data are publicly available at
  https://github.com/THUDM/CogCoM.
paper_authors._id: 65c2f87dd14c096f8a69bd08
paper_authors.name: Ji Qi
paper_authors._id: 65c2f87dd14c096f8a69bd09
paper_authors.name: Ming Ding
paper_authors._id: 65c2f87dd14c096f8a69bd0a
paper_authors.name: Weihan Wang
paper_authors._id: 65c2f87dd14c096f8a69bd0b
paper_authors.name: Yushi Bai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T20:22:22.146Z
paper_authors._id: 64ed568ccf6118a9379a61b8
paper_authors.avatarUrl: /avatars/6d040cbcb4a9b624cbe64c9d01cd5c88.svg
paper_authors.fullname: Yushi Bai
paper_authors.type: user
paper_authors.user: bys0318
paper_authors._id: 65c2f87dd14c096f8a69bd0c
paper_authors.name: Qingsong Lv
paper_authors._id: 65c2f87dd14c096f8a69bd0d
paper_authors.name: Wenyi Hong
paper_authors._id: 65c2f87dd14c096f8a69bd0e
paper_authors.name: Bin Xu
paper_authors._id: 65c2f87dd14c096f8a69bd0f
paper_authors.name: Lei Hou
paper_authors._id: 65c2f87dd14c096f8a69bd10
paper_authors.name: Juanzi Li
paper_authors._id: 65c2f87dd14c096f8a69bd11
paper_authors.name: Yuxiao Dong
paper_authors._id: 65c2f87dd14c096f8a69bd12
paper_authors.name: Jie Tang
paper_title: CogCoM: Train Large Vision-Language Models Diving into Details through
    Chain of Manipulations
paper_summary: Vision-Language Models (VLMs) have demonstrated their widespread viability
  thanks to extensive training in aligning visual instructions to answers.
  However, this conclusive alignment leads models to ignore critical visual
  reasoning, and further result in failures on meticulous visual problems and
  unfaithful responses. In this paper, we propose Chain of Manipulations, a
  mechanism that enables VLMs to solve problems with a series of manipulations,
  where each manipulation refers to an operation on the visual input, either from
  intrinsic abilities (e.g., grounding) acquired through prior training or from
  imitating human-like behaviors (e.g., zoom in). This mechanism encourages VLMs
  to generate faithful responses with evidential visual reasoning, and permits
  users to trace error causes in the interpretable paths. We thus train CogCoM, a
  general 17B VLM with a memory-based compatible architecture endowed this
  reasoning mechanism. Experiments show that our model achieves the
  state-of-the-art performance across 8 benchmarks from 3 categories, and a
  limited number of training steps with the data swiftly gains a competitive
  performance. The code and data are publicly available at
  https://github.com/THUDM/CogCoM.
paper_ai_keywords: Chain of Manipulations visual reasoning evidential visual reasoning intrinsic abilities memory-based compatible architecture CogCoM