title: Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning
    Instruction Using Language Model
summary: Although most current large multimodal models (LMMs) can already understand
  photos of natural scenes and portraits, their understanding of abstract images,
  e.g., charts, maps, or layouts, and visual reasoning capabilities remains quite
  rudimentary. They often struggle with simple daily tasks, such as reading time
  from a clock, understanding a flowchart, or planning a route using a road map.
  In light of this, we design a multi-modal self-instruct, utilizing large
  language models and their code capabilities to synthesize massive abstract
  images and visual reasoning instructions across daily scenarios. Our strategy
  effortlessly creates a multimodal benchmark with 11,193 instructions for eight
  visual scenarios: charts, tables, simulated maps, dashboards, flowcharts,
  relation graphs, floor plans, and visual puzzles. This benchmark,
  constructed with simple lines and geometric elements, exposes the shortcomings
  of most advanced LMMs like Claude-3.5-Sonnet and GPT-4o in abstract image
  understanding, spatial relations reasoning, and visual element induction.
  Besides, to verify the quality of our synthetic data, we fine-tune an LMM using
  62,476 synthetic chart, table and road map instructions. The results
  demonstrate improved chart understanding and map navigation performance, and
  also demonstrate potential benefits for other visual reasoning tasks. Our code
  is available at: https://github.com/zwq2018/Multi-modal-Self-instruct.
paper_authors._id: 668df563b09a05f3d7e46c93
paper_authors.name: Wenqi Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-12T07:41:30.700Z
paper_authors._id: 6485bd278d14bcd5cdbb7c8d
paper_authors.avatarUrl: /avatars/1427cf1a72b5db0cb263ad45885cf925.svg
paper_authors.fullname: Wenqi Zhang
paper_authors.type: user
paper_authors.user: zwq2018
paper_authors._id: 668df563b09a05f3d7e46c94
paper_authors.name: Zhenglin Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-11T14:19:02.062Z
paper_authors._id: 65028e8389707f182386588c
paper_authors.avatarUrl: /avatars/86a748a3264e6e0f4ee5eaf8f7032ecb.svg
paper_authors.fullname: Zhenglin Cheng
paper_authors.type: user
paper_authors.user: kenshinn
paper_authors._id: 668df563b09a05f3d7e46c95
paper_authors.name: Yuanyu He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-12T14:53:13.988Z
paper_authors._id: 668a94d30b04331a0b6dc457
paper_authors.avatarUrl: /avatars/66a311a7474e9d297231868dbbd6a18c.svg
paper_authors.fullname: heyuanyu
paper_authors.type: user
paper_authors.user: heyuanyu
paper_authors._id: 668df563b09a05f3d7e46c96
paper_authors.name: Mengna Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-12T09:19:03.533Z
paper_authors._id: 64f8092fdcd7b028c16cb8bd
paper_authors.avatarUrl: /avatars/381d51af9e7b040a940f5631018da6e9.svg
paper_authors.fullname: wang meng
paper_authors.type: user
paper_authors.user: meng-03
paper_authors._id: 668df563b09a05f3d7e46c97
paper_authors.name: Yongliang Shen
paper_authors._id: 668df563b09a05f3d7e46c98
paper_authors.name: Zeqi Tan
paper_authors._id: 668df563b09a05f3d7e46c99
paper_authors.name: Guiyang Hou
paper_authors._id: 668df563b09a05f3d7e46c9a
paper_authors.name: Mingqian He
paper_authors._id: 668df563b09a05f3d7e46c9b
paper_authors.name: Yanna Ma
paper_authors._id: 668df563b09a05f3d7e46c9c
paper_authors.name: Weiming Lu
paper_authors._id: 668df563b09a05f3d7e46c9d
paper_authors.name: Yueting Zhuang
paper_title: Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning
    Instruction Using Language Model
paper_summary: Although most current large multimodal models (LMMs) can already understand
  photos of natural scenes and portraits, their understanding of abstract images,
  e.g., charts, maps, or layouts, and visual reasoning capabilities remains quite
  rudimentary. They often struggle with simple daily tasks, such as reading time
  from a clock, understanding a flowchart, or planning a route using a road map.
  In light of this, we design a multi-modal self-instruct, utilizing large
  language models and their code capabilities to synthesize massive abstract
  images and visual reasoning instructions across daily scenarios. Our strategy
  effortlessly creates a multimodal benchmark with 11,193 instructions for eight
  visual scenarios: charts, tables, simulated maps, dashboards, flowcharts,
  relation graphs, floor plans, and visual puzzles. This benchmark,
  constructed with simple lines and geometric elements, exposes the shortcomings
  of most advanced LMMs like Claude-3.5-Sonnet and GPT-4o in abstract image
  understanding, spatial relations reasoning, and visual element induction.
  Besides, to verify the quality of our synthetic data, we fine-tune an LMM using
  62,476 synthetic chart, table and road map instructions. The results
  demonstrate improved chart understanding and map navigation performance, and
  also demonstrate potential benefits for other visual reasoning tasks. Our code
  is available at: https://github.com/zwq2018/Multi-modal-Self-instruct.
paper_ai_keywords: multimodal models self-instruct large language models synthetic abstract images visual reasoning instructions multimodal benchmark visual scenarios charts tables simulated maps dashboards flowcharts relation graphs floor plans visual puzzles geometric elements abstract image understanding spatial relations reasoning visual element induction fine-tune chart understanding map navigation synthetic chart synthetic table synthetic road map instructions