Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning
  Instruction Using Language Model
Although most current large multimodal models (LMMs) can already understand
photos of natural scenes and portraits, their understanding of abstract images,
e.g., charts, maps, or layouts, and visual reasoning capabilities remains quite
rudimentary. They often struggle with simple daily tasks, such as reading time
from a clock, understanding a flowchart, or planning a route using a road map.
In light of this, we design a multi-modal self-instruct, utilizing large
language models and their code capabilities to synthesize massive abstract
images and visual reasoning instructions across daily scenarios. Our strategy
effortlessly creates a multimodal benchmark with 11,193 instructions for eight
visual scenarios: charts, tables, simulated maps, dashboards, flowcharts,
relation graphs, floor plans, and visual puzzles. This benchmark,
constructed with simple lines and geometric elements, exposes the shortcomings
of most advanced LMMs like Claude-3.5-Sonnet and GPT-4o in abstract image
understanding, spatial relations reasoning, and visual element induction.
Besides, to verify the quality of our synthetic data, we fine-tune an LMM using
62,476 synthetic chart, table and road map instructions. The results
demonstrate improved chart understanding and map navigation performance, and
also demonstrate potential benefits for other visual reasoning tasks. Our code
is available at: https://github.com/zwq2018/Multi-modal-Self-instruct.
668df563b09a05f3d7e46c93
Wenqi Zhang
claimed_verified
2024-07-12T07:41:30.700Z
6485bd278d14bcd5cdbb7c8d
/avatars/1427cf1a72b5db0cb263ad45885cf925.svg
Wenqi Zhang
user
zwq2018
668df563b09a05f3d7e46c94
Zhenglin Cheng
claimed_verified
2024-07-11T14:19:02.062Z
65028e8389707f182386588c
/avatars/86a748a3264e6e0f4ee5eaf8f7032ecb.svg
Zhenglin Cheng
user
kenshinn
668df563b09a05f3d7e46c95
Yuanyu He
claimed_verified
2024-07-12T14:53:13.988Z
668a94d30b04331a0b6dc457
/avatars/66a311a7474e9d297231868dbbd6a18c.svg
heyuanyu
user
heyuanyu
668df563b09a05f3d7e46c96
Mengna Wang
claimed_verified
2024-07-12T09:19:03.533Z
64f8092fdcd7b028c16cb8bd
/avatars/381d51af9e7b040a940f5631018da6e9.svg
wang meng
user
meng-03
668df563b09a05f3d7e46c97
Yongliang Shen
claimed_verified
2025-05-20T07:24:10.791Z
5e1058e9fcf41d740b69966d
/avatars/ce74839ba871f2b54313a670a233ba82.svg
Yongliang Shen
user
tricktreat
668df563b09a05f3d7e46c98
Zeqi Tan
668df563b09a05f3d7e46c99
Guiyang Hou
668df563b09a05f3d7e46c9a
Mingqian He
668df563b09a05f3d7e46c9b
Yanna Ma
668df563b09a05f3d7e46c9c
Weiming Lu
668df563b09a05f3d7e46c9d
Yueting Zhuang
Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning
  Instruction Using Language Model
Although most current large multimodal models (LMMs) can already understand
photos of natural scenes and portraits, their understanding of abstract images,
e.g., charts, maps, or layouts, and visual reasoning capabilities remains quite
rudimentary. They often struggle with simple daily tasks, such as reading time
from a clock, understanding a flowchart, or planning a route using a road map.
In light of this, we design a multi-modal self-instruct, utilizing large
language models and their code capabilities to synthesize massive abstract
images and visual reasoning instructions across daily scenarios. Our strategy
effortlessly creates a multimodal benchmark with 11,193 instructions for eight
visual scenarios: charts, tables, simulated maps, dashboards, flowcharts,
relation graphs, floor plans, and visual puzzles. This benchmark,
constructed with simple lines and geometric elements, exposes the shortcomings
of most advanced LMMs like Claude-3.5-Sonnet and GPT-4o in abstract image
understanding, spatial relations reasoning, and visual element induction.
Besides, to verify the quality of our synthetic data, we fine-tune an LMM using
62,476 synthetic chart, table and road map instructions. The results
demonstrate improved chart understanding and map navigation performance, and
also demonstrate potential benefits for other visual reasoning tasks. Our code
is available at: https://github.com/zwq2018/Multi-modal-Self-instruct.
A multi-modal self-instruct benchmark using large language models and code capabilities enhances understanding and reasoning for abstract images and visual tasks, improving performance metrics like chart understanding and map navigation.
large multimodal models self-instruct large language models synthetic abstract images visual reasoning instructions multimodal benchmark visual scenarios charts tables simulated maps dashboards flowcharts relation graphs floor plans visual puzzles spatial relations reasoning visual element induction fine-tuning chart understanding map navigation