Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning
Humans often use visual aids, for example diagrams or sketches, when solving
complex problems. Training multimodal models to do the same, known as Visual
Chain of Thought (Visual CoT), is challenging due to: (1) poor off-the-shelf
visual CoT performance, which hinders reinforcement learning, and (2) the lack
of high-quality visual CoT training data. We introduce Zebra-CoT, a
diverse large-scale dataset with 182,384 samples, containing logically coherent
interleaved text-image reasoning traces. We focus on four categories of tasks
where sketching or visual reasoning is especially natural, spanning scientific
questions such as geometry, physics, and algorithms; 2D visual reasoning tasks
like visual search and jigsaw puzzles; 3D reasoning tasks including 3D
multi-hop inference, embodied and robot planning; visual logic problems and
strategic games like chess. Fine-tuning the Anole-7B model on the Zebra-CoT
training corpus results in an improvement of +12% in our test-set accuracy and
yields up to +13% performance gain on standard VLM benchmark evaluations.
Fine-tuning Bagel-7B yields a model that generates high-quality interleaved
visual reasoning chains, underscoring Zebra-CoT's effectiveness for developing
multimodal reasoning abilities. We open-source our dataset and models to
support development and evaluation of visual CoT.
68803f5300f4b5a05f2fbdba
Ang Li
claimed_verified
2025-07-23T08:36:42.364Z
64a1b18b98fad0c8a5b04e3a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/8EvZSqz_GumooxK0uHkyR.png
Leon Li
user
leonli66
68803f5300f4b5a05f2fbdbb
Charles Wang
claimed_verified
2025-07-23T08:36:40.606Z
67c604ff3f8c7b027d479671
https://cdn-avatars.huggingface.co/v1/production/uploads/67c604ff3f8c7b027d479671/IhfN3eWRQ8iNfcsDKA1Ep.png
Charles Wang
user
charleslwang
68803f5300f4b5a05f2fbdbc
Kaiyu Yue
claimed_verified
2025-07-23T13:30:52.346Z
640d0dbc8036cc2142273a83
https://cdn-avatars.huggingface.co/v1/production/uploads/640d0dbc8036cc2142273a83/L1KJwMGYIAmDLaUeSkPnb.jpeg
Kaiyu Yue
user
kaiyuyue
68803f5300f4b5a05f2fbdbd
Zikui Cai
68803f5300f4b5a05f2fbdbe
Ollie Liu
68803f5300f4b5a05f2fbdbf
Deqing Fu
claimed_verified
2025-07-23T08:36:45.702Z
63c8454e46421a2efe82709d
https://cdn-avatars.huggingface.co/v1/production/uploads/63c8454e46421a2efe82709d/3BcSk4KOwAgWHEPVtsAV3.png
Deqing Fu
user
deqing
68803f5300f4b5a05f2fbdc0
Peng Guo
68803f5300f4b5a05f2fbdc1
Wang Bill Zhu
68803f5300f4b5a05f2fbdc2
Vatsal Sharan
68803f5300f4b5a05f2fbdc3
Robin Jia
68803f5300f4b5a05f2fbdc4
Willie Neiswanger
68803f5300f4b5a05f2fbdc5
Furong Huang
68803f5300f4b5a05f2fbdc6
Tom Goldstein
68803f5300f4b5a05f2fbdc7
Micah Goldblum
Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning
Humans often use visual aids, for example diagrams or sketches, when solving
complex problems. Training multimodal models to do the same, known as Visual
Chain of Thought (Visual CoT), is challenging due to: (1) poor off-the-shelf
visual CoT performance, which hinders reinforcement learning, and (2) the lack
of high-quality visual CoT training data. We introduce Zebra-CoT, a
diverse large-scale dataset with 182,384 samples, containing logically coherent
interleaved text-image reasoning traces. We focus on four categories of tasks
where sketching or visual reasoning is especially natural, spanning scientific
questions such as geometry, physics, and algorithms; 2D visual reasoning tasks
like visual search and jigsaw puzzles; 3D reasoning tasks including 3D
multi-hop inference, embodied and robot planning; visual logic problems and
strategic games like chess. Fine-tuning the Anole-7B model on the Zebra-CoT
training corpus results in an improvement of +12% in our test-set accuracy and
yields up to +13% performance gain on standard VLM benchmark evaluations.
Fine-tuning Bagel-7B yields a model that generates high-quality interleaved
visual reasoning chains, underscoring Zebra-CoT's effectiveness for developing
multimodal reasoning abilities. We open-source our dataset and models to
support development and evaluation of visual CoT.
Zebra-CoT, a large-scale dataset with diverse visual and text reasoning tasks, improves multimodal model performance through fine-tuning, enhancing accuracy and visual chain of thought generation.
Visual Chain of Thought Visual CoT Anole-7B Bagel-7B multimodal models text-image reasoning scientific questions geometry physics algorithms 2D visual reasoning visual search jigsaw puzzles 3D reasoning 3D multi-hop inference embodied planning robot planning visual logic problems strategic games chess VLM benchmark