Whiteboard-of-Thought: Thinking Step-by-Step Across Modalities
When presented with questions involving visual thinking, humans naturally
switch reasoning modalities, often forming mental images or drawing visual
aids. Large language models have shown promising results in arithmetic and
symbolic reasoning by expressing intermediate reasoning in text as a chain of
thought, yet struggle to extend this capability to answer text queries that are
easily solved by visual reasoning, even with extensive multimodal pretraining.
We introduce a simple method, whiteboard-of-thought prompting, to unlock the
visual reasoning capabilities of multimodal large language models across
modalities. Whiteboard-of-thought prompting provides multimodal large language
models with a metaphorical `whiteboard' to draw out reasoning steps as images,
then returns these images back to the model for further processing. We find
this can be accomplished with no demonstrations or specialized modules, instead
leveraging models' existing ability to write code with libraries such as
Matplotlib and Turtle. This simple approach shows state-of-the-art results on
four difficult natural language tasks that involve visual and spatial
reasoning. We identify multiple settings where GPT-4o using chain-of-thought
fails dramatically, including more than one where it achieves 0% accuracy,
while whiteboard-of-thought enables up to 92% accuracy in these same
settings. We present a detailed exploration of where the technique succeeds as
well as its sources of error.
667517fdd229a7f83090560e
Sachit Menon
extracted_confirmed
2024-06-21T06:28:17.611Z
62fbbfaa7a577e83ccccbffb
/avatars/f4e8cf1e9145a1df39082544eb709264.svg
Sachit Menon
user
sachit-menon
667517fdd229a7f83090560f
Richard Zemel
667517fdd229a7f830905610
Carl Vondrick
admin_assigned
2024-06-21T11:40:33.375Z
63a4bfcbcde2b28f82a2c50c
/avatars/9d978a6f0caf6ad92ec4138a84a39e53.svg
Carl Vondrick
user
cvondrick
Whiteboard-of-Thought: Thinking Step-by-Step Across Modalities
When presented with questions involving visual thinking, humans naturally
switch reasoning modalities, often forming mental images or drawing visual
aids. Large language models have shown promising results in arithmetic and
symbolic reasoning by expressing intermediate reasoning in text as a chain of
thought, yet struggle to extend this capability to answer text queries that are
easily solved by visual reasoning, even with extensive multimodal pretraining.
We introduce a simple method, whiteboard-of-thought prompting, to unlock the
visual reasoning capabilities of multimodal large language models across
modalities. Whiteboard-of-thought prompting provides multimodal large language
models with a metaphorical `whiteboard' to draw out reasoning steps as images,
then returns these images back to the model for further processing. We find
this can be accomplished with no demonstrations or specialized modules, instead
leveraging models' existing ability to write code with libraries such as
Matplotlib and Turtle. This simple approach shows state-of-the-art results on
four difficult natural language tasks that involve visual and spatial
reasoning. We identify multiple settings where GPT-4o using chain-of-thought
fails dramatically, including more than one where it achieves 0% accuracy,
while whiteboard-of-thought enables up to 92% accuracy in these same
settings. We present a detailed exploration of where the technique succeeds as
well as its sources of error.
A novel method, whiteboard-of-thought prompting, enhances the visual reasoning capabilities of large language models by leveraging their ability to generate images using coding libraries, achieving superior accuracy on spatial and visual reasoning tasks compared to traditional chain-of-thought methods.
whiteboard-of-thought prompting multimodal large language models visual reasoning spatial reasoning chain-of-thought Matplotlib Turtle