title: Whiteboard-of-Thought: Thinking Step-by-Step Across Modalities
summary: When presented with questions involving visual thinking, humans naturally
  switch reasoning modalities, often forming mental images or drawing visual
  aids. Large language models have shown promising results in arithmetic and
  symbolic reasoning by expressing intermediate reasoning in text as a chain of
  thought, yet struggle to extend this capability to answer text queries that are
  easily solved by visual reasoning, even with extensive multimodal pretraining.
  We introduce a simple method, whiteboard-of-thought prompting, to unlock the
  visual reasoning capabilities of multimodal large language models across
  modalities. Whiteboard-of-thought prompting provides multimodal large language
  models with a metaphorical `whiteboard' to draw out reasoning steps as images,
  then returns these images back to the model for further processing. We find
  this can be accomplished with no demonstrations or specialized modules, instead
  leveraging models' existing ability to write code with libraries such as
  Matplotlib and Turtle. This simple approach shows state-of-the-art results on
  four difficult natural language tasks that involve visual and spatial
  reasoning. We identify multiple settings where GPT-4o using chain-of-thought
  fails dramatically, including more than one where it achieves 0% accuracy,
  while whiteboard-of-thought enables up to 92% accuracy in these same
  settings. We present a detailed exploration of where the technique succeeds as
  well as its sources of error.
paper_authors._id: 667517fdd229a7f83090560e
paper_authors.name: Sachit Menon
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-21T06:28:17.611Z
paper_authors._id: 62fbbfaa7a577e83ccccbffb
paper_authors.avatarUrl: /avatars/f4e8cf1e9145a1df39082544eb709264.svg
paper_authors.fullname: Sachit Menon
paper_authors.type: user
paper_authors.user: sachit-menon
paper_authors._id: 667517fdd229a7f83090560f
paper_authors.name: Richard Zemel
paper_authors._id: 667517fdd229a7f830905610
paper_authors.name: Carl Vondrick
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T11:40:33.375Z
paper_authors._id: 63a4bfcbcde2b28f82a2c50c
paper_authors.avatarUrl: /avatars/9d978a6f0caf6ad92ec4138a84a39e53.svg
paper_authors.fullname: Carl Vondrick
paper_authors.type: user
paper_authors.user: cvondrick
paper_title: Whiteboard-of-Thought: Thinking Step-by-Step Across Modalities
paper_summary: When presented with questions involving visual thinking, humans naturally
  switch reasoning modalities, often forming mental images or drawing visual
  aids. Large language models have shown promising results in arithmetic and
  symbolic reasoning by expressing intermediate reasoning in text as a chain of
  thought, yet struggle to extend this capability to answer text queries that are
  easily solved by visual reasoning, even with extensive multimodal pretraining.
  We introduce a simple method, whiteboard-of-thought prompting, to unlock the
  visual reasoning capabilities of multimodal large language models across
  modalities. Whiteboard-of-thought prompting provides multimodal large language
  models with a metaphorical `whiteboard' to draw out reasoning steps as images,
  then returns these images back to the model for further processing. We find
  this can be accomplished with no demonstrations or specialized modules, instead
  leveraging models' existing ability to write code with libraries such as
  Matplotlib and Turtle. This simple approach shows state-of-the-art results on
  four difficult natural language tasks that involve visual and spatial
  reasoning. We identify multiple settings where GPT-4o using chain-of-thought
  fails dramatically, including more than one where it achieves 0% accuracy,
  while whiteboard-of-thought enables up to 92% accuracy in these same
  settings. We present a detailed exploration of where the technique succeeds as
  well as its sources of error.
paper_ai_summary: A novel method, whiteboard-of-thought prompting, enhances the visual reasoning capabilities of large language models by leveraging their ability to generate images using coding libraries, achieving superior accuracy on spatial and visual reasoning tasks compared to traditional chain-of-thought methods.
paper_ai_keywords: whiteboard-of-thought prompting multimodal large language models visual reasoning spatial reasoning chain-of-thought Matplotlib Turtle