title: Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal
    Language Models
summary: Humans draw to facilitate reasoning: we draw auxiliary lines when solving
  geometry problems; we mark and circle when reasoning on maps; we use sketches
  to amplify our ideas and relieve our limited-capacity working memory. However,
  such actions are missing in current multimodal language models (LMs). Current
  chain-of-thought and tool-use paradigms only use text as intermediate reasoning
  steps. In this work, we introduce Sketchpad, a framework that gives multimodal
  LMs a visual sketchpad and tools to draw on the sketchpad. The LM conducts
  planning and reasoning according to the visual artifacts it has drawn.
  Different from prior work, which uses text-to-image models to enable LMs to
  draw, Sketchpad enables LMs to draw with lines, boxes, marks, etc., which is
  closer to human sketching and better facilitates reasoning. Sketchpad can also
  use specialist vision models during the sketching process (e.g., draw bounding
  boxes with object detection models, draw masks with segmentation models), to
  further enhance visual perception and reasoning. We experiment with a wide
  range of math tasks (including geometry, functions, graphs, and chess) and
  complex visual reasoning tasks. Sketchpad substantially improves performance on
  all tasks over strong base models with no sketching, yielding an average gain
  of 12.7% on math tasks, and 8.6% on vision tasks. GPT-4o with Sketchpad sets a
  new state of the art on all tasks, including V*Bench (80.3%), BLINK spatial
  reasoning (83.9%), and visual correspondence (80.8%). All codes and data are in
  https://visualsketchpad.github.io/.
paper_authors._id: 666bb62ed0e6a329a3e46a1d
paper_authors.name: Yushi Hu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-06-14T03:17:09.471Z
paper_authors._id: 62b1474bdcbad6848a91a54e
paper_authors.avatarUrl: /avatars/d7308899b46232cad4a48a0e876449a8.svg
paper_authors.fullname: Yushi Hu
paper_authors.type: user
paper_authors.user: yushihu
paper_authors._id: 666bb62ed0e6a329a3e46a1e
paper_authors.name: Weijia Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T10:06:58.721Z
paper_authors._id: 6400f2ed568dbe30c9161e47
paper_authors.avatarUrl: /avatars/c55938df5bce82b5d96e592a1ec36a8b.svg
paper_authors.fullname: Weijia Shi
paper_authors.type: user
paper_authors.user: swj0419
paper_authors._id: 666bb62ed0e6a329a3e46a1f
paper_authors.name: Xingyu Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T10:07:13.871Z
paper_authors._id: 6336091b2db86a181ccd6054
paper_authors.avatarUrl: /avatars/4d29cef783603dc8ba123f01cde93ad8.svg
paper_authors.fullname: Xingyu Fu
paper_authors.type: user
paper_authors.user: Fiaa
paper_authors._id: 666bb62ed0e6a329a3e46a20
paper_authors.name: Dan Roth
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T10:07:21.248Z
paper_authors._id: 63191aa1f18d0b5d999a47fc
paper_authors.avatarUrl: /avatars/d7eadb22aa3d8a067189b924767864e9.svg
paper_authors.fullname: Dan Roth
paper_authors.type: user
paper_authors.user: danielpaulroth
paper_authors._id: 666bb62ed0e6a329a3e46a21
paper_authors.name: Mari Ostendorf
paper_authors._id: 666bb62ed0e6a329a3e46a22
paper_authors.name: Luke Zettlemoyer
paper_authors._id: 666bb62ed0e6a329a3e46a23
paper_authors.name: Noah A Smith
paper_authors._id: 666bb62ed0e6a329a3e46a24
paper_authors.name: Ranjay Krishna
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T10:07:42.545Z
paper_authors._id: 66429868ab89e3a3a85668b0
paper_authors.avatarUrl: /avatars/170e0daa454838deee2bf946f7118651.svg
paper_authors.fullname: Ranjay Krishna
paper_authors.type: user
paper_authors.user: ranjaykrishna
paper_title: Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal
    Language Models
paper_summary: Humans draw to facilitate reasoning: we draw auxiliary lines when solving
  geometry problems; we mark and circle when reasoning on maps; we use sketches
  to amplify our ideas and relieve our limited-capacity working memory. However,
  such actions are missing in current multimodal language models (LMs). Current
  chain-of-thought and tool-use paradigms only use text as intermediate reasoning
  steps. In this work, we introduce Sketchpad, a framework that gives multimodal
  LMs a visual sketchpad and tools to draw on the sketchpad. The LM conducts
  planning and reasoning according to the visual artifacts it has drawn.
  Different from prior work, which uses text-to-image models to enable LMs to
  draw, Sketchpad enables LMs to draw with lines, boxes, marks, etc., which is
  closer to human sketching and better facilitates reasoning. Sketchpad can also
  use specialist vision models during the sketching process (e.g., draw bounding
  boxes with object detection models, draw masks with segmentation models), to
  further enhance visual perception and reasoning. We experiment with a wide
  range of math tasks (including geometry, functions, graphs, and chess) and
  complex visual reasoning tasks. Sketchpad substantially improves performance on
  all tasks over strong base models with no sketching, yielding an average gain
  of 12.7% on math tasks, and 8.6% on vision tasks. GPT-4o with Sketchpad sets a
  new state of the art on all tasks, including V*Bench (80.3%), BLINK spatial
  reasoning (83.9%), and visual correspondence (80.8%). All codes and data are in
  https://visualsketchpad.github.io/.
paper_ai_keywords: Sketchpad multimodal language models chain-of-thought tool-use visual sketchpad visual artifacts text-to-image models bounding boxes object detection models masks segmentation models visual perception reasoning V*Bench BLINK spatial reasoning visual correspondence