title: SketchAgent: Language-Driven Sequential Sketch Generation
summary: Sketching serves as a versatile tool for externalizing ideas, enabling rapid
  exploration and visual communication that spans various disciplines. While
  artificial systems have driven substantial advances in content creation and
  human-computer interaction, capturing the dynamic and abstract nature of human
  sketching remains challenging. In this work, we introduce SketchAgent, a
  language-driven, sequential sketch generation method that enables users to
  create, modify, and refine sketches through dynamic, conversational
  interactions. Our approach requires no training or fine-tuning. Instead, we
  leverage the sequential nature and rich prior knowledge of off-the-shelf
  multimodal large language models (LLMs). We present an intuitive sketching
  language, introduced to the model through in-context examples, enabling it to
  "draw" using string-based actions. These are processed into vector graphics and
  then rendered to create a sketch on a pixel canvas, which can be accessed again
  for further tasks. By drawing stroke by stroke, our agent captures the
  evolving, dynamic qualities intrinsic to sketching. We demonstrate that
  SketchAgent can generate sketches from diverse prompts, engage in
  dialogue-driven drawing, and collaborate meaningfully with human users.
paper_authors._id: 6746935b7f9c1728d230630d
paper_authors.name: Yael Vinker
paper_authors._id: 6746935b7f9c1728d230630e
paper_authors.name: Tamar Rott Shaham
paper_authors._id: 6746935b7f9c1728d230630f
paper_authors.name: Kristine Zheng
paper_authors._id: 6746935b7f9c1728d2306310
paper_authors.name: Alex Zhao
paper_authors._id: 6746935b7f9c1728d2306311
paper_authors.name: Judith E Fan
paper_authors._id: 6746935b7f9c1728d2306312
paper_authors.name: Antonio Torralba
paper_title: SketchAgent: Language-Driven Sequential Sketch Generation
paper_summary: Sketching serves as a versatile tool for externalizing ideas, enabling rapid
  exploration and visual communication that spans various disciplines. While
  artificial systems have driven substantial advances in content creation and
  human-computer interaction, capturing the dynamic and abstract nature of human
  sketching remains challenging. In this work, we introduce SketchAgent, a
  language-driven, sequential sketch generation method that enables users to
  create, modify, and refine sketches through dynamic, conversational
  interactions. Our approach requires no training or fine-tuning. Instead, we
  leverage the sequential nature and rich prior knowledge of off-the-shelf
  multimodal large language models (LLMs). We present an intuitive sketching
  language, introduced to the model through in-context examples, enabling it to
  "draw" using string-based actions. These are processed into vector graphics and
  then rendered to create a sketch on a pixel canvas, which can be accessed again
  for further tasks. By drawing stroke by stroke, our agent captures the
  evolving, dynamic qualities intrinsic to sketching. We demonstrate that
  SketchAgent can generate sketches from diverse prompts, engage in
  dialogue-driven drawing, and collaborate meaningfully with human users.
paper_ai_summary: SketchAgent, a language-driven method, generates and refines sketches through dynamic conversations using off-the-shelf LLMs and string-based actions processed into vector graphics.
paper_ai_keywords: SketchAgent language-driven sequential sketch generation multimodal large language models sketching language vector graphics dialogue-driven drawing