title: Scaling Text-Rich Image Understanding via Code-Guided Synthetic
    Multimodal Data Generation
summary: Reasoning about images with rich text, such as charts and documents, is a
  critical application of vision-language models (VLMs). However, VLMs often
  struggle in these domains due to the scarcity of diverse text-rich
  vision-language data. To address this challenge, we present CoSyn, a framework
  that leverages the coding capabilities of text-only large language models
  (LLMs) to automatically create synthetic text-rich multimodal data. Given input
  text describing a target domain (e.g., "nutrition fact labels"), CoSyn prompts
  an LLM to generate code (Python, HTML, LaTeX, etc.) for rendering synthetic
  images. With the underlying code as textual representations of the synthetic
  images, CoSyn can generate high-quality instruction-tuning data, again relying
  on a text-only LLM. Using CoSyn, we constructed a dataset comprising 400K
  images and 2.7M rows of vision-language instruction-tuning data. Comprehensive
  experiments on seven benchmarks demonstrate that models trained on our
  synthetic data achieve state-of-the-art performance among competitive
  open-source models, including Llama 3.2, and surpass proprietary models such as
  GPT-4V and Gemini 1.5 Flash. Furthermore, CoSyn can produce synthetic pointing
  data, enabling VLMs to ground information within input images, showcasing its
  potential for developing multimodal agents capable of acting in real-world
  environments.
paper_authors._id: 67b7f4f1b15c19d57189fc5e
paper_authors.name: Yue Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T15:06:20.097Z
paper_authors._id: 62f6c68904e5e02f82b04690
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f6c68904e5e02f82b04690/kK2-PkeAGzAOLhkfajswf.jpeg
paper_authors.fullname: Yue Yang
paper_authors.type: user
paper_authors.user: yyupenn
paper_authors._id: 67b7f4f1b15c19d57189fc5f
paper_authors.name: Ajay Patel
paper_authors._id: 67b7f4f1b15c19d57189fc60
paper_authors.name: Matt Deitke
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:03:20.545Z
paper_authors._id: 61c388aa727d1257bf3cf58b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1670871898141-61c388aa727d1257bf3cf58b.jpeg
paper_authors.fullname: Matt Deitke
paper_authors.type: user
paper_authors.user: mattdeitke
paper_authors._id: 67b7f4f1b15c19d57189fc61
paper_authors.name: Tanmay Gupta
paper_authors._id: 67b7f4f1b15c19d57189fc62
paper_authors.name: Luca Weihs
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:02:58.922Z
paper_authors._id: 620acbdb3c0931626a7c9297
paper_authors.avatarUrl: /avatars/f63b1d225ed81e223d3e8876a5c708c4.svg
paper_authors.fullname: Luca Weihs
paper_authors.type: user
paper_authors.user: lucaweihs
paper_authors._id: 67b7f4f1b15c19d57189fc63
paper_authors.name: Andrew Head
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:02:53.395Z
paper_authors._id: 6360289588e41d249ecd3e26
paper_authors.avatarUrl: /avatars/0e84dbd72f07e99967a8d25cda938efe.svg
paper_authors.fullname: Andrew Head
paper_authors.type: user
paper_authors.user: ChittyChins
paper_authors._id: 67b7f4f1b15c19d57189fc64
paper_authors.name: Mark Yatskar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:02:47.575Z
paper_authors._id: 631f42e1b6628770f6efd87a
paper_authors.avatarUrl: /avatars/0fa93b3513ebca737cce26dfa5611cf1.svg
paper_authors.fullname: Mark Yatskar
paper_authors.type: user
paper_authors.user: myatskar
paper_authors._id: 67b7f4f1b15c19d57189fc65
paper_authors.name: Chris Callison-Burch
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:02:41.757Z
paper_authors._id: 6303ce25fc783bfc744216af
paper_authors.avatarUrl: /avatars/09f5e87c1f56a1b7f6ef9c5037682285.svg
paper_authors.fullname: Chris Callison-Burch
paper_authors.type: user
paper_authors.user: CCB
paper_authors._id: 67b7f4f1b15c19d57189fc66
paper_authors.name: Ranjay Krishna
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:02:35.267Z
paper_authors._id: 66429868ab89e3a3a85668b0
paper_authors.avatarUrl: /avatars/170e0daa454838deee2bf946f7118651.svg
paper_authors.fullname: Ranjay Krishna
paper_authors.type: user
paper_authors.user: ranjaykrishna
paper_authors._id: 67b7f4f1b15c19d57189fc67
paper_authors.name: Aniruddha Kembhavi
paper_authors._id: 67b7f4f1b15c19d57189fc68
paper_authors.name: Christopher Clark
paper_title: Scaling Text-Rich Image Understanding via Code-Guided Synthetic
    Multimodal Data Generation
paper_summary: Reasoning about images with rich text, such as charts and documents, is a
  critical application of vision-language models (VLMs). However, VLMs often
  struggle in these domains due to the scarcity of diverse text-rich
  vision-language data. To address this challenge, we present CoSyn, a framework
  that leverages the coding capabilities of text-only large language models
  (LLMs) to automatically create synthetic text-rich multimodal data. Given input
  text describing a target domain (e.g., "nutrition fact labels"), CoSyn prompts
  an LLM to generate code (Python, HTML, LaTeX, etc.) for rendering synthetic
  images. With the underlying code as textual representations of the synthetic
  images, CoSyn can generate high-quality instruction-tuning data, again relying
  on a text-only LLM. Using CoSyn, we constructed a dataset comprising 400K
  images and 2.7M rows of vision-language instruction-tuning data. Comprehensive
  experiments on seven benchmarks demonstrate that models trained on our
  synthetic data achieve state-of-the-art performance among competitive
  open-source models, including Llama 3.2, and surpass proprietary models such as
  GPT-4V and Gemini 1.5 Flash. Furthermore, CoSyn can produce synthetic pointing
  data, enabling VLMs to ground information within input images, showcasing its
  potential for developing multimodal agents capable of acting in real-world
  environments.
paper_ai_keywords: vision-language models (VLMs) large language models (LLMs) synthetic text-rich multimodal data instruction-tuning data synthetic images multimodal agents