Scaling Text-Rich Image Understanding via Code-Guided Synthetic
  Multimodal Data Generation
Reasoning about images with rich text, such as charts and documents, is a
critical application of vision-language models (VLMs). However, VLMs often
struggle in these domains due to the scarcity of diverse text-rich
vision-language data. To address this challenge, we present CoSyn, a framework
that leverages the coding capabilities of text-only large language models
(LLMs) to automatically create synthetic text-rich multimodal data. Given input
text describing a target domain (e.g., "nutrition fact labels"), CoSyn prompts
an LLM to generate code (Python, HTML, LaTeX, etc.) for rendering synthetic
images. With the underlying code as textual representations of the synthetic
images, CoSyn can generate high-quality instruction-tuning data, again relying
on a text-only LLM. Using CoSyn, we constructed a dataset comprising 400K
images and 2.7M rows of vision-language instruction-tuning data. Comprehensive
experiments on seven benchmarks demonstrate that models trained on our
synthetic data achieve state-of-the-art performance among competitive
open-source models, including Llama 3.2, and surpass proprietary models such as
GPT-4V and Gemini 1.5 Flash. Furthermore, CoSyn can produce synthetic pointing
data, enabling VLMs to ground information within input images, showcasing its
potential for developing multimodal agents capable of acting in real-world
environments.
67b7f4f1b15c19d57189fc5e
Yue Yang
claimed_verified
2025-02-21T15:06:20.097Z
62f6c68904e5e02f82b04690
https://cdn-avatars.huggingface.co/v1/production/uploads/62f6c68904e5e02f82b04690/kK2-PkeAGzAOLhkfajswf.jpeg
Yue Yang
user
yyupenn
67b7f4f1b15c19d57189fc5f
Ajay Patel
67b7f4f1b15c19d57189fc60
Matt Deitke
admin_assigned
2025-02-21T15:03:20.545Z
61c388aa727d1257bf3cf58b
https://cdn-avatars.huggingface.co/v1/production/uploads/1670871898141-61c388aa727d1257bf3cf58b.jpeg
Matt Deitke
user
mattdeitke
67b7f4f1b15c19d57189fc61
Tanmay Gupta
claimed_verified
2025-06-10T09:31:13.963Z
62a108242e30aaf94ec283bb
/avatars/4da19ae99df6a3fa2d81336eb59cdaa7.svg
Tanmay Gupta
user
tanmayg
67b7f4f1b15c19d57189fc62
Luca Weihs
admin_assigned
2025-02-21T15:02:58.922Z
620acbdb3c0931626a7c9297
/avatars/f63b1d225ed81e223d3e8876a5c708c4.svg
Luca Weihs
user
lucaweihs
67b7f4f1b15c19d57189fc63
Andrew Head
admin_assigned
2025-02-21T15:02:53.395Z
6360289588e41d249ecd3e26
/avatars/0e84dbd72f07e99967a8d25cda938efe.svg
Andrew Head
user
ChittyChins
67b7f4f1b15c19d57189fc64
Mark Yatskar
admin_assigned
2025-02-21T15:02:47.575Z
631f42e1b6628770f6efd87a
/avatars/0fa93b3513ebca737cce26dfa5611cf1.svg
Mark Yatskar
user
myatskar
67b7f4f1b15c19d57189fc65
Chris Callison-Burch
admin_assigned
2025-02-21T15:02:41.757Z
6303ce25fc783bfc744216af
/avatars/09f5e87c1f56a1b7f6ef9c5037682285.svg
Chris Callison-Burch
user
CCB
67b7f4f1b15c19d57189fc66
Ranjay Krishna
admin_assigned
2025-02-21T15:02:35.267Z
66429868ab89e3a3a85668b0
/avatars/170e0daa454838deee2bf946f7118651.svg
Ranjay Krishna
user
ranjaykrishna
67b7f4f1b15c19d57189fc67
Aniruddha Kembhavi
67b7f4f1b15c19d57189fc68
Christopher Clark
Scaling Text-Rich Image Understanding via Code-Guided Synthetic
  Multimodal Data Generation
Reasoning about images with rich text, such as charts and documents, is a
critical application of vision-language models (VLMs). However, VLMs often
struggle in these domains due to the scarcity of diverse text-rich
vision-language data. To address this challenge, we present CoSyn, a framework
that leverages the coding capabilities of text-only large language models
(LLMs) to automatically create synthetic text-rich multimodal data. Given input
text describing a target domain (e.g., "nutrition fact labels"), CoSyn prompts
an LLM to generate code (Python, HTML, LaTeX, etc.) for rendering synthetic
images. With the underlying code as textual representations of the synthetic
images, CoSyn can generate high-quality instruction-tuning data, again relying
on a text-only LLM. Using CoSyn, we constructed a dataset comprising 400K
images and 2.7M rows of vision-language instruction-tuning data. Comprehensive
experiments on seven benchmarks demonstrate that models trained on our
synthetic data achieve state-of-the-art performance among competitive
open-source models, including Llama 3.2, and surpass proprietary models such as
GPT-4V and Gemini 1.5 Flash. Furthermore, CoSyn can produce synthetic pointing
data, enabling VLMs to ground information within input images, showcasing its
potential for developing multimodal agents capable of acting in real-world
environments.
CoSyn uses text-only LLMs to generate synthetic text-rich multimodal data, achieving state-of-the-art performance on VLM benchmarks and enabling synthetic pointing data for grounding information within images.
vision-language models LLMs synthetic text-rich multimodal data instruction-tuning data multimodal agents