title: TextSquare: Scaling up Text-Centric Visual Instruction Tuning
summary: Text-centric visual question answering (VQA) has made great strides with the
  development of Multimodal Large Language Models (MLLMs), yet open-source models
  still fall short of leading models like GPT4V and Gemini, partly due to a lack
  of extensive, high-quality instruction tuning data. To this end, we introduce a
  new approach for creating a massive, high-quality instruction-tuning dataset,
  Square-10M, which is generated using closed-source MLLMs. The data construction
  process, termed Square, consists of four steps: Self-Questioning, Answering,
  Reasoning, and Evaluation. Our experiments with Square-10M led to three key
  findings: 1) Our model, TextSquare, considerably surpasses open-source previous
  state-of-the-art Text-centric MLLMs and sets a new standard on OCRBench(62.2%).
  It even outperforms top-tier models like GPT4V and Gemini in 6 of 10
  text-centric benchmarks. 2) Additionally, we demonstrate the critical role of
  VQA reasoning data in offering comprehensive contextual insights for specific
  questions. This not only improves accuracy but also significantly mitigates
  hallucinations. Specifically, TextSquare scores an average of 75.1% across four
  general VQA and hallucination evaluation datasets, outperforming previous
  state-of-the-art models. 3) Notably, the phenomenon observed in scaling
  text-centric VQA datasets reveals a vivid pattern: the exponential increase of
  instruction tuning data volume is directly proportional to the improvement in
  model performance, thereby validating the necessity of the dataset scale and
  the high quality of Square-10M.
paper_authors._id: 6625f1688fee5ba324494021
paper_authors.name: Jingqun Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-22T08:05:44.330Z
paper_authors._id: 642bcc5ee8dfcc1fe4f4edea
paper_authors.avatarUrl: /avatars/0e9142a0d5f6341fb3d4dcd76455eb3f.svg
paper_authors.fullname: jingqun tang
paper_authors.type: user
paper_authors.user: jingqun
paper_authors._id: 6625f1688fee5ba324494022
paper_authors.name: Chunhui Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-22T08:17:45.582Z
paper_authors._id: 6363b6d561b008e03e7bfae4
paper_authors.avatarUrl: /avatars/1af24131896a6c2be9e81e2997a8e0f7.svg
paper_authors.fullname: Chunhui Lin
paper_authors.type: user
paper_authors.user: Chunhuizx
paper_authors._id: 6625f1688fee5ba324494023
paper_authors.name: Zhen Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-22T08:17:47.654Z
paper_authors._id: 64c0b6cea73ac49b2c129d42
paper_authors.avatarUrl: /avatars/4b2d993c0f176292ff032eec07889570.svg
paper_authors.fullname: zhaozhen
paper_authors.type: user
paper_authors.user: zhaozhen026
paper_authors._id: 6625f1688fee5ba324494024
paper_authors.name: Shu Wei
paper_authors._id: 6625f1688fee5ba324494025
paper_authors.name: Binghong Wu
paper_authors._id: 6625f1688fee5ba324494026
paper_authors.name: Qi Liu
paper_authors._id: 6625f1688fee5ba324494027
paper_authors.name: Hao Feng
paper_authors._id: 6625f1688fee5ba324494028
paper_authors.name: Yang Li
paper_authors._id: 6625f1688fee5ba324494029
paper_authors.name: Siqi Wang
paper_authors._id: 6625f1688fee5ba32449402a
paper_authors.name: Lei Liao
paper_authors._id: 6625f1688fee5ba32449402b
paper_authors.name: Wei Shi
paper_authors._id: 6625f1688fee5ba32449402c
paper_authors.name: Yuliang Liu
paper_authors._id: 6625f1688fee5ba32449402d
paper_authors.name: Hao Liu
paper_authors._id: 6625f1688fee5ba32449402e
paper_authors.name: Yuan Xie
paper_authors._id: 6625f1688fee5ba32449402f
paper_authors.name: Xiang Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-22T11:43:47.494Z
paper_authors._id: 641790e2f1e86908935d82a0
paper_authors.avatarUrl: /avatars/ced7a137c6344c74b7ac0d5c84833fc8.svg
paper_authors.fullname: Xiang Bai
paper_authors.type: user
paper_authors.user: baixianger
paper_authors._id: 6625f1688fee5ba324494030
paper_authors.name: Can Huang
paper_title: TextSquare: Scaling up Text-Centric Visual Instruction Tuning
paper_summary: Text-centric visual question answering (VQA) has made great strides with the
  development of Multimodal Large Language Models (MLLMs), yet open-source models
  still fall short of leading models like GPT4V and Gemini, partly due to a lack
  of extensive, high-quality instruction tuning data. To this end, we introduce a
  new approach for creating a massive, high-quality instruction-tuning dataset,
  Square-10M, which is generated using closed-source MLLMs. The data construction
  process, termed Square, consists of four steps: Self-Questioning, Answering,
  Reasoning, and Evaluation. Our experiments with Square-10M led to three key
  findings: 1) Our model, TextSquare, considerably surpasses open-source previous
  state-of-the-art Text-centric MLLMs and sets a new standard on OCRBench(62.2%).
  It even outperforms top-tier models like GPT4V and Gemini in 6 of 10
  text-centric benchmarks. 2) Additionally, we demonstrate the critical role of
  VQA reasoning data in offering comprehensive contextual insights for specific
  questions. This not only improves accuracy but also significantly mitigates
  hallucinations. Specifically, TextSquare scores an average of 75.1% across four
  general VQA and hallucination evaluation datasets, outperforming previous
  state-of-the-art models. 3) Notably, the phenomenon observed in scaling
  text-centric VQA datasets reveals a vivid pattern: the exponential increase of
  instruction tuning data volume is directly proportional to the improvement in
  model performance, thereby validating the necessity of the dataset scale and
  the high quality of Square-10M.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) vQ&A instruction tuning Square-10M Self-Questioning Answering Reasoning Evaluation VQA reasoning OCRBench GPT4V Gemini hallucinations VQA benchmarking dataset scaling