title: LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image
    Understanding
summary: Instruction tuning unlocks the superior capability of Large Language Models
  (LLM) to interact with humans. Furthermore, recent instruction-following
  datasets include images as visual inputs, collecting responses for image-based
  instructions. However, visual instruction-tuned models cannot comprehend
  textual details within images well. This work enhances the current visual
  instruction tuning pipeline with text-rich images (e.g., movie posters, book
  covers, etc.). Specifically, we first use publicly available OCR tools to
  collect results on 422K text-rich images from the LAION dataset. Moreover, we
  prompt text-only GPT-4 with recognized texts and image captions to generate 16K
  conversations, each containing question-answer pairs for text-rich images. By
  combining our collected data with previous multi-modal instruction-following
  data, our model, LLaVAR, substantially improves the LLaVA model's capability on
  text-based VQA datasets (up to 20% accuracy improvement) while achieving an
  accuracy of 91.42% on ScienceQA. The GPT-4-based instruction-following
  evaluation also demonstrates the improvement of our model on both natural
  images and text-rich images. Through qualitative analysis, LLaVAR shows
  promising interaction (e.g., reasoning, writing, and elaboration) skills with
  humans based on the latest real-world online content that combines text and
  images. We make our code/data/models publicly available at
  https://llavar.github.io/.
paper_authors._id: 649e2c8500963003b7d23ea6
paper_authors.name: Yanzhe Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-30T19:46:05.998Z
paper_authors._id: 61362307df63bbd1c2459c4c
paper_authors.avatarUrl: /avatars/28d3ba4b0fa00acb1eed623ab3620789.svg
paper_authors.fullname: Yanzhe Zhang
paper_authors.type: user
paper_authors.user: zyanzhe
paper_authors._id: 649e2c8500963003b7d23ea7
paper_authors.name: Ruiyi Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-03T09:46:26.954Z
paper_authors._id: 64668f982da1abc242355cbb
paper_authors.avatarUrl: /avatars/c9a248b8d70b1a8f7b78265c98690570.svg
paper_authors.fullname: Ruiyi Zhang
paper_authors.type: user
paper_authors.user: zhangry868
paper_authors._id: 649e2c8500963003b7d23ea8
paper_authors.name: Jiuxiang Gu
paper_authors._id: 649e2c8500963003b7d23ea9
paper_authors.name: Yufan Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-30T19:48:11.738Z
paper_authors._id: 635c2c2a7a165601151d3f85
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666984965757-noauth.png
paper_authors.fullname: Yufan Zhou
paper_authors.type: user
paper_authors.user: YfZ
paper_authors._id: 649e2c8500963003b7d23eaa
paper_authors.name: Nedim Lipka
paper_authors._id: 649e2c8500963003b7d23eab
paper_authors.name: Diyi Yang
paper_authors._id: 649e2c8500963003b7d23eac
paper_authors.name: Tong Sun
paper_title: LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image
    Understanding
paper_summary: Instruction tuning unlocks the superior capability of Large Language Models
  (LLM) to interact with humans. Furthermore, recent instruction-following
  datasets include images as visual inputs, collecting responses for image-based
  instructions. However, visual instruction-tuned models cannot comprehend
  textual details within images well. This work enhances the current visual
  instruction tuning pipeline with text-rich images (e.g., movie posters, book
  covers, etc.). Specifically, we first use publicly available OCR tools to
  collect results on 422K text-rich images from the LAION dataset. Moreover, we
  prompt text-only GPT-4 with recognized texts and image captions to generate 16K
  conversations, each containing question-answer pairs for text-rich images. By
  combining our collected data with previous multi-modal instruction-following
  data, our model, LLaVAR, substantially improves the LLaVA model's capability on
  text-based VQA datasets (up to 20% accuracy improvement) while achieving an
  accuracy of 91.42% on ScienceQA. The GPT-4-based instruction-following
  evaluation also demonstrates the improvement of our model on both natural
  images and text-rich images. Through qualitative analysis, LLaVAR shows
  promising interaction (e.g., reasoning, writing, and elaboration) skills with
  humans based on the latest real-world online content that combines text and
  images. We make our code/data/models publicly available at
  https://llavar.github.io/.
paper_ai_summary: LLaVAR enhances visual instruction-tuned models by incorporating text recognition from images, resulting in improved performance on text-based VQA datasets and better interaction capabilities.
paper_ai_keywords: instruction tuning Large Language Models (LLM) image-based instructions OCR LAION dataset GPT-4 text-only GPT-4 image captions multi-modal instruction-following LLaVAR LLaVA text-based VQA datasets ScienceQA qualitative analysis