title: LLaVA-o1: Let Vision Language Models Reason Step-by-Step
summary: Large language models have demonstrated substantial advancements in reasoning
  capabilities, particularly through inference-time scaling, as illustrated by
  models such as OpenAI's o1. However, current Vision-Language Models (VLMs)
  often struggle to perform systematic and structured reasoning, especially when
  handling complex visual question-answering tasks. In this work, we introduce
  LLaVA-o1, a novel VLM designed to conduct autonomous multistage reasoning.
  Unlike chain-of-thought prompting, LLaVA-o1 independently engages in sequential
  stages of summarization, visual interpretation, logical reasoning, and
  conclusion generation. This structured approach enables LLaVA-o1 to achieve
  marked improvements in precision on reasoning-intensive tasks. To accomplish
  this, we compile the LLaVA-o1-100k dataset, integrating samples from various
  visual question answering sources and providing structured reasoning
  annotations. Besides, we propose an inference-time stage-level beam search
  method, which enables effective inference-time scaling. Remarkably, with only
  100k training samples and a simple yet effective inference time scaling method,
  LLaVA-o1 not only outperforms its base model by 8.9% on a wide range of
  multimodal reasoning benchmarks, but also surpasses the performance of larger
  and even closed-source models, such as Gemini-1.5-pro, GPT-4o-mini, and
  Llama-3.2-90B-Vision-Instruct.
paper_authors._id: 673aa0816c12c4b98bda19ea
paper_authors.name: Guowei Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-18T07:47:22.405Z
paper_authors._id: 63e992cdccae1fe5c6222f84
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e992cdccae1fe5c6222f84/IvksSUf2DENfUwCZSmNPd.jpeg
paper_authors.fullname: Guowei Xu
paper_authors.type: user
paper_authors.user: Xkev
paper_authors._id: 673aa0816c12c4b98bda19eb
paper_authors.name: Peng Jin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-18T15:19:57.329Z
paper_authors._id: 651585ea18cfed0d30bee586
paper_authors.avatarUrl: /avatars/579e468334102472d870875fe40302e6.svg
paper_authors.fullname: Peng Jin
paper_authors.type: user
paper_authors.user: Chat-UniVi
paper_authors._id: 673aa0816c12c4b98bda19ec
paper_authors.name: Li Hao
paper_authors._id: 673aa0816c12c4b98bda19ed
paper_authors.name: Yibing Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-18T07:49:29.520Z
paper_authors._id: 62c51800cb7033fd49b8efb7
paper_authors.avatarUrl: /avatars/06c2be0015f8022f9912f2279f2b3597.svg
paper_authors.fullname: Song
paper_authors.type: user
paper_authors.user: Yibing
paper_authors._id: 673aa0816c12c4b98bda19ee
paper_authors.name: Lichao Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-18T07:49:20.178Z
paper_authors._id: 65a52766215aabac489e3468
paper_authors.avatarUrl: /avatars/fe05e22cd7e12e961296426434e17c76.svg
paper_authors.fullname: Lichao Sun
paper_authors.type: user
paper_authors.user: sunlichao137
paper_authors._id: 673aa0816c12c4b98bda19ef
paper_authors.name: Li Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-18T07:49:13.527Z
paper_authors._id: 614030bd8cbdb613b82f36a8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1631596713749-noauth.jpeg
paper_authors.fullname: Li Yuan
paper_authors.type: user
paper_authors.user: LiYuan
paper_title: LLaVA-o1: Let Vision Language Models Reason Step-by-Step
paper_summary: Large language models have demonstrated substantial advancements in reasoning
  capabilities, particularly through inference-time scaling, as illustrated by
  models such as OpenAI's o1. However, current Vision-Language Models (VLMs)
  often struggle to perform systematic and structured reasoning, especially when
  handling complex visual question-answering tasks. In this work, we introduce
  LLaVA-o1, a novel VLM designed to conduct autonomous multistage reasoning.
  Unlike chain-of-thought prompting, LLaVA-o1 independently engages in sequential
  stages of summarization, visual interpretation, logical reasoning, and
  conclusion generation. This structured approach enables LLaVA-o1 to achieve
  marked improvements in precision on reasoning-intensive tasks. To accomplish
  this, we compile the LLaVA-o1-100k dataset, integrating samples from various
  visual question answering sources and providing structured reasoning
  annotations. Besides, we propose an inference-time stage-level beam search
  method, which enables effective inference-time scaling. Remarkably, with only
  100k training samples and a simple yet effective inference time scaling method,
  LLaVA-o1 not only outperforms its base model by 8.9% on a wide range of
  multimodal reasoning benchmarks, but also surpasses the performance of larger
  and even closed-source models, such as Gemini-1.5-pro, GPT-4o-mini, and
  Llama-3.2-90B-Vision-Instruct.
paper_ai_keywords: Vision-Language Models (VLMs) sequential stages summarization visual interpretation Logical reasoning conclusion generation structured reasoning annotations inference-time stage-level beam search succinctly multimodal reasoning benchmarks