LLaVA-o1: Let Vision Language Models Reason Step-by-Step
Large language models have demonstrated substantial advancements in reasoning
capabilities, particularly through inference-time scaling, as illustrated by
models such as OpenAI's o1. However, current Vision-Language Models (VLMs)
often struggle to perform systematic and structured reasoning, especially when
handling complex visual question-answering tasks. In this work, we introduce
LLaVA-o1, a novel VLM designed to conduct autonomous multistage reasoning.
Unlike chain-of-thought prompting, LLaVA-o1 independently engages in sequential
stages of summarization, visual interpretation, logical reasoning, and
conclusion generation. This structured approach enables LLaVA-o1 to achieve
marked improvements in precision on reasoning-intensive tasks. To accomplish
this, we compile the LLaVA-o1-100k dataset, integrating samples from various
visual question answering sources and providing structured reasoning
annotations. Besides, we propose an inference-time stage-level beam search
method, which enables effective inference-time scaling. Remarkably, with only
100k training samples and a simple yet effective inference time scaling method,
LLaVA-o1 not only outperforms its base model by 8.9% on a wide range of
multimodal reasoning benchmarks, but also surpasses the performance of larger
and even closed-source models, such as Gemini-1.5-pro, GPT-4o-mini, and
Llama-3.2-90B-Vision-Instruct.
673aa0816c12c4b98bda19ea
Guowei Xu
claimed_verified
2024-11-18T07:47:22.405Z
63e992cdccae1fe5c6222f84
https://cdn-avatars.huggingface.co/v1/production/uploads/63e992cdccae1fe5c6222f84/IvksSUf2DENfUwCZSmNPd.jpeg
Guowei Xu
user
Xkev
673aa0816c12c4b98bda19eb
Peng Jin
claimed_verified
2024-11-18T15:19:57.329Z
651585ea18cfed0d30bee586
/avatars/579e468334102472d870875fe40302e6.svg
Peng Jin
user
Chat-UniVi
673aa0816c12c4b98bda19ec
Li Hao
673aa0816c12c4b98bda19ed
Yibing Song
admin_assigned
2024-11-18T07:49:29.520Z
62c51800cb7033fd49b8efb7
/avatars/06c2be0015f8022f9912f2279f2b3597.svg
Song
user
Yibing
673aa0816c12c4b98bda19ee
Lichao Sun
admin_assigned
2024-11-18T07:49:20.178Z
65a52766215aabac489e3468
/avatars/fe05e22cd7e12e961296426434e17c76.svg
Lichao Sun
user
sunlichao137
673aa0816c12c4b98bda19ef
Li Yuan
admin_assigned
2024-11-18T07:49:13.527Z
614030bd8cbdb613b82f36a8
https://cdn-avatars.huggingface.co/v1/production/uploads/1631596713749-noauth.jpeg
Li Yuan
user
LiYuan
LLaVA-o1: Let Vision Language Models Reason Step-by-Step
Large language models have demonstrated substantial advancements in reasoning
capabilities, particularly through inference-time scaling, as illustrated by
models such as OpenAI's o1. However, current Vision-Language Models (VLMs)
often struggle to perform systematic and structured reasoning, especially when
handling complex visual question-answering tasks. In this work, we introduce
LLaVA-o1, a novel VLM designed to conduct autonomous multistage reasoning.
Unlike chain-of-thought prompting, LLaVA-o1 independently engages in sequential
stages of summarization, visual interpretation, logical reasoning, and
conclusion generation. This structured approach enables LLaVA-o1 to achieve
marked improvements in precision on reasoning-intensive tasks. To accomplish
this, we compile the LLaVA-o1-100k dataset, integrating samples from various
visual question answering sources and providing structured reasoning
annotations. Besides, we propose an inference-time stage-level beam search
method, which enables effective inference-time scaling. Remarkably, with only
100k training samples and a simple yet effective inference time scaling method,
LLaVA-o1 not only outperforms its base model by 8.9% on a wide range of
multimodal reasoning benchmarks, but also surpasses the performance of larger
and even closed-source models, such as Gemini-1.5-pro, GPT-4o-mini, and
Llama-3.2-90B-Vision-Instruct.
LLaVA-o1, a novel Vision-Language Model, enhances reasoning capabilities through a structured multistage approach, outperforming larger models on multimodal reasoning benchmarks using a limited dataset and efficient inference-time scaling.
Vision-Language Models multistage reasoning chain-of-thought prompting structured reasoning LLaVA-o1 LLaVA-o1-100k dataset inference-time stage-level beam search multimodal reasoning benchmarks