title: LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs
summary: Reasoning is a fundamental capability for solving complex multi-step
  problems, particularly in visual contexts where sequential step-wise
  understanding is essential. Existing approaches lack a comprehensive framework
  for evaluating visual reasoning and do not emphasize step-wise problem-solving.
  To this end, we propose a comprehensive framework for advancing step-by-step
  visual reasoning in large language models (LMMs) through three key
  contributions. First, we introduce a visual reasoning benchmark specifically
  designed to evaluate multi-step reasoning tasks. The benchmark presents a
  diverse set of challenges with eight different categories ranging from complex
  visual perception to scientific reasoning with over 4k reasoning steps in
  total, enabling robust evaluation of LLMs' abilities to perform accurate and
  interpretable visual reasoning across multiple steps. Second, we propose a
  novel metric that assesses visual reasoning quality at the granularity of
  individual steps, emphasizing both correctness and logical coherence. The
  proposed metric offers deeper insights into reasoning performance compared to
  traditional end-task accuracy metrics. Third, we present a new multimodal
  visual reasoning model, named LlamaV-o1, trained using a multi-step curriculum
  learning approach, where tasks are progressively organized to facilitate
  incremental skill acquisition and problem-solving. The proposed LlamaV-o1 is
  designed for multi-step reasoning and learns step-by-step through a structured
  training paradigm. Extensive experiments show that our LlamaV-o1 outperforms
  existing open-source models and performs favorably against close-source
  proprietary models. Compared to the recent Llava-CoT, our LlamaV-o1 achieves an
  average score of 67.3 with an absolute gain of 3.8\% across six benchmarks
  while being 5 times faster during inference scaling. Our benchmark, model, and
  code are publicly available.
paper_authors._id: 6784817c956a6e09449547bd
paper_authors.name: Omkar Thawakar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:25:42.768Z
paper_authors._id: 64b7d2ad8c632fbca9507431
paper_authors.avatarUrl: /avatars/76c31ea218108cf6c3715269f7605404.svg
paper_authors.fullname: Omkar Thawakar
paper_authors.type: user
paper_authors.user: omkarthawakar
paper_authors._id: 6784817c956a6e09449547be
paper_authors.name: Dinura Dissanayake
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-04T09:39:49.771Z
paper_authors._id: 6639cd849662bb58d5fe5793
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6639cd849662bb58d5fe5793/PHUa38L5_llipN52ngajY.png
paper_authors.fullname: Dissanayake
paper_authors.type: user
paper_authors.user: Dinura
paper_authors._id: 6784817c956a6e09449547bf
paper_authors.name: Ketan More
paper_authors._id: 6784817c956a6e09449547c0
paper_authors.name: Ritesh Thawkar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:26:18.535Z
paper_authors._id: 664cec3ba016876f526afe7a
paper_authors.avatarUrl: /avatars/69007f7e7f7ff78069101deb4b354e37.svg
paper_authors.fullname: Ritesh Thawkar 
paper_authors.type: user
paper_authors.user: Ritesh-hf
paper_authors._id: 6784817c956a6e09449547c1
paper_authors.name: Ahmed Heakl
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-13T08:54:30.443Z
paper_authors._id: 656864e12d73834278a8dea7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/656864e12d73834278a8dea7/sfAWS2eyPtFHb_2GZIypp.jpeg
paper_authors.fullname: Ahmed Heakl
paper_authors.type: user
paper_authors.user: ahmedheakl
paper_authors._id: 6784817c956a6e09449547c2
paper_authors.name: Noor Ahsan
paper_authors._id: 6784817c956a6e09449547c3
paper_authors.name: Yuhao Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T08:04:27.329Z
paper_authors._id: 667840df246665be1af09cb5
paper_authors.avatarUrl: /avatars/c714d3fbb32611704effc7964a248edc.svg
paper_authors.fullname: Yuhao Li
paper_authors.type: user
paper_authors.user: UHow
paper_authors._id: 6784817c956a6e09449547c4
paper_authors.name: Mohammed Zumri
paper_authors._id: 6784817c956a6e09449547c5
paper_authors.name: Jean Lahoud
paper_authors._id: 6784817c956a6e09449547c6
paper_authors.name: Rao Muhammad Anwer
paper_authors._id: 6784817c956a6e09449547c7
paper_authors.name: Hisham Cholakkal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-13T09:27:47.044Z
paper_authors._id: 654a5f4f9b8bd6406d45bb46
paper_authors.avatarUrl: /avatars/ac0d7eef62cd98a280b162cf7896b1a2.svg
paper_authors.fullname: Hisham Cholakkal
paper_authors.type: user
paper_authors.user: hishamcholakkal
paper_authors._id: 6784817c956a6e09449547c8
paper_authors.name: Ivan Laptev
paper_authors._id: 6784817c956a6e09449547c9
paper_authors.name: Mubarak Shah
paper_authors._id: 6784817c956a6e09449547ca
paper_authors.name: Fahad Shahbaz Khan
paper_authors._id: 6784817c956a6e09449547cb
paper_authors.name: Salman Khan
paper_title: LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs
paper_summary: Reasoning is a fundamental capability for solving complex multi-step
  problems, particularly in visual contexts where sequential step-wise
  understanding is essential. Existing approaches lack a comprehensive framework
  for evaluating visual reasoning and do not emphasize step-wise problem-solving.
  To this end, we propose a comprehensive framework for advancing step-by-step
  visual reasoning in large language models (LMMs) through three key
  contributions. First, we introduce a visual reasoning benchmark specifically
  designed to evaluate multi-step reasoning tasks. The benchmark presents a
  diverse set of challenges with eight different categories ranging from complex
  visual perception to scientific reasoning with over 4k reasoning steps in
  total, enabling robust evaluation of LLMs' abilities to perform accurate and
  interpretable visual reasoning across multiple steps. Second, we propose a
  novel metric that assesses visual reasoning quality at the granularity of
  individual steps, emphasizing both correctness and logical coherence. The
  proposed metric offers deeper insights into reasoning performance compared to
  traditional end-task accuracy metrics. Third, we present a new multimodal
  visual reasoning model, named LlamaV-o1, trained using a multi-step curriculum
  learning approach, where tasks are progressively organized to facilitate
  incremental skill acquisition and problem-solving. The proposed LlamaV-o1 is
  designed for multi-step reasoning and learns step-by-step through a structured
  training paradigm. Extensive experiments show that our LlamaV-o1 outperforms
  existing open-source models and performs favorably against close-source
  proprietary models. Compared to the recent Llava-CoT, our LlamaV-o1 achieves an
  average score of 67.3 with an absolute gain of 3.8\% across six benchmarks
  while being 5 times faster during inference scaling. Our benchmark, model, and
  code are publicly available.
paper_ai_summary: A framework for evaluating and improving step-by-step visual reasoning in large language models using a specialized benchmark and a novel multimodal model trained with curriculum learning.
paper_ai_keywords: visual reasoning large language models visual reasoning benchmark multi-step reasoning tasks reasoning steps visual perception scientific reasoning visual reasoning quality multimodal visual reasoning model LlamaV-o1 curriculum learning incremental skill acquisition problem-solving Llava-CoT end-task accuracy metrics