title: Meteor: Mamba-based Traversal of Rationale for Large Language and Vision
    Models
summary: The rapid development of large language and vision models (LLVMs) has been
  driven by advances in visual instruction tuning. Recently, open-source LLVMs
  have curated high-quality visual instruction tuning datasets and utilized
  additional vision encoders or multiple computer vision models in order to
  narrow the performance gap with powerful closed-source LLVMs. These
  advancements are attributed to multifaceted information required for diverse
  capabilities, including fundamental image understanding, real-world knowledge
  about common-sense and non-object concepts (e.g., charts, diagrams, symbols,
  signs, and math problems), and step-by-step procedures for solving complex
  questions. Drawing from the multifaceted information, we present a new
  efficient LLVM, Mamba-based traversal of rationales (Meteor), which leverages
  multifaceted rationale to enhance understanding and answering capabilities. To
  embed lengthy rationales containing abundant information, we employ the Mamba
  architecture, capable of processing sequential data with linear time
  complexity. We introduce a new concept of traversal of rationale that
  facilitates efficient embedding of rationale. Subsequently, the backbone
  multimodal language model (MLM) is trained to generate answers with the aid of
  rationale. Through these steps, Meteor achieves significant improvements in
  vision language performances across multiple evaluation benchmarks requiring
  diverse capabilities, without scaling up the model size or employing additional
  vision encoders and computer vision models.
paper_authors._id: 6653f0c6607894ea80484b14
paper_authors.name: Byung-Kwan Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-27T07:17:58.143Z
paper_authors._id: 657152eb12f162153b50ec9d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/657152eb12f162153b50ec9d/qnldHP35PclV0pDz_05q8.jpeg
paper_authors.fullname: Byung-Kwan Lee
paper_authors.type: user
paper_authors.user: BK-Lee
paper_authors._id: 6653f0c6607894ea80484b15
paper_authors.name: Chae Won Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-27T07:59:47.261Z
paper_authors._id: 65aa2e84e2a2c86356263340
paper_authors.avatarUrl: /avatars/cba9ecceff0f67b2b5dde02e7c0a9991.svg
paper_authors.fullname: Chae Won Kim
paper_authors.type: user
paper_authors.user: chae-won-kim
paper_authors._id: 6653f0c6607894ea80484b16
paper_authors.name: Beomchan Park
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-27T07:17:54.134Z
paper_authors._id: 65390e7ce279d2fc80a3f800
paper_authors.avatarUrl: /avatars/a930c1f20fa1eef372ca92139cb32887.svg
paper_authors.fullname: Beomchan Park
paper_authors.type: user
paper_authors.user: bpbpbp0810
paper_authors._id: 6653f0c6607894ea80484b17
paper_authors.name: Yong Man Ro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-27T07:59:55.451Z
paper_authors._id: 660603529e3555d648e3baf1
paper_authors.avatarUrl: /avatars/0f54479afcfc19df00b25d5aedb4cf67.svg
paper_authors.fullname: Yong Man Ro
paper_authors.type: user
paper_authors.user: dwightro
paper_title: Meteor: Mamba-based Traversal of Rationale for Large Language and Vision
    Models
paper_summary: The rapid development of large language and vision models (LLVMs) has been
  driven by advances in visual instruction tuning. Recently, open-source LLVMs
  have curated high-quality visual instruction tuning datasets and utilized
  additional vision encoders or multiple computer vision models in order to
  narrow the performance gap with powerful closed-source LLVMs. These
  advancements are attributed to multifaceted information required for diverse
  capabilities, including fundamental image understanding, real-world knowledge
  about common-sense and non-object concepts (e.g., charts, diagrams, symbols,
  signs, and math problems), and step-by-step procedures for solving complex
  questions. Drawing from the multifaceted information, we present a new
  efficient LLVM, Mamba-based traversal of rationales (Meteor), which leverages
  multifaceted rationale to enhance understanding and answering capabilities. To
  embed lengthy rationales containing abundant information, we employ the Mamba
  architecture, capable of processing sequential data with linear time
  complexity. We introduce a new concept of traversal of rationale that
  facilitates efficient embedding of rationale. Subsequently, the backbone
  multimodal language model (MLM) is trained to generate answers with the aid of
  rationale. Through these steps, Meteor achieves significant improvements in
  vision language performances across multiple evaluation benchmarks requiring
  diverse capabilities, without scaling up the model size or employing additional
  vision encoders and computer vision models.
paper_ai_keywords: large language and vision models (LLVMs) visual instruction tuning high-quality visual instruction tuning datasets vision encoders computer vision models fundamental image understanding real-world knowledge common-sense non-object concepts charts diagrams symbols signs math problems step-by-step procedures multifaceted rationale Mamba-based traversal of rationales (Meteor) Mamba architecture sequential data linear time complexity traversal of rationale multimodal language model (MLM) vision language performances evaluation benchmarks