title: MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning
summary: Multi-modal large language models (MLLMs) have made significant strides in
  various visual understanding tasks. However, the majority of these models are
  constrained to process low-resolution images, which limits their effectiveness
  in perception tasks that necessitate detailed visual information. In our study,
  we present MG-LLaVA, an innovative MLLM that enhances the model's visual
  processing capabilities by incorporating a multi-granularity vision flow, which
  includes low-resolution, high-resolution, and object-centric features. We
  propose the integration of an additional high-resolution visual encoder to
  capture fine-grained details, which are then fused with base visual features
  through a Conv-Gate fusion network. To further refine the model's object
  recognition abilities, we incorporate object-level features derived from
  bounding boxes identified by offline detectors. Being trained solely on
  publicly available multimodal data through instruction tuning, MG-LLaVA
  demonstrates exceptional perception skills. We instantiate MG-LLaVA with a wide
  variety of language encoders, ranging from 3.8B to 34B, to evaluate the model's
  performance comprehensively. Extensive evaluations across multiple benchmarks
  demonstrate that MG-LLaVA outperforms existing MLLMs of comparable parameter
  sizes, showcasing its remarkable efficacy. The code will be available at
  https://github.com/PhoenixZ810/MG-LLaVA.
paper_authors._id: 667b8c509ccdbeba7c5fb537
paper_authors.name: Xiangyu Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-26T08:03:44.481Z
paper_authors._id: 6530e62f536dbca918e71c3e
paper_authors.avatarUrl: /avatars/efc93bc767e561c6c6d429f65c23382d.svg
paper_authors.fullname: Xiangyu Z
paper_authors.type: user
paper_authors.user: PhoenixZ
paper_authors._id: 667b8c509ccdbeba7c5fb538
paper_authors.name: Xiangtai Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T12:47:13.545Z
paper_authors._id: 63958b4414513eaf9029ebf1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png
paper_authors.fullname: Xiangtai Li
paper_authors.type: user
paper_authors.user: LXT
paper_authors._id: 667b8c509ccdbeba7c5fb539
paper_authors.name: Haodong Duan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T12:47:25.009Z
paper_authors._id: 63ee1379190ddd6214efd73a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
paper_authors.fullname: HAODONG DUAN
paper_authors.type: user
paper_authors.user: KennyUTC
paper_authors._id: 667b8c509ccdbeba7c5fb53a
paper_authors.name: Haian Huang
paper_authors._id: 667b8c509ccdbeba7c5fb53b
paper_authors.name: Yining Li
paper_authors._id: 667b8c509ccdbeba7c5fb53c
paper_authors.name: Kai Chen
paper_authors._id: 667b8c509ccdbeba7c5fb53d
paper_authors.name: Hua Yang
paper_title: MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning
paper_summary: Multi-modal large language models (MLLMs) have made significant strides in
  various visual understanding tasks. However, the majority of these models are
  constrained to process low-resolution images, which limits their effectiveness
  in perception tasks that necessitate detailed visual information. In our study,
  we present MG-LLaVA, an innovative MLLM that enhances the model's visual
  processing capabilities by incorporating a multi-granularity vision flow, which
  includes low-resolution, high-resolution, and object-centric features. We
  propose the integration of an additional high-resolution visual encoder to
  capture fine-grained details, which are then fused with base visual features
  through a Conv-Gate fusion network. To further refine the model's object
  recognition abilities, we incorporate object-level features derived from
  bounding boxes identified by offline detectors. Being trained solely on
  publicly available multimodal data through instruction tuning, MG-LLaVA
  demonstrates exceptional perception skills. We instantiate MG-LLaVA with a wide
  variety of language encoders, ranging from 3.8B to 34B, to evaluate the model's
  performance comprehensively. Extensive evaluations across multiple benchmarks
  demonstrate that MG-LLaVA outperforms existing MLLMs of comparable parameter
  sizes, showcasing its remarkable efficacy. The code will be available at
  https://github.com/PhoenixZ810/MG-LLaVA.
paper_ai_keywords: multi-modal large language models visual understanding tasks low-resolution images multi-granularity vision flow high-resolution visual encoder Conv-Gate fusionnetwork bounding boxes instruction tuning performance