MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning
Multi-modal large language models (MLLMs) have made significant strides in
various visual understanding tasks. However, the majority of these models are
constrained to process low-resolution images, which limits their effectiveness
in perception tasks that necessitate detailed visual information. In our study,
we present MG-LLaVA, an innovative MLLM that enhances the model's visual
processing capabilities by incorporating a multi-granularity vision flow, which
includes low-resolution, high-resolution, and object-centric features. We
propose the integration of an additional high-resolution visual encoder to
capture fine-grained details, which are then fused with base visual features
through a Conv-Gate fusion network. To further refine the model's object
recognition abilities, we incorporate object-level features derived from
bounding boxes identified by offline detectors. Being trained solely on
publicly available multimodal data through instruction tuning, MG-LLaVA
demonstrates exceptional perception skills. We instantiate MG-LLaVA with a wide
variety of language encoders, ranging from 3.8B to 34B, to evaluate the model's
performance comprehensively. Extensive evaluations across multiple benchmarks
demonstrate that MG-LLaVA outperforms existing MLLMs of comparable parameter
sizes, showcasing its remarkable efficacy. The code will be available at
https://github.com/PhoenixZ810/MG-LLaVA.
667b8c509ccdbeba7c5fb537
Xiangyu Zhao
claimed_verified
2024-06-26T08:03:44.481Z
6530e62f536dbca918e71c3e
/avatars/efc93bc767e561c6c6d429f65c23382d.svg
Xiangyu Z
user
PhoenixZ
667b8c509ccdbeba7c5fb538
Xiangtai Li
admin_assigned
2024-06-26T12:47:13.545Z
63958b4414513eaf9029ebf1
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png
Xiangtai Li
user
LXT
667b8c509ccdbeba7c5fb539
Haodong Duan
admin_assigned
2024-06-26T12:47:25.009Z
63ee1379190ddd6214efd73a
https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
HAODONG DUAN
user
KennyUTC
667b8c509ccdbeba7c5fb53a
Haian Huang
667b8c509ccdbeba7c5fb53b
Yining Li
claimed_verified
2025-04-22T10:17:45.744Z
649988726677f66c2b486392
/avatars/b649a77370660e129726e29504daba34.svg
Yining Li
user
ly015
667b8c509ccdbeba7c5fb53c
Kai Chen
667b8c509ccdbeba7c5fb53d
Hua Yang
MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning
Multi-modal large language models (MLLMs) have made significant strides in
various visual understanding tasks. However, the majority of these models are
constrained to process low-resolution images, which limits their effectiveness
in perception tasks that necessitate detailed visual information. In our study,
we present MG-LLaVA, an innovative MLLM that enhances the model's visual
processing capabilities by incorporating a multi-granularity vision flow, which
includes low-resolution, high-resolution, and object-centric features. We
propose the integration of an additional high-resolution visual encoder to
capture fine-grained details, which are then fused with base visual features
through a Conv-Gate fusion network. To further refine the model's object
recognition abilities, we incorporate object-level features derived from
bounding boxes identified by offline detectors. Being trained solely on
publicly available multimodal data through instruction tuning, MG-LLaVA
demonstrates exceptional perception skills. We instantiate MG-LLaVA with a wide
variety of language encoders, ranging from 3.8B to 34B, to evaluate the model's
performance comprehensively. Extensive evaluations across multiple benchmarks
demonstrate that MG-LLaVA outperforms existing MLLMs of comparable parameter
sizes, showcasing its remarkable efficacy. The code will be available at
https://github.com/PhoenixZ810/MG-LLaVA.
MG-LLaVA enhances multimodal language models with a multi-granularity vision flow, including high-resolution and object-centric features, to improve visual understanding and performance on multimodal tasks.
multi-modal large language models MLLMs multi-granularity vision flow high-resolution visual encoder Conv-Gate fusion network object-level features bounding boxes instruction tuning