title: DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal
    Perception
summary: Existing Multimodal Large Language Models (MLLMs) increasingly emphasize
  complex understanding of various visual elements, including multiple objects,
  text information, and spatial relations. Their development for comprehensive
  visual perception hinges on the availability of high-quality image-text
  datasets that offer diverse visual elements and throughout image descriptions.
  However, the scarcity of such hyper-detailed datasets currently hinders
  progress within the MLLM community. The bottleneck stems from the limited
  perceptual capabilities of current caption engines, which fall short in
  providing complete and accurate annotations. To facilitate the cutting-edge
  research of MLLMs on comprehensive vision perception, we thereby propose
  Perceptual Fusion, using a low-budget but highly effective caption engine for
  complete and accurate image descriptions. Specifically, Perceptual Fusion
  integrates diverse perception experts as image priors to provide explicit
  information on visual elements and adopts an efficient MLLM as a centric pivot
  to mimic advanced MLLMs' perception abilities. We carefully select 1M highly
  representative images from uncurated LAION dataset and generate dense
  descriptions using our engine, dubbed DenseFusion-1M. Extensive experiments
  validate that our engine outperforms its counterparts, where the resulting
  dataset significantly improves the perception and cognition abilities of
  existing MLLMs across diverse vision-language benchmarks, especially with
  high-resolution images as inputs. The dataset and code are publicly available
  at https://github.com/baaivision/DenseFusion.
paper_authors._id: 66908d786dc63c0346f99abb
paper_authors.name: Xiaotong Li
paper_authors._id: 66908d786dc63c0346f99abc
paper_authors.name: Fan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T07:22:39.363Z
paper_authors._id: 640ed40dc025ddf618950af7
paper_authors.avatarUrl: /avatars/a99abd4346fa649ad4144f284ebcc972.svg
paper_authors.fullname: Fan Zhang
paper_authors.type: user
paper_authors.user: ryanzhangfan
paper_authors._id: 66908d786dc63c0346f99abd
paper_authors.name: Haiwen Diao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T11:48:23.442Z
paper_authors._id: 64b4a717aa03b6520839e9b8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b4a717aa03b6520839e9b8/Rt3ERG-6BVEA4hAwOz0_I.jpeg
paper_authors.fullname: Haiwen Diao
paper_authors.type: user
paper_authors.user: Paranioar
paper_authors._id: 66908d786dc63c0346f99abe
paper_authors.name: Yueze Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-27T07:00:00.388Z
paper_authors._id: 6458b59c7a7e192202df8fa0
paper_authors.avatarUrl: /avatars/33ee716477e5686da8723d01e199cd27.svg
paper_authors.fullname: Yueze Wang
paper_authors.type: user
paper_authors.user: yzwang
paper_authors._id: 66908d786dc63c0346f99abf
paper_authors.name: Xinlong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T11:49:12.383Z
paper_authors._id: 63ca558304c979828311c5a5
paper_authors.avatarUrl: /avatars/2a439d79fba2f987cabe780d10c94d25.svg
paper_authors.fullname: Xinlong Wang
paper_authors.type: user
paper_authors.user: xinlongwang
paper_authors._id: 66908d786dc63c0346f99ac0
paper_authors.name: Ling-Yu Duan
paper_title: DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal
    Perception
paper_summary: Existing Multimodal Large Language Models (MLLMs) increasingly emphasize
  complex understanding of various visual elements, including multiple objects,
  text information, and spatial relations. Their development for comprehensive
  visual perception hinges on the availability of high-quality image-text
  datasets that offer diverse visual elements and throughout image descriptions.
  However, the scarcity of such hyper-detailed datasets currently hinders
  progress within the MLLM community. The bottleneck stems from the limited
  perceptual capabilities of current caption engines, which fall short in
  providing complete and accurate annotations. To facilitate the cutting-edge
  research of MLLMs on comprehensive vision perception, we thereby propose
  Perceptual Fusion, using a low-budget but highly effective caption engine for
  complete and accurate image descriptions. Specifically, Perceptual Fusion
  integrates diverse perception experts as image priors to provide explicit
  information on visual elements and adopts an efficient MLLM as a centric pivot
  to mimic advanced MLLMs' perception abilities. We carefully select 1M highly
  representative images from uncurated LAION dataset and generate dense
  descriptions using our engine, dubbed DenseFusion-1M. Extensive experiments
  validate that our engine outperforms its counterparts, where the resulting
  dataset significantly improves the perception and cognition abilities of
  existing MLLMs across diverse vision-language benchmarks, especially with
  high-resolution images as inputs. The dataset and code are publicly available
  at https://github.com/baaivision/DenseFusion.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) visual perception high-quality image-text datasets visual elements image descriptions perceptual capabilities caption engines Perceptual Fusion image priors efficient MLLM LAION dataset DenseFusion-1M vision-language benchmarks high-resolution images