title: Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Models
summary: Seeing clearly with high resolution is a foundation of Large Multimodal
  Models (LMMs), which has been proven to be vital for visual perception and
  reasoning. Existing works usually employ a straightforward resolution upscaling
  method, where the image consists of global and local branches, with the latter
  being the sliced image patches but resized to the same resolution as the
  former. This means that higher resolution requires more local patches,
  resulting in exorbitant computational expenses, and meanwhile, the dominance of
  local image tokens may diminish the global context. In this paper, we dive into
  the problems and propose a new framework as well as an elaborate optimization
  strategy. Specifically, we extract contextual information from the global view
  using a mixture of adapters, based on the observation that different adapters
  excel at different tasks. With regard to local patches, learnable query
  embeddings are introduced to reduce image tokens, the most important tokens
  accounting for the user question will be further selected by a similarity-based
  selector. Our empirical results demonstrate a `less is more' pattern, where
  utilizing fewer but more informative local image tokens leads to
  improved performance. Besides, a significant challenge lies in the training
  strategy, as simultaneous end-to-end training of the global mining block and
  local compression block does not yield optimal results. We thus advocate for an
  alternating training way, ensuring balanced learning between global and local
  aspects. Finally, we also introduce a challenging dataset with high
  requirements for image detail, enhancing the training of the local compression
  layer. The proposed method, termed LMM with Sophisticated Tasks, Local image
  compression, and Mixture of global Experts (SliME), achieves leading
  performance across various benchmarks with only 2 million training data.
paper_authors._id: 666a8722d70d5346028c92de
paper_authors.name: Yi-Fan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T09:30:21.831Z
paper_authors._id: 623d8ca4c29adf5ef6175615
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/623d8ca4c29adf5ef6175615/q7lHao7UPwU1u7YLSP56m.jpeg
paper_authors.fullname: Yi-Fan Zhang
paper_authors.type: user
paper_authors.user: yifanzhang114
paper_authors._id: 666a8722d70d5346028c92df
paper_authors.name: Qingsong Wen
paper_authors._id: 666a8722d70d5346028c92e0
paper_authors.name: Chaoyou Fu
paper_authors._id: 666a8722d70d5346028c92e1
paper_authors.name: Xue Wang
paper_authors._id: 666a8722d70d5346028c92e2
paper_authors.name: Zhang Zhang
paper_authors._id: 666a8722d70d5346028c92e3
paper_authors.name: Liang Wang
paper_authors._id: 666a8722d70d5346028c92e4
paper_authors.name: Rong Jin
paper_title: Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Models
paper_summary: Seeing clearly with high resolution is a foundation of Large Multimodal
  Models (LMMs), which has been proven to be vital for visual perception and
  reasoning. Existing works usually employ a straightforward resolution upscaling
  method, where the image consists of global and local branches, with the latter
  being the sliced image patches but resized to the same resolution as the
  former. This means that higher resolution requires more local patches,
  resulting in exorbitant computational expenses, and meanwhile, the dominance of
  local image tokens may diminish the global context. In this paper, we dive into
  the problems and propose a new framework as well as an elaborate optimization
  strategy. Specifically, we extract contextual information from the global view
  using a mixture of adapters, based on the observation that different adapters
  excel at different tasks. With regard to local patches, learnable query
  embeddings are introduced to reduce image tokens, the most important tokens
  accounting for the user question will be further selected by a similarity-based
  selector. Our empirical results demonstrate a `less is more' pattern, where
  utilizing fewer but more informative local image tokens leads to
  improved performance. Besides, a significant challenge lies in the training
  strategy, as simultaneous end-to-end training of the global mining block and
  local compression block does not yield optimal results. We thus advocate for an
  alternating training way, ensuring balanced learning between global and local
  aspects. Finally, we also introduce a challenging dataset with high
  requirements for image detail, enhancing the training of the local compression
  layer. The proposed method, termed LMM with Sophisticated Tasks, Local image
  compression, and Mixture of global Experts (SliME), achieves leading
  performance across various benchmarks with only 2 million training data.
paper_ai_keywords: Large Multimodal Models (LMMs) resolution upscaling global and local branches sliced image patches local image tokens mixture of adapters learnable query embeddings similarity-based selector global mining block local compression block alternating training way SliME