title: Attention Prompting on Image for Large Vision-Language Models
summary: Compared with Large Language Models (LLMs), Large Vision-Language Models
  (LVLMs) can also accept images as input, thus showcasing more interesting
  emergent capabilities and demonstrating impressive performance on various
  vision-language tasks. Motivated by text prompting in LLMs, visual prompting
  has been explored to enhance LVLMs' capabilities of perceiving visual
  information. However, previous visual prompting techniques solely process
  visual inputs without considering text queries, limiting the models' ability to
  follow text instructions to complete tasks. To fill this gap, in this work, we
  propose a new prompting technique named Attention Prompting on Image, which
  just simply overlays a text-query-guided attention heatmap on the original
  input image and effectively enhances LVLM on various tasks. Specifically, we
  generate an attention heatmap for the input image dependent on the text query
  with an auxiliary model like CLIP. Then the heatmap simply multiplies the pixel
  values of the original image to obtain the actual input image for the LVLM.
  Extensive experiments on various vison-language benchmarks verify the
  effectiveness of our technique. For example, Attention Prompting on Image
  improves LLaVA-1.5 by 3.8% and 2.9% on MM-Vet and LLaVA-Wild benchmarks,
  respectively.
paper_authors._id: 66f4b50381a6b5bf75f11eba
paper_authors.name: Runpeng Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-07T07:30:28.618Z
paper_authors._id: 635364b3c41f548fe39db945
paper_authors.avatarUrl: /avatars/ad1916bbfabca0b6651c8eabacc5eba8.svg
paper_authors.fullname: Runpeng Yu
paper_authors.type: user
paper_authors.user: rp-yu
paper_authors._id: 66f4b50381a6b5bf75f11ebb
paper_authors.name: Weihao Yu
paper_authors._id: 66f4b50381a6b5bf75f11ebc
paper_authors.name: Xinchao Wang
paper_title: Attention Prompting on Image for Large Vision-Language Models
paper_summary: Compared with Large Language Models (LLMs), Large Vision-Language Models
  (LVLMs) can also accept images as input, thus showcasing more interesting
  emergent capabilities and demonstrating impressive performance on various
  vision-language tasks. Motivated by text prompting in LLMs, visual prompting
  has been explored to enhance LVLMs' capabilities of perceiving visual
  information. However, previous visual prompting techniques solely process
  visual inputs without considering text queries, limiting the models' ability to
  follow text instructions to complete tasks. To fill this gap, in this work, we
  propose a new prompting technique named Attention Prompting on Image, which
  just simply overlays a text-query-guided attention heatmap on the original
  input image and effectively enhances LVLM on various tasks. Specifically, we
  generate an attention heatmap for the input image dependent on the text query
  with an auxiliary model like CLIP. Then the heatmap simply multiplies the pixel
  values of the original image to obtain the actual input image for the LVLM.
  Extensive experiments on various vison-language benchmarks verify the
  effectiveness of our technique. For example, Attention Prompting on Image
  improves LLaVA-1.5 by 3.8% and 2.9% on MM-Vet and LLaVA-Wild benchmarks,
  respectively.
paper_ai_keywords: Large Language Models (LLMs) Large Vision-Language Models (LVLMs) visual prompting attention heatmap text-query CLIP MM-Vet LLaVA-Wild LLaVA-1.5