title: When Less is Enough: Adaptive Token Reduction for Efficient Image
    Representation
summary: Vision encoders typically generate a large number of visual tokens, providing
  information-rich representations but significantly increasing computational
  demands. This raises the question of whether all generated tokens are equally
  valuable or if some of them can be discarded to reduce computational costs
  without compromising quality. In this paper, we introduce a new method for
  determining feature utility based on the idea that less valuable features can
  be reconstructed from more valuable ones. We implement this concept by
  integrating an autoencoder with a Gumbel-Softmax selection mechanism, that
  allows identifying and retaining only the most informative visual tokens. To
  validate our approach, we compared the performance of the LLaVA-NeXT model,
  using features selected by our method with randomly selected features. We found
  that on OCR-based tasks, more than 50% of the visual context can be removed
  with minimal performance loss, whereas randomly discarding the same proportion
  of features significantly affects the model capabilities. Furthermore, in
  general-domain tasks, even randomly retaining only 30% of tokens achieves
  performance comparable to using the full set of visual tokens. Our results
  highlight a promising direction towards adaptive and efficient multimodal
  pruning that facilitates scalable and low-overhead inference without
  compromising performance.
paper_authors._id: 67e0ffb029682c8065e1c223
paper_authors.name: Eduard Allakhverdov
paper_authors._id: 67e0ffb029682c8065e1c224
paper_authors.name: Elizaveta Goncharova
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-24T12:38:44.495Z
paper_authors._id: 6310ff34bc152fa3e810c186
paper_authors.avatarUrl: /avatars/bfd63bcd81548283f5e496e3693bf143.svg
paper_authors.fullname: Elizaveta Goncharova
paper_authors.type: user
paper_authors.user: Elizaveta
paper_authors._id: 67e0ffb029682c8065e1c225
paper_authors.name: Andrey Kuznetsov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-24T14:49:31.067Z
paper_authors._id: 643984dceb7c5616ef3f5d54
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643984dceb7c5616ef3f5d54/10JRkblrRIEVci6UJwvPz.jpeg
paper_authors.fullname: Andrey Kuznetsov
paper_authors.type: user
paper_authors.user: kuznetsoffandrey
paper_title: When Less is Enough: Adaptive Token Reduction for Efficient Image
    Representation
paper_summary: Vision encoders typically generate a large number of visual tokens, providing
  information-rich representations but significantly increasing computational
  demands. This raises the question of whether all generated tokens are equally
  valuable or if some of them can be discarded to reduce computational costs
  without compromising quality. In this paper, we introduce a new method for
  determining feature utility based on the idea that less valuable features can
  be reconstructed from more valuable ones. We implement this concept by
  integrating an autoencoder with a Gumbel-Softmax selection mechanism, that
  allows identifying and retaining only the most informative visual tokens. To
  validate our approach, we compared the performance of the LLaVA-NeXT model,
  using features selected by our method with randomly selected features. We found
  that on OCR-based tasks, more than 50% of the visual context can be removed
  with minimal performance loss, whereas randomly discarding the same proportion
  of features significantly affects the model capabilities. Furthermore, in
  general-domain tasks, even randomly retaining only 30% of tokens achieves
  performance comparable to using the full set of visual tokens. Our results
  highlight a promising direction towards adaptive and efficient multimodal
  pruning that facilitates scalable and low-overhead inference without
  compromising performance.
paper_ai_summary: A method using autoencoder and Gumbel-Softmax selection identifies and retains only the most informative visual tokens, enabling efficient multimodal pruning with minimal performance loss.
paper_ai_keywords: autoencoder Gumbel-Softmax selection mechanism visual tokens feature utility informative visual tokens LLaVA-NeXT OCR-based tasks general-domain tasks multimodal pruning scalable inference low-overhead inference