title: QuoTA: Query-oriented Token Assignment via CoT Query Decouple for Long
    Video Comprehension
summary: Recent advances in long video understanding typically mitigate visual
  redundancy through visual token pruning based on attention distribution.
  However, while existing methods employ post-hoc low-response token pruning in
  decoder layers, they overlook the input-level semantic correlation between
  visual tokens and instructions (query). In this paper, we propose QuoTA, an
  ante-hoc training-free modular that extends existing large video-language
  models (LVLMs) for visual token assignment based on query-oriented frame-level
  importance assessment. The query-oriented token selection is crucial as it
  aligns visual processing with task-specific requirements, optimizing token
  budget utilization while preserving semantically relevant content.
  Specifically, (i) QuoTA strategically allocates frame-level importance scores
  based on query relevance, enabling one-time visual token assignment before
  cross-modal interactions in decoder layers, (ii) we decouple the query through
  Chain-of-Thoughts reasoning to facilitate more precise LVLM-based frame
  importance scoring, and (iii) QuoTA offers a plug-and-play functionality that
  extends to existing LVLMs. Extensive experimental results demonstrate that
  implementing QuoTA with LLaVA-Video-7B yields an average performance
  improvement of 3.2% across six benchmarks (including Video-MME and MLVU) while
  operating within an identical visual token budget as the baseline. Codes are
  open-sourced at https://github.com/MAC-AutoML/QuoTA.
paper_authors._id: 67d0f759cb5bf46c22ac8af1
paper_authors.name: Yongdong Luo
paper_authors._id: 67d0f759cb5bf46c22ac8af2
paper_authors.name: Wang Chen
paper_authors._id: 67d0f759cb5bf46c22ac8af3
paper_authors.name: Xiawu Zheng
paper_authors._id: 67d0f759cb5bf46c22ac8af4
paper_authors.name: Weizhong Huang
paper_authors._id: 67d0f759cb5bf46c22ac8af5
paper_authors.name: Shukang Yin
paper_authors._id: 67d0f759cb5bf46c22ac8af6
paper_authors.name: Haojia Lin
paper_authors._id: 67d0f759cb5bf46c22ac8af7
paper_authors.name: Chaoyou Fu
paper_authors._id: 67d0f759cb5bf46c22ac8af8
paper_authors.name: Jinfa Huang
paper_authors._id: 67d0f759cb5bf46c22ac8af9
paper_authors.name: Jiayi Ji
paper_authors._id: 67d0f759cb5bf46c22ac8afa
paper_authors.name: Jiebo Luo
paper_authors._id: 67d0f759cb5bf46c22ac8afb
paper_authors.name: Rongrong Ji
paper_title: QuoTA: Query-oriented Token Assignment via CoT Query Decouple for Long
    Video Comprehension
paper_summary: Recent advances in long video understanding typically mitigate visual
  redundancy through visual token pruning based on attention distribution.
  However, while existing methods employ post-hoc low-response token pruning in
  decoder layers, they overlook the input-level semantic correlation between
  visual tokens and instructions (query). In this paper, we propose QuoTA, an
  ante-hoc training-free modular that extends existing large video-language
  models (LVLMs) for visual token assignment based on query-oriented frame-level
  importance assessment. The query-oriented token selection is crucial as it
  aligns visual processing with task-specific requirements, optimizing token
  budget utilization while preserving semantically relevant content.
  Specifically, (i) QuoTA strategically allocates frame-level importance scores
  based on query relevance, enabling one-time visual token assignment before
  cross-modal interactions in decoder layers, (ii) we decouple the query through
  Chain-of-Thoughts reasoning to facilitate more precise LVLM-based frame
  importance scoring, and (iii) QuoTA offers a plug-and-play functionality that
  extends to existing LVLMs. Extensive experimental results demonstrate that
  implementing QuoTA with LLaVA-Video-7B yields an average performance
  improvement of 3.2% across six benchmarks (including Video-MME and MLVU) while
  operating within an identical visual token budget as the baseline. Codes are
  open-sourced at https://github.com/MAC-AutoML/QuoTA.
paper_ai_keywords: QuoTA ante-hoc training-free modular long video understanding visual token pruning attention distribution decoder layers input-level semantic correlation visual tokens instructions query frame-level importance assessment task-specific requirements token budget utilization semantically relevant content Chain-of-Thoughts reasoning cross-modal interactions plug-and-play functionality LLaVA-Video-7B Video-MME MLVU