QuoTA: Query-oriented Token Assignment via CoT Query Decouple for Long
  Video Comprehension
Recent advances in long video understanding typically mitigate visual
redundancy through visual token pruning based on attention distribution.
However, while existing methods employ post-hoc low-response token pruning in
decoder layers, they overlook the input-level semantic correlation between
visual tokens and instructions (query). In this paper, we propose QuoTA, an
ante-hoc training-free modular that extends existing large video-language
models (LVLMs) for visual token assignment based on query-oriented frame-level
importance assessment. The query-oriented token selection is crucial as it
aligns visual processing with task-specific requirements, optimizing token
budget utilization while preserving semantically relevant content.
Specifically, (i) QuoTA strategically allocates frame-level importance scores
based on query relevance, enabling one-time visual token assignment before
cross-modal interactions in decoder layers, (ii) we decouple the query through
Chain-of-Thoughts reasoning to facilitate more precise LVLM-based frame
importance scoring, and (iii) QuoTA offers a plug-and-play functionality that
extends to existing LVLMs. Extensive experimental results demonstrate that
implementing QuoTA with LLaVA-Video-7B yields an average performance
improvement of 3.2% across six benchmarks (including Video-MME and MLVU) while
operating within an identical visual token budget as the baseline. Codes are
open-sourced at https://github.com/MAC-AutoML/QuoTA.
67d0f759cb5bf46c22ac8af1
Yongdong Luo
67d0f759cb5bf46c22ac8af2
Wang Chen
67d0f759cb5bf46c22ac8af3
Xiawu Zheng
67d0f759cb5bf46c22ac8af4
Weizhong Huang
67d0f759cb5bf46c22ac8af5
Shukang Yin
67d0f759cb5bf46c22ac8af6
Haojia Lin
67d0f759cb5bf46c22ac8af7
Chaoyou Fu
67d0f759cb5bf46c22ac8af8
Jinfa Huang
claimed_verified
2025-07-09T08:54:24.099Z
63f37af60be81bdc5d92eebb
/avatars/b8dfdff4ab36988ec9a8643e82a3d2db.svg
Huang
user
Jinfa
67d0f759cb5bf46c22ac8af9
Jiayi Ji
67d0f759cb5bf46c22ac8afa
Jiebo Luo
67d0f759cb5bf46c22ac8afb
Rongrong Ji
QuoTA: Query-oriented Token Assignment via CoT Query Decouple for Long
  Video Comprehension
Recent advances in long video understanding typically mitigate visual
redundancy through visual token pruning based on attention distribution.
However, while existing methods employ post-hoc low-response token pruning in
decoder layers, they overlook the input-level semantic correlation between
visual tokens and instructions (query). In this paper, we propose QuoTA, an
ante-hoc training-free modular that extends existing large video-language
models (LVLMs) for visual token assignment based on query-oriented frame-level
importance assessment. The query-oriented token selection is crucial as it
aligns visual processing with task-specific requirements, optimizing token
budget utilization while preserving semantically relevant content.
Specifically, (i) QuoTA strategically allocates frame-level importance scores
based on query relevance, enabling one-time visual token assignment before
cross-modal interactions in decoder layers, (ii) we decouple the query through
Chain-of-Thoughts reasoning to facilitate more precise LVLM-based frame
importance scoring, and (iii) QuoTA offers a plug-and-play functionality that
extends to existing LVLMs. Extensive experimental results demonstrate that
implementing QuoTA with LLaVA-Video-7B yields an average performance
improvement of 3.2% across six benchmarks (including Video-MME and MLVU) while
operating within an identical visual token budget as the baseline. Codes are
open-sourced at https://github.com/MAC-AutoML/QuoTA.
QuoTA, a modular extension for video-language models, enhances performance by strategically assigning visual tokens based on query relevance and chain-of-thoughts reasoning, without requiring additional training.
visual token pruning attention distribution decoder layers visual token assignment query-oriented frame-level importance assessment Chain-of-Thoughts reasoning video-language models LLaVA-Video-7B Video-MME MLVU