Scalable Frame Sampling for Video Classification: A Semi-Optimal Policy Approach with Reduced Search Space
<|reference_start|>Scalable Frame Sampling for Video Classification: A Semi-Optimal Policy Approach with Reduced Search Space: Given a video with $T$ frames, frame sampling is a task to select $N \ll T$ frames, so as to maximize the performance of a fixed video classifier. Not just brute-force search, but most existing methods suffer from its vast search space of $\binom{T}{N}$, especially when $N$ gets large. To address this challenge, we introduce a novel perspective of reducing the search space from $O(T^N)$ to $O(T)$. Instead of exploring the entire $O(T^N)$ space, our proposed semi-optimal policy selects the top $N$ frames based on the independently estimated value of each frame using per-frame confidence, significantly reducing the computational complexity. We verify that our semi-optimal policy can efficiently approximate the optimal policy, particularly under practical settings. Additionally, through extensive experiments on various datasets and model architectures, we demonstrate that learning our semi-optimal policy ensures stable and high performance regardless of the size of $N$ and $T$.<|reference_end|>
@article{lee2024scalable,
    title={Scalable Frame Sampling for Video Classification: A Semi-Optimal Policy
  Approach with Reduced Search Space},
    author={Junho Lee, Jeongwoo Shin, Seung Woo Ko, Seongsu Ha, Joonseok Lee},
    journal={arXiv preprint arXiv:2409.05260},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.05260},
    primaryClass={cs.CV}
}