title: CoS: Chain-of-Shot Prompting for Long Video Understanding
summary: Multi-modal Large Language Models (MLLMs) struggle with long videos due to
  the need for excessive visual tokens. These tokens exceed massively the context
  length of MLLMs, resulting in filled by redundant task-irrelevant shots. How to
  select shots is an unsolved critical problem: sparse sampling risks missing key
  details, while exhaustive sampling overwhelms the model with irrelevant
  content, leading to video misunderstanding. To solve this problem, we propose
  Chain-of-Shot prompting (CoS). The key idea is to frame shot selection as
  test-time visual prompt optimisation, choosing shots adaptive to video
  understanding semantic task by optimising shots-task alignment. CoS has two key
  parts: (1) a binary video summary mechanism that performs pseudo temporal
  grounding, discovering a binary coding to identify task-relevant shots, and (2)
  a video co-reasoning module that deploys the binary coding to pair (learning to
  align) task-relevant positive shots with irrelevant negative shots. It embeds
  the optimised shot selections into the original video, facilitating a focus on
  relevant context to optimize long video understanding. Experiments across three
  baselines and five datasets demonstrate the effectiveness and adaptability of
  CoS. Code given in https://lwpyh.github.io/CoS.
paper_authors._id: 67ac99089e12456bdb1d2e9d
paper_authors.name: Jian Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T11:12:25.146Z
paper_authors._id: 65e1b6e9501590df0173cbd3
paper_authors.avatarUrl: /avatars/a73e2139700e23eff455734c99cef5ba.svg
paper_authors.fullname: Jian Hu
paper_authors.type: user
paper_authors.user: lwpyh
paper_authors._id: 67ac99089e12456bdb1d2e9e
paper_authors.name: Zixu Cheng
paper_authors._id: 67ac99089e12456bdb1d2e9f
paper_authors.name: Chenyang Si
paper_authors._id: 67ac99089e12456bdb1d2ea0
paper_authors.name: Wei Li
paper_authors._id: 67ac99089e12456bdb1d2ea1
paper_authors.name: Shaogang Gong
paper_title: CoS: Chain-of-Shot Prompting for Long Video Understanding
paper_summary: Multi-modal Large Language Models (MLLMs) struggle with long videos due to
  the need for excessive visual tokens. These tokens exceed massively the context
  length of MLLMs, resulting in filled by redundant task-irrelevant shots. How to
  select shots is an unsolved critical problem: sparse sampling risks missing key
  details, while exhaustive sampling overwhelms the model with irrelevant
  content, leading to video misunderstanding. To solve this problem, we propose
  Chain-of-Shot prompting (CoS). The key idea is to frame shot selection as
  test-time visual prompt optimisation, choosing shots adaptive to video
  understanding semantic task by optimising shots-task alignment. CoS has two key
  parts: (1) a binary video summary mechanism that performs pseudo temporal
  grounding, discovering a binary coding to identify task-relevant shots, and (2)
  a video co-reasoning module that deploys the binary coding to pair (learning to
  align) task-relevant positive shots with irrelevant negative shots. It embeds
  the optimised shot selections into the original video, facilitating a focus on
  relevant context to optimize long video understanding. Experiments across three
  baselines and five datasets demonstrate the effectiveness and adaptability of
  CoS. Code given in https://lwpyh.github.io/CoS.
paper_ai_keywords: Multi-modal Large Language Models (MLLMs) visual tokens context length task-irrelevant shots Chain-of-Shot prompting (CoS) shot selection visual prompt optimisation semantic task binary video summary mechanism pseudo temporal grounding binary coding task-relevant shots video co-reasoning module positive shots negative shots optimised shot selections long video understanding FlashAttention FlashAttention baselines