title: LiveCC: Learning Video LLM with Streaming Speech Transcription at Scale
summary: Recent video large language models (Video LLMs) often depend on costly human
  annotations or proprietary model APIs (e.g., GPT-4o) to produce training data,
  which limits their training at scale. In this paper, we explore large-scale
  training for Video LLM with cheap automatic speech recognition (ASR)
  transcripts. Specifically, we propose a novel streaming training approach that
  densely interleaves the ASR words and video frames according to their
  timestamps. Compared to previous studies in vision-language representation with
  ASR, our method naturally fits the streaming characteristics of ASR, thus
  enabling the model to learn temporally-aligned, fine-grained vision-language
  modeling. To support the training algorithm, we introduce a data production
  pipeline to process YouTube videos and their closed captions (CC, same as ASR),
  resulting in Live-CC-5M dataset for pre-training and Live-WhisperX-526K dataset
  for high-quality supervised fine-tuning (SFT). Remarkably, even without SFT,
  the ASR-only pre-trained LiveCC-7B-Base model demonstrates competitive general
  video QA performance and exhibits a new capability in real-time video
  commentary. To evaluate this, we carefully design a new LiveSports-3K
  benchmark, using LLM-as-a-judge to measure the free-form commentary.
  Experiments show our final LiveCC-7B-Instruct model can surpass advanced 72B
  models (Qwen2.5-VL-72B-Instruct, LLaVA-Video-72B) in commentary quality even
  working in a real-time mode. Meanwhile, it achieves state-of-the-art results at
  the 7B/8B scale on popular video QA benchmarks such as VideoMME and OVOBench,
  demonstrating the broad generalizability of our approach. All resources of this
  paper have been released at https://showlab.github.io/livecc.
paper_authors._id: 68084e2c59762f55a5a8b5f3
paper_authors.name: Joya Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:51:43.894Z
paper_authors._id: 642435a1a3adbc7142c3b0a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642435a1a3adbc7142c3b0a6/wgLT_w9jNWRU3O0jU0646.jpeg
paper_authors.fullname: Joya Chen
paper_authors.type: user
paper_authors.user: chenjoya
paper_authors._id: 68084e2c59762f55a5a8b5f4
paper_authors.name: Ziyun Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:51:51.000Z
paper_authors._id: 65e6a9a4c799256ec5f7ef77
paper_authors.avatarUrl: /avatars/539a455841675539a62b44ff9803a948.svg
paper_authors.fullname: ziyun zeng
paper_authors.type: user
paper_authors.user: zengziyun
paper_authors._id: 68084e2c59762f55a5a8b5f5
paper_authors.name: Yiqi Lin
paper_authors._id: 68084e2c59762f55a5a8b5f6
paper_authors.name: Wei Li
paper_authors._id: 68084e2c59762f55a5a8b5f7
paper_authors.name: Zejun Ma
paper_authors._id: 68084e2c59762f55a5a8b5f8
paper_authors.name: Mike Zheng Shou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:52:15.395Z
paper_authors._id: 661ab3da2b14565c7acccf5c
paper_authors.avatarUrl: /avatars/fa4fc03664803e02aede4d4c3d50b393.svg
paper_authors.fullname: Mike Zheng Shou
paper_authors.type: user
paper_authors.user: AnalMom
paper_title: LiveCC: Learning Video LLM with Streaming Speech Transcription at Scale
paper_summary: Recent video large language models (Video LLMs) often depend on costly human
  annotations or proprietary model APIs (e.g., GPT-4o) to produce training data,
  which limits their training at scale. In this paper, we explore large-scale
  training for Video LLM with cheap automatic speech recognition (ASR)
  transcripts. Specifically, we propose a novel streaming training approach that
  densely interleaves the ASR words and video frames according to their
  timestamps. Compared to previous studies in vision-language representation with
  ASR, our method naturally fits the streaming characteristics of ASR, thus
  enabling the model to learn temporally-aligned, fine-grained vision-language
  modeling. To support the training algorithm, we introduce a data production
  pipeline to process YouTube videos and their closed captions (CC, same as ASR),
  resulting in Live-CC-5M dataset for pre-training and Live-WhisperX-526K dataset
  for high-quality supervised fine-tuning (SFT). Remarkably, even without SFT,
  the ASR-only pre-trained LiveCC-7B-Base model demonstrates competitive general
  video QA performance and exhibits a new capability in real-time video
  commentary. To evaluate this, we carefully design a new LiveSports-3K
  benchmark, using LLM-as-a-judge to measure the free-form commentary.
  Experiments show our final LiveCC-7B-Instruct model can surpass advanced 72B
  models (Qwen2.5-VL-72B-Instruct, LLaVA-Video-72B) in commentary quality even
  working in a real-time mode. Meanwhile, it achieves state-of-the-art results at
  the 7B/8B scale on popular video QA benchmarks such as VideoMME and OVOBench,
  demonstrating the broad generalizability of our approach. All resources of this
  paper have been released at https://showlab.github.io/livecc.
paper_ai_summary: Large-scale training for Video LLMs using ASR transcripts enables competitive video QA performance and real-time commentary capabilities.
paper_ai_keywords: video large language models Video LLMs automatic speech recognition ASR streaming training vision-language representation temporally-aligned fine-grained vision-language modeling data production pipeline closed captions Live-CC-5M Live-WhisperX-526K supervised fine-tuning SFT general video QA real-time video commentary LiveSports-3K LLM-as-a-judge VideoMME OVOBench LiveCC-7B-Base LiveCC-7B-Instruct Qwen2.5-VL-72B-Instruct LLaVA-Video-72B