title: video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models
summary: Speech understanding as an element of the more generic video understanding
  using audio-visual large language models (av-LLMs) is a crucial yet
  understudied aspect. This paper proposes video-SALMONN, a single end-to-end
  av-LLM for video processing, which can understand not only visual frame
  sequences, audio events and music, but speech as well. To obtain fine-grained
  temporal information required by speech understanding, while keeping efficient
  for other video elements, this paper proposes a novel multi-resolution causal
  Q-Former (MRC Q-Former) structure to connect pre-trained audio-visual encoders
  and the backbone large language model. Moreover, dedicated training approaches
  including the diversity loss and the unpaired audio-visual mixed training
  scheme are proposed to avoid frames or modality dominance. On the introduced
  speech-audio-visual evaluation benchmark, video-SALMONN achieves more than 25\%
  absolute accuracy improvements on the video-QA task and over 30\% absolute
  accuracy improvements on audio-visual QA tasks with human speech. In addition,
  video-SALMONN demonstrates remarkable video comprehension and reasoning
  abilities on tasks that are unprecedented by other av-LLMs. Our training code
  and model checkpoints are available at
  \url{https://github.com/bytedance/SALMONN/}.
paper_authors._id: 667a64f34af5328a70faac49
paper_authors.name: Guangzhi Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T14:02:41.604Z
paper_authors._id: 64ab544489aa67e4a2505eeb
paper_authors.avatarUrl: /avatars/f1a9def3afbec2f8b89ef4450770d67e.svg
paper_authors.fullname: Guangzhi Sun
paper_authors.type: user
paper_authors.user: BrianatCambridge
paper_authors._id: 667a64f34af5328a70faac4a
paper_authors.name: Wenyi Yu
paper_authors._id: 667a64f34af5328a70faac4b
paper_authors.name: Changli Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T14:02:55.818Z
paper_authors._id: 63770389cdcc1bf630870758
paper_authors.avatarUrl: /avatars/6bc7c602e79688bc42a4c79ecf5b6d2d.svg
paper_authors.fullname: Changli Tang
paper_authors.type: user
paper_authors.user: Changli
paper_authors._id: 667a64f34af5328a70faac4c
paper_authors.name: Xianzhao Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T14:03:01.502Z
paper_authors._id: 65a895f09fa2a0f9a2a5800f
paper_authors.avatarUrl: /avatars/a0bc08fec550ff022a1d948d3ad9d42c.svg
paper_authors.fullname: xianzhao chen
paper_authors.type: user
paper_authors.user: cxzcxzcxz
paper_authors._id: 667a64f34af5328a70faac4d
paper_authors.name: Tian Tan
paper_authors._id: 667a64f34af5328a70faac4e
paper_authors.name: Wei Li
paper_authors._id: 667a64f34af5328a70faac4f
paper_authors.name: Lu Lu
paper_authors._id: 667a64f34af5328a70faac50
paper_authors.name: Zejun Ma
paper_authors._id: 667a64f34af5328a70faac51
paper_authors.name: Yuxuan Wang
paper_authors._id: 667a64f34af5328a70faac52
paper_authors.name: Chao Zhang
paper_title: video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models
paper_summary: Speech understanding as an element of the more generic video understanding
  using audio-visual large language models (av-LLMs) is a crucial yet
  understudied aspect. This paper proposes video-SALMONN, a single end-to-end
  av-LLM for video processing, which can understand not only visual frame
  sequences, audio events and music, but speech as well. To obtain fine-grained
  temporal information required by speech understanding, while keeping efficient
  for other video elements, this paper proposes a novel multi-resolution causal
  Q-Former (MRC Q-Former) structure to connect pre-trained audio-visual encoders
  and the backbone large language model. Moreover, dedicated training approaches
  including the diversity loss and the unpaired audio-visual mixed training
  scheme are proposed to avoid frames or modality dominance. On the introduced
  speech-audio-visual evaluation benchmark, video-SALMONN achieves more than 25\%
  absolute accuracy improvements on the video-QA task and over 30\% absolute
  accuracy improvements on audio-visual QA tasks with human speech. In addition,
  video-SALMONN demonstrates remarkable video comprehension and reasoning
  abilities on tasks that are unprecedented by other av-LLMs. Our training code
  and model checkpoints are available at
  \url{https://github.com/bytedance/SALMONN/}.
paper_ai_keywords: audio-visual large language models (av-LLMs) video-SALMONN multi-resolution causal Q-Former (MRC Q-Former) pre-trained audio-visual encoders diversity loss unpaired audio-visual mixed training scheme video-QA task audio-visual QA tasks