title: Live2Diff: Live Stream Translation via Uni-directional Attention in
    Video Diffusion Models
summary: Large Language Models have shown remarkable efficacy in generating streaming
  data such as text and audio, thanks to their temporally uni-directional
  attention mechanism, which models correlations between the current token and
  previous tokens. However, video streaming remains much less explored, despite a
  growing need for live video processing. State-of-the-art video diffusion models
  leverage bi-directional temporal attention to model the correlations between
  the current frame and all the surrounding (i.e. including future) frames, which
  hinders them from processing streaming videos. To address this problem, we
  present Live2Diff, the first attempt at designing a video diffusion model with
  uni-directional temporal attention, specifically targeting live streaming video
  translation. Compared to previous works, our approach ensures temporal
  consistency and smoothness by correlating the current frame with its
  predecessors and a few initial warmup frames, without any future frames.
  Additionally, we use a highly efficient denoising scheme featuring a KV-cache
  mechanism and pipelining, to facilitate streaming video translation at
  interactive framerates. Extensive experiments demonstrate the effectiveness of
  the proposed attention mechanism and pipeline, outperforming previous methods
  in terms of temporal smoothness and/or efficiency.
paper_authors._id: 66909203b70d356ed3bf03a7
paper_authors.name: Zhening Xing
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-12T07:40:11.783Z
paper_authors._id: 62fb2a9dc95d426ff8f74c8d
paper_authors.avatarUrl: /avatars/25c1a68ee7b7d0cc7e9f56bde37f4914.svg
paper_authors.fullname: Zhening Xing
paper_authors.type: user
paper_authors.user: Leoxing
paper_authors._id: 66909203b70d356ed3bf03a8
paper_authors.name: Gereon Fox
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T12:36:03.149Z
paper_authors._id: 6615447fa15c52fa7aa5d4b3
paper_authors.avatarUrl: /avatars/fbdefe0a848ce65528dd38829e111072.svg
paper_authors.fullname: Gereon Fox
paper_authors.type: user
paper_authors.user: gfhcs
paper_authors._id: 66909203b70d356ed3bf03a9
paper_authors.name: Yanhong Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T12:36:10.050Z
paper_authors._id: 63d4b843df01ef426a0f79fb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676365795587-63d4b843df01ef426a0f79fb.jpeg
paper_authors.fullname: Yanhong Zeng
paper_authors.type: user
paper_authors.user: zengyh1900
paper_authors._id: 66909203b70d356ed3bf03aa
paper_authors.name: Xingang Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T12:37:14.896Z
paper_authors._id: 646758d6a9b4610868a138ff
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646758d6a9b4610868a138ff/fNBlK5g4g7vlPAR1bQimZ.png
paper_authors.fullname: Xingang Pan
paper_authors.type: user
paper_authors.user: Xingang-Pan
paper_authors._id: 66909203b70d356ed3bf03ab
paper_authors.name: Mohamed Elgharib
paper_authors._id: 66909203b70d356ed3bf03ac
paper_authors.name: Christian Theobalt
paper_authors._id: 66909203b70d356ed3bf03ad
paper_authors.name: Kai Chen
paper_title: Live2Diff: Live Stream Translation via Uni-directional Attention in
    Video Diffusion Models
paper_summary: Large Language Models have shown remarkable efficacy in generating streaming
  data such as text and audio, thanks to their temporally uni-directional
  attention mechanism, which models correlations between the current token and
  previous tokens. However, video streaming remains much less explored, despite a
  growing need for live video processing. State-of-the-art video diffusion models
  leverage bi-directional temporal attention to model the correlations between
  the current frame and all the surrounding (i.e. including future) frames, which
  hinders them from processing streaming videos. To address this problem, we
  present Live2Diff, the first attempt at designing a video diffusion model with
  uni-directional temporal attention, specifically targeting live streaming video
  translation. Compared to previous works, our approach ensures temporal
  consistency and smoothness by correlating the current frame with its
  predecessors and a few initial warmup frames, without any future frames.
  Additionally, we use a highly efficient denoising scheme featuring a KV-cache
  mechanism and pipelining, to facilitate streaming video translation at
  interactive framerates. Extensive experiments demonstrate the effectiveness of
  the proposed attention mechanism and pipeline, outperforming previous methods
  in terms of temporal smoothness and/or efficiency.
paper_ai_summary: Live2Diff is a video diffusion model with uni-directional temporal attention, enabling efficient live streaming video translation with improved temporal smoothness.
paper_ai_keywords: video diffusion models bi-directional temporal attention uni-directional temporal attention live streaming video translation KV-cache mechanism pipelining