title: JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical
    Spatio-Temporal Prior Synchronization
summary: This paper introduces JavisDiT, a novel Joint Audio-Video Diffusion
  Transformer designed for synchronized audio-video generation (JAVG). Built upon
  the powerful Diffusion Transformer (DiT) architecture, JavisDiT is able to
  generate high-quality audio and video content simultaneously from open-ended
  user prompts. To ensure optimal synchronization, we introduce a fine-grained
  spatio-temporal alignment mechanism through a Hierarchical Spatial-Temporal
  Synchronized Prior (HiST-Sypo) Estimator. This module extracts both global and
  fine-grained spatio-temporal priors, guiding the synchronization between the
  visual and auditory components. Furthermore, we propose a new benchmark,
  JavisBench, consisting of 10,140 high-quality text-captioned sounding videos
  spanning diverse scenes and complex real-world scenarios. Further, we
  specifically devise a robust metric for evaluating the synchronization between
  generated audio-video pairs in real-world complex content. Experimental results
  demonstrate that JavisDiT significantly outperforms existing methods by
  ensuring both high-quality generation and precise synchronization, setting a
  new standard for JAVG tasks. Our code, model, and dataset will be made publicly
  available at https://javisdit.github.io/.
paper_authors._id: 67eea084cdb7049bd6f79dd7
paper_authors.name: Kai Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T19:20:50.100Z
paper_authors._id: 678bdcbe600666579235a1f3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/FoPxPc-T3xbzzkgQkrIbg.png
paper_authors.fullname: KAI LIU
paper_authors.type: user
paper_authors.user: kkail8
paper_authors._id: 67eea084cdb7049bd6f79dd8
paper_authors.name: Wei Li
paper_authors._id: 67eea084cdb7049bd6f79dd9
paper_authors.name: Lai Chen
paper_authors._id: 67eea084cdb7049bd6f79dda
paper_authors.name: Shengqiong Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-08T06:56:47.490Z
paper_authors._id: 64c139d867eff857ea51caa8
paper_authors.avatarUrl: /avatars/4b7b3f41c2e2cfa21dd43bbac6e081ae.svg
paper_authors.fullname: Shengqiong Wu
paper_authors.type: user
paper_authors.user: ChocoWu
paper_authors._id: 67eea084cdb7049bd6f79ddb
paper_authors.name: Yanhao Zheng
paper_authors._id: 67eea084cdb7049bd6f79ddc
paper_authors.name: Jiayi Ji
paper_authors._id: 67eea084cdb7049bd6f79ddd
paper_authors.name: Fan Zhou
paper_authors._id: 67eea084cdb7049bd6f79dde
paper_authors.name: Rongxin Jiang
paper_authors._id: 67eea084cdb7049bd6f79ddf
paper_authors.name: Jiebo Luo
paper_authors._id: 67eea084cdb7049bd6f79de0
paper_authors.name: Hao Fei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:09:46.362Z
paper_authors._id: 647773a1168cb428e00e9a8f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647773a1168cb428e00e9a8f/NiRR3ScY6Plzjibfwy1hC.jpeg
paper_authors.fullname: Hao Fei
paper_authors.type: user
paper_authors.user: scofield7419
paper_authors._id: 67eea084cdb7049bd6f79de1
paper_authors.name: Tat-Seng Chua
paper_title: JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical
    Spatio-Temporal Prior Synchronization
paper_summary: This paper introduces JavisDiT, a novel Joint Audio-Video Diffusion
  Transformer designed for synchronized audio-video generation (JAVG). Built upon
  the powerful Diffusion Transformer (DiT) architecture, JavisDiT is able to
  generate high-quality audio and video content simultaneously from open-ended
  user prompts. To ensure optimal synchronization, we introduce a fine-grained
  spatio-temporal alignment mechanism through a Hierarchical Spatial-Temporal
  Synchronized Prior (HiST-Sypo) Estimator. This module extracts both global and
  fine-grained spatio-temporal priors, guiding the synchronization between the
  visual and auditory components. Furthermore, we propose a new benchmark,
  JavisBench, consisting of 10,140 high-quality text-captioned sounding videos
  spanning diverse scenes and complex real-world scenarios. Further, we
  specifically devise a robust metric for evaluating the synchronization between
  generated audio-video pairs in real-world complex content. Experimental results
  demonstrate that JavisDiT significantly outperforms existing methods by
  ensuring both high-quality generation and precise synchronization, setting a
  new standard for JAVG tasks. Our code, model, and dataset will be made publicly
  available at https://javisdit.github.io/.
paper_ai_summary: JavisDiT, a Joint Audio-Video Diffusion Transformer, generates high-quality synchronized audio-video content using a Hierarchical Spatial-Temporal Synchronized Prior Estimator, excelling in a new benchmark with a robust synchronization metric.
paper_ai_keywords: Diffusion Transformer DiT Joint Audio-Video Diffusion Transformer JAVISDiT Hierarchical Spatial-Temporal Synchronized Prior HiST-Sypo JavisBench synchronized audio-video generation