JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical
  Spatio-Temporal Prior Synchronization
This paper introduces JavisDiT, a novel Joint Audio-Video Diffusion
Transformer designed for synchronized audio-video generation (JAVG). Built upon
the powerful Diffusion Transformer (DiT) architecture, JavisDiT is able to
generate high-quality audio and video content simultaneously from open-ended
user prompts. To ensure optimal synchronization, we introduce a fine-grained
spatio-temporal alignment mechanism through a Hierarchical Spatial-Temporal
Synchronized Prior (HiST-Sypo) Estimator. This module extracts both global and
fine-grained spatio-temporal priors, guiding the synchronization between the
visual and auditory components. Furthermore, we propose a new benchmark,
JavisBench, consisting of 10,140 high-quality text-captioned sounding videos
spanning diverse scenes and complex real-world scenarios. Further, we
specifically devise a robust metric for evaluating the synchronization between
generated audio-video pairs in real-world complex content. Experimental results
demonstrate that JavisDiT significantly outperforms existing methods by
ensuring both high-quality generation and precise synchronization, setting a
new standard for JAVG tasks. Our code, model, and dataset will be made publicly
available at https://javisdit.github.io/.
67eea084cdb7049bd6f79dd7
Kai Liu
claimed_verified
2025-04-03T19:20:50.100Z
678bdcbe600666579235a1f3
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/FoPxPc-T3xbzzkgQkrIbg.png
KAI LIU
user
kkail8
67eea084cdb7049bd6f79dd8
Wei Li
67eea084cdb7049bd6f79dd9
Lai Chen
67eea084cdb7049bd6f79dda
Shengqiong Wu
claimed_verified
2025-04-08T06:56:47.490Z
64c139d867eff857ea51caa8
/avatars/4b7b3f41c2e2cfa21dd43bbac6e081ae.svg
Shengqiong Wu
user
ChocoWu
67eea084cdb7049bd6f79ddb
Yanhao Zheng
67eea084cdb7049bd6f79ddc
Jiayi Ji
67eea084cdb7049bd6f79ddd
Fan Zhou
67eea084cdb7049bd6f79dde
Rongxin Jiang
67eea084cdb7049bd6f79ddf
Jiebo Luo
67eea084cdb7049bd6f79de0
Hao Fei
claimed_verified
2025-04-04T07:09:46.362Z
647773a1168cb428e00e9a8f
https://cdn-avatars.huggingface.co/v1/production/uploads/647773a1168cb428e00e9a8f/NiRR3ScY6Plzjibfwy1hC.jpeg
Hao Fei
user
scofield7419
67eea084cdb7049bd6f79de1
Tat-Seng Chua
JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical
  Spatio-Temporal Prior Synchronization
This paper introduces JavisDiT, a novel Joint Audio-Video Diffusion
Transformer designed for synchronized audio-video generation (JAVG). Built upon
the powerful Diffusion Transformer (DiT) architecture, JavisDiT is able to
generate high-quality audio and video content simultaneously from open-ended
user prompts. To ensure optimal synchronization, we introduce a fine-grained
spatio-temporal alignment mechanism through a Hierarchical Spatial-Temporal
Synchronized Prior (HiST-Sypo) Estimator. This module extracts both global and
fine-grained spatio-temporal priors, guiding the synchronization between the
visual and auditory components. Furthermore, we propose a new benchmark,
JavisBench, consisting of 10,140 high-quality text-captioned sounding videos
spanning diverse scenes and complex real-world scenarios. Further, we
specifically devise a robust metric for evaluating the synchronization between
generated audio-video pairs in real-world complex content. Experimental results
demonstrate that JavisDiT significantly outperforms existing methods by
ensuring both high-quality generation and precise synchronization, setting a
new standard for JAVG tasks. Our code, model, and dataset will be made publicly
available at https://javisdit.github.io/.
JavisDiT, a Joint Audio-Video Diffusion Transformer, generates high-quality synchronized audio-video content using a Hierarchical Spatial-Temporal Synchronized Prior Estimator, excelling in a new benchmark with a robust synchronization metric.
Diffusion Transformer DiT Joint Audio-Video Diffusion Transformer JAVISDiT Hierarchical Spatial-Temporal Synchronized Prior HiST-Sypo JavisBench synchronized audio-video generation