DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction
<|reference_start|>DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction: Audio-visual saliency prediction can draw support from diverse modality complements, but further performance enhancement is still challenged by customized architectures as well as task-specific loss functions. In recent studies, denoising diffusion models have shown more promising in unifying task frameworks owing to their inherent ability of generalization. Following this motivation, a novel Diffusion architecture for generalized audio-visual Saliency prediction (DiffSal) is proposed in this work, which formulates the prediction problem as a conditional generative task of the saliency map by utilizing input audio and video as the conditions. Based on the spatio-temporal audio-visual features, an extra network Saliency-UNet is designed to perform multi-modal attention modulation for progressive refinement of the ground-truth saliency map from the noisy map. Extensive experiments demonstrate that the proposed DiffSal can achieve excellent performance across six challenging audio-visual benchmarks, with an average relative improvement of 6.3\% over the previous state-of-the-art results by six metrics.<|reference_end|>
@article{xiong2024diffsal:,
    title={DiffSal: Joint Audio and Video Learning for Diffusion Saliency
  Prediction},
    author={Junwen Xiong, Peng Zhang, Tao You, Chuanyue Li, Wei Huang, Yufei Zha},
    journal={arXiv preprint arXiv:2403.01226},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.01226},
    primaryClass={cs.CV}
}