LoVA: Long-form Video-to-Audio Generation
<|reference_start|>LoVA: Long-form Video-to-Audio Generation: Video-to-audio (V2A) generation is important for video editing and post-processing, enabling the creation of semantics-aligned audio for silent video. However, most existing methods focus on generating short-form audio for short video segment (less than 10 seconds), while giving little attention to the scenario of long-form video inputs. For current UNet-based diffusion V2A models, an inevitable problem when handling long-form audio generation is the inconsistencies within the final concatenated audio. In this paper, we first highlight the importance of long-form V2A problem. Besides, we propose LoVA, a novel model for Long-form Video-to-Audio generation. Based on the Diffusion Transformer (DiT) architecture, LoVA proves to be more effective at generating long-form audio compared to existing autoregressive models and UNet-based diffusion models. Extensive objective and subjective experiments demonstrate that LoVA achieves comparable performance on 10-second V2A benchmark and outperforms all other baselines on a benchmark with long-form video input.<|reference_end|>
@article{cheng2024lova:,
    title={LoVA: Long-form Video-to-Audio Generation},
    author={Xin Cheng, Xihua Wang, Yihan Wu, Yuyue Wang and Ruihua Song},
    journal={arXiv preprint arXiv:2409.15157},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.15157},
    primaryClass={cs.SD cs.MM eess.AS}
}