MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation
<|reference_start|>MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation: We introduce MDSGen, a novel framework for vision-guided open-domain sound generation optimized for model parameter size, memory consumption, and inference speed. This framework incorporates two key innovations: (1) a redundant video feature removal module that filters out unnecessary visual information, and (2) a temporal-aware masking strategy that leverages temporal context for enhanced audio generation accuracy. In contrast to existing resource-heavy Unet-based models, MDSGen employs denoising masked diffusion transformers, facilitating efficient generation without reliance on pre-trained diffusion models. Evaluated on the benchmark VGGSound dataset, our smallest model (5M parameters) achieves 97.9% alignment accuracy, using 172x fewer parameters, 371% less memory, and offering 36x faster inference than the current 860M-parameter state-of-the-art model (93.9% accuracy). The larger model (131M parameters) reaches nearly 99% accuracy while requiring 6.5x fewer parameters. These results highlight the scalability and effectiveness of our approach.<|reference_end|>
@article{pham2024mdsgen:,
    title={MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers
  for Open-Domain Sound Generation},
    author={Trung X. Pham, Tri Ton, Chang D. Yoo},
    journal={arXiv preprint arXiv:2410.02130},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.02130},
    primaryClass={cs.SD cs.CV eess.AS}
}