Audio-text Retrieval with Transformer-based Hierarchical Alignment and Disentangled Cross-modal Representation
<|reference_start|>Audio-text Retrieval with Transformer-based Hierarchical Alignment and Disentangled Cross-modal Representation: Most existing audio-text retrieval (ATR) approaches typically rely on a single-level interaction to associate audio and text, limiting their ability to align different modalities and leading to suboptimal matches. In this work, we present a novel ATR framework that leverages two-stream Transformers in conjunction with a Hierarchical Alignment (THA) module to identify multi-level correspondences of different Transformer blocks between audio and text. Moreover, current ATR methods mainly focus on learning a global-level representation, missing out on intricate details to capture audio occurrences that correspond to textual semantics. To bridge this gap, we introduce a Disentangled Cross-modal Representation (DCR) approach that disentangles high-dimensional features into compact latent factors to grasp fine-grained audio-text semantic correlations. Additionally, we develop a confidence-aware (CA) module to estimate the confidence of each latent factor pair and adaptively aggregate cross-modal latent factors to achieve local semantic alignment. Experiments show that our THA effectively boosts ATR performance, with the DCR approach further contributing to consistent performance gains.<|reference_end|>
@article{xin2024audio-text,
    title={Audio-text Retrieval with Transformer-based Hierarchical Alignment and
  Disentangled Cross-modal Representation},
    author={Yifei Xin, Zhihong Zhu, Xuxin Cheng, Xusheng Yang, Yuexian Zou},
    journal={arXiv preprint arXiv:2409.09256},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.09256},
    primaryClass={cs.SD eess.AS}
}