Improving Audio Spectrogram Transformers for Sound Event Detection Through Multi-Stage Training
<|reference_start|>Improving Audio Spectrogram Transformers for Sound Event Detection Through Multi-Stage Training: This technical report describes the CP-JKU team's submission for Task 4 Sound Event Detection with Heterogeneous Training Datasets and Potentially Missing Labels of the DCASE 24 Challenge. We fine-tune three large Audio Spectrogram Transformers, PaSST, BEATs, and ATST, on the joint DESED and MAESTRO datasets in a two-stage training procedure. The first stage closely matches the baseline system setup and trains a CRNN model while keeping the large pre-trained transformer model frozen. In the second stage, both CRNN and transformer are fine-tuned using heavily weighted self-supervised losses. After the second stage, we compute strong pseudo-labels for all audio clips in the training set using an ensemble of all three fine-tuned transformers. Then, in a second iteration, we repeat the two-stage training process and include a distillation loss based on the pseudo-labels, boosting single-model performance substantially. Additionally, we pre-train PaSST and ATST on the subset of AudioSet that comes with strong temporal labels, before fine-tuning them on the Task 4 datasets.<|reference_end|>
@article{schmid2024improving,
    title={Improving Audio Spectrogram Transformers for Sound Event Detection
  Through Multi-Stage Training},
    author={Florian Schmid, Paul Primus, Tobias Morocutti, Jonathan Greif, Gerhard
  Widmer},
    journal={arXiv preprint arXiv:2408.00791},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.00791},
    primaryClass={eess.AS cs.SD}
}