title: Efficient Audio Captioning with Encoder-Level Knowledge Distillation
summary: Significant improvement has been achieved in automated audio captioning (AAC)
  with recent models. However, these models have become increasingly large as
  their performance is enhanced. In this work, we propose a knowledge
  distillation (KD) framework for AAC. Our analysis shows that in the
  encoder-decoder based AAC models, it is more effective to distill knowledge
  into the encoder as compared with the decoder. To this end, we incorporate
  encoder-level KD loss into training, in addition to the standard supervised
  loss and sequence-level KD loss. We investigate two encoder-level KD methods,
  based on mean squared error (MSE) loss and contrastive loss, respectively.
  Experimental results demonstrate that contrastive KD is more robust than MSE
  KD, exhibiting superior performance in data-scarce situations. By leveraging
  audio-only data into training in the KD framework, our student model achieves
  competitive performance, with an inference speed that is 19 times
  fasterAn online demo is available at
  \url{https://huggingface.co/spaces/wsntxxn/efficient_audio_captioning}.
paper_authors._id: 669dbdc754725cd6e0051485
paper_authors.name: Xuenan Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T12:50:30.406Z
paper_authors._id: 6457623c90bf61c87f9ffdc4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6457623c90bf61c87f9ffdc4/pKG76YipWfYFhSnorepsl.jpeg
paper_authors.fullname: Xuenan Xu
paper_authors.type: user
paper_authors.user: wsntxxn
paper_authors._id: 669dbdc754725cd6e0051486
paper_authors.name: Haohe Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T12:50:38.888Z
paper_authors._id: 6155245d1c762d4d61b51d5d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677233708065-6155245d1c762d4d61b51d5d.png
paper_authors.fullname: haoheliu
paper_authors.type: user
paper_authors.user: haoheliu
paper_authors._id: 669dbdc754725cd6e0051487
paper_authors.name: Mengyue Wu
paper_authors._id: 669dbdc754725cd6e0051488
paper_authors.name: Wenwu Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T12:50:58.168Z
paper_authors._id: 65c30ec23d8075e46a3da733
paper_authors.avatarUrl: /avatars/164d4da7ba66f451ed08f45b76a42ffe.svg
paper_authors.fullname: wangwenwu
paper_authors.type: user
paper_authors.user: wangwenwu2
paper_authors._id: 669dbdc754725cd6e0051489
paper_authors.name: Mark D. Plumbley
paper_title: Efficient Audio Captioning with Encoder-Level Knowledge Distillation
paper_summary: Significant improvement has been achieved in automated audio captioning (AAC)
  with recent models. However, these models have become increasingly large as
  their performance is enhanced. In this work, we propose a knowledge
  distillation (KD) framework for AAC. Our analysis shows that in the
  encoder-decoder based AAC models, it is more effective to distill knowledge
  into the encoder as compared with the decoder. To this end, we incorporate
  encoder-level KD loss into training, in addition to the standard supervised
  loss and sequence-level KD loss. We investigate two encoder-level KD methods,
  based on mean squared error (MSE) loss and contrastive loss, respectively.
  Experimental results demonstrate that contrastive KD is more robust than MSE
  KD, exhibiting superior performance in data-scarce situations. By leveraging
  audio-only data into training in the KD framework, our student model achieves
  competitive performance, with an inference speed that is 19 times
  fasterAn online demo is available at
  \url{https://huggingface.co/spaces/wsntxxn/efficient_audio_captioning}.
paper_ai_summary: A knowledge distillation framework for automated audio captioning improves performance and speed by focusing on distilling knowledge into the encoder, with contrastive KD showing particular effectiveness in data-scarce settings.
paper_ai_keywords: knowledge distillation kd framework encoder-decoder encoder-level kd mse loss contrastive loss automated audio captioning audio-only data competitive performance inference speed