title: FLAP: Fast Language-Audio Pre-training
summary: We propose Fast Language-Audio Pre-training (FLAP), a self-supervised
  approach that efficiently and effectively learns aligned audio and language
  representations through masking, contrastive learning and reconstruction. For
  efficiency, FLAP randomly drops audio spectrogram tokens, focusing solely on
  the remaining ones for self-supervision. Through inter-modal contrastive
  learning, FLAP learns to align paired audio and text representations in a
  shared latent space. Notably, FLAP leverages multiple augmented views via
  masking for inter-modal contrast and learns to reconstruct the masked portion
  of audio tokens. Moreover, FLAP leverages large language models (LLMs) to
  augment the text inputs, contributing to improved performance. These approaches
  lead to more robust and informative audio-text representations, enabling FLAP
  to achieve state-of-the-art (SoTA) performance on audio-text retrieval tasks on
  AudioCaps (achieving 53.0% R@1) and Clotho (achieving 25.5% R@1).
paper_authors._id: 65487b26cf50edb69f314b7d
paper_authors.name: Ching-Feng Yeh
paper_authors._id: 65487b26cf50edb69f314b7e
paper_authors.name: Po-Yao Huang
paper_authors._id: 65487b26cf50edb69f314b7f
paper_authors.name: Vasu Sharma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-07T11:20:46.555Z
paper_authors._id: 65492730a3d1682f79f3ab7a
paper_authors.avatarUrl: /avatars/fc62605bdf10947dab393305c51ace96.svg
paper_authors.fullname: Vasu Sharma
paper_authors.type: user
paper_authors.user: vasusharma55
paper_authors._id: 65487b26cf50edb69f314b80
paper_authors.name: Shang-Wen Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-28T19:31:36.021Z
paper_authors._id: 64e3a7b8e12618b261fff364
paper_authors.avatarUrl: /avatars/860a5011f150537eb923f76d354f1bc5.svg
paper_authors.fullname: Dan Li
paper_authors.type: user
paper_authors.user: swdanielli
paper_authors._id: 65487b26cf50edb69f314b81
paper_authors.name: Gargi Gosh
paper_title: FLAP: Fast Language-Audio Pre-training
paper_summary: We propose Fast Language-Audio Pre-training (FLAP), a self-supervised
  approach that efficiently and effectively learns aligned audio and language
  representations through masking, contrastive learning and reconstruction. For
  efficiency, FLAP randomly drops audio spectrogram tokens, focusing solely on
  the remaining ones for self-supervision. Through inter-modal contrastive
  learning, FLAP learns to align paired audio and text representations in a
  shared latent space. Notably, FLAP leverages multiple augmented views via
  masking for inter-modal contrast and learns to reconstruct the masked portion
  of audio tokens. Moreover, FLAP leverages large language models (LLMs) to
  augment the text inputs, contributing to improved performance. These approaches
  lead to more robust and informative audio-text representations, enabling FLAP
  to achieve state-of-the-art (SoTA) performance on audio-text retrieval tasks on
  AudioCaps (achieving 53.0% R@1) and Clotho (achieving 25.5% R@1).
paper_ai_keywords: Fast Language-Audio Pre-training (FLAP) self-supervised audio and language representations masking contrastive learning reconstruction audio spectrogram tokens inter-modal contrastive learning shared latent space augmented views large language models (LLMs) audio-text retrieval tasks AudioCaps Clotho R@1