title: Enhance audio generation controllability through representation
    similarity regularization
summary: This paper presents an innovative approach to enhance control over audio
  generation by emphasizing the alignment between audio and text representations
  during model training. In the context of language model-based audio generation,
  the model leverages input from both textual and audio token representations to
  predict subsequent audio tokens. However, the current configuration lacks
  explicit regularization to ensure the alignment between the chosen text
  representation and the language model's predictions. Our proposal involves the
  incorporation of audio and text representation regularization, particularly
  during the classifier-free guidance (CFG) phase, where the text condition is
  excluded from cross attention during language model training. The aim of this
  proposed representation regularization is to minimize discrepancies in audio
  and text similarity compared to other samples within the same training batch.
  Experimental results on both music and audio generation tasks demonstrate that
  our proposed methods lead to improvements in objective metrics for both audio
  and music generation, as well as an enhancement in the human perception for
  audio generation.
paper_authors._id: 65090b74b44445e9b378e19f
paper_authors.name: Yangyang Shi
paper_authors._id: 65090b74b44445e9b378e1a0
paper_authors.name: Gael Le Lan
paper_authors._id: 65090b74b44445e9b378e1a1
paper_authors.name: Varun Nagaraja
paper_authors._id: 65090b74b44445e9b378e1a2
paper_authors.name: Zhaoheng Ni
paper_authors._id: 65090b74b44445e9b378e1a3
paper_authors.name: Xinhao Mei
paper_authors._id: 65090b74b44445e9b378e1a4
paper_authors.name: Ernie Chang
paper_authors._id: 65090b74b44445e9b378e1a5
paper_authors.name: Forrest Iandola
paper_authors._id: 65090b74b44445e9b378e1a6
paper_authors.name: Yang Liu
paper_authors._id: 65090b74b44445e9b378e1a7
paper_authors.name: Vikas Chandra
paper_title: Enhance audio generation controllability through representation
    similarity regularization
paper_summary: This paper presents an innovative approach to enhance control over audio
  generation by emphasizing the alignment between audio and text representations
  during model training. In the context of language model-based audio generation,
  the model leverages input from both textual and audio token representations to
  predict subsequent audio tokens. However, the current configuration lacks
  explicit regularization to ensure the alignment between the chosen text
  representation and the language model's predictions. Our proposal involves the
  incorporation of audio and text representation regularization, particularly
  during the classifier-free guidance (CFG) phase, where the text condition is
  excluded from cross attention during language model training. The aim of this
  proposed representation regularization is to minimize discrepancies in audio
  and text similarity compared to other samples within the same training batch.
  Experimental results on both music and audio generation tasks demonstrate that
  our proposed methods lead to improvements in objective metrics for both audio
  and music generation, as well as an enhancement in the human perception for
  audio generation.
paper_ai_keywords: classifier-free guidance (CFG) cross attention text representation regularization audio and text similarity