title: Whisper-GPT: A Hybrid Representation Audio Large Language Model
summary: We propose WHISPER-GPT: A generative large language model (LLM) for speech
  and music that allows us to work with continuous audio representations and
  discrete tokens simultaneously as part of a single architecture. There has been
  a huge surge in generative audio, speech, and music models that utilize
  discrete audio tokens derived from neural compression algorithms, e.g. ENCODEC.
  However, one of the major drawbacks of this approach is handling the context
  length. It blows up for high-fidelity generative architecture if one has to
  account for all the audio contents at various frequencies for the next token
  prediction. By combining continuous audio representation like the spectrogram
  and discrete acoustic tokens, we retain the best of both worlds: Have all the
  information needed from the audio at a specific time instance in a single
  token, yet allow LLM to predict the future token to allow for sampling and
  other benefits discrete space provides. We show how our architecture improves
  the perplexity and negative log-likelihood scores for the next token prediction
  compared to a token-based LLM for speech and music.
paper_authors._id: 67615ccf42d2c5bcb831b56e
paper_authors.name: Prateek Verma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:26:33.703Z
paper_authors._id: 62d7f1119b629105a5d84aad
paper_authors.avatarUrl: /avatars/c74045063e7c06cb7be0fa41ebb1d824.svg
paper_authors.fullname: Prateek Verma
paper_authors.type: user
paper_authors.user: prateekv
paper_title: Whisper-GPT: A Hybrid Representation Audio Large Language Model
paper_summary: We propose WHISPER-GPT: A generative large language model (LLM) for speech
  and music that allows us to work with continuous audio representations and
  discrete tokens simultaneously as part of a single architecture. There has been
  a huge surge in generative audio, speech, and music models that utilize
  discrete audio tokens derived from neural compression algorithms, e.g. ENCODEC.
  However, one of the major drawbacks of this approach is handling the context
  length. It blows up for high-fidelity generative architecture if one has to
  account for all the audio contents at various frequencies for the next token
  prediction. By combining continuous audio representation like the spectrogram
  and discrete acoustic tokens, we retain the best of both worlds: Have all the
  information needed from the audio at a specific time instance in a single
  token, yet allow LLM to predict the future token to allow for sampling and
  other benefits discrete space provides. We show how our architecture improves
  the perplexity and negative log-likelihood scores for the next token prediction
  compared to a token-based LLM for speech and music.
paper_ai_summary: WHISPER-GPT integrates continuous audio representations and discrete tokens in a single generative large language model architecture, enhancing next token prediction performance for speech and music.
paper_ai_keywords: generative large language model speech music continuous audio representations discrete tokens neural compression algorithms ENCODEC spectrogram discrete acoustic tokens perplexity negative log-likelihood scores