title: StemGen: A music generation model that listens
summary: End-to-end generation of musical audio using deep learning techniques has
  seen an explosion of activity recently. However, most models concentrate on
  generating fully mixed music in response to abstract conditioning information.
  In this work, we present an alternative paradigm for producing music generation
  models that can listen and respond to musical context. We describe how such a
  model can be constructed using a non-autoregressive, transformer-based model
  architecture and present a number of novel architectural and sampling
  improvements. We train the described architecture on both an open-source and a
  proprietary dataset. We evaluate the produced models using standard quality
  metrics and a new approach based on music information retrieval descriptors.
  The resulting model reaches the audio quality of state-of-the-art
  text-conditioned models, as well as exhibiting strong musical coherence with
  its context.
paper_authors._id: 657bd8f11fa4e4e152c12793
paper_authors.name: Julian D. Parker
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-15T10:43:28.573Z
paper_authors._id: 657c2cf25c6f0b1f36077033
paper_authors.avatarUrl: /avatars/af538ee6d185d49efc0b7cf8660217ad.svg
paper_authors.fullname: Julian Parker
paper_authors.type: user
paper_authors.user: julian-parker
paper_authors._id: 657bd8f11fa4e4e152c12794
paper_authors.name: Janne Spijkervet
paper_authors._id: 657bd8f11fa4e4e152c12795
paper_authors.name: Katerina Kosta
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-15T12:53:27.574Z
paper_authors._id: 657c40e664dab2f4c90d6071
paper_authors.avatarUrl: /avatars/cd960f8bad12b18ee23271303aff5eb1.svg
paper_authors.fullname: Katerina Kosta
paper_authors.type: user
paper_authors.user: katkost
paper_authors._id: 657bd8f11fa4e4e152c12796
paper_authors.name: Furkan Yesiler
paper_authors._id: 657bd8f11fa4e4e152c12797
paper_authors.name: Boris Kuznetsov
paper_authors._id: 657bd8f11fa4e4e152c12798
paper_authors.name: Ju-Chiang Wang
paper_authors._id: 657bd8f11fa4e4e152c12799
paper_authors.name: Matt Avent
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T10:06:58.538Z
paper_authors._id: 63de3c9915266dd945f0d3d0
paper_authors.avatarUrl: /avatars/bebe36499f0e1f2482e9323d19e0fa90.svg
paper_authors.fullname: Matt Avent
paper_authors.type: user
paper_authors.user: matildah
paper_authors._id: 657bd8f11fa4e4e152c1279a
paper_authors.name: Jitong Chen
paper_authors._id: 657bd8f11fa4e4e152c1279b
paper_authors.name: Duc Le
paper_title: StemGen: A music generation model that listens
paper_summary: End-to-end generation of musical audio using deep learning techniques has
  seen an explosion of activity recently. However, most models concentrate on
  generating fully mixed music in response to abstract conditioning information.
  In this work, we present an alternative paradigm for producing music generation
  models that can listen and respond to musical context. We describe how such a
  model can be constructed using a non-autoregressive, transformer-based model
  architecture and present a number of novel architectural and sampling
  improvements. We train the described architecture on both an open-source and a
  proprietary dataset. We evaluate the produced models using standard quality
  metrics and a new approach based on music information retrieval descriptors.
  The resulting model reaches the audio quality of state-of-the-art
  text-conditioned models, as well as exhibiting strong musical coherence with
  its context.
paper_ai_summary: A transformer-based, non-autoregressive model generates musically coherent audio by responding to context, achieving audio quality comparable to text-conditioned models.
paper_ai_keywords: non-autoregressive transformer-based music generation musical coherence music information retrieval descriptors state-of-the-art text-conditioned models