title: AudioSlots: A slot-centric generative model for audio separation
summary: In a range of recent works, object-centric architectures have been shown to
  be suitable for unsupervised scene decomposition in the vision domain. Inspired
  by these methods we present AudioSlots, a slot-centric generative model for
  blind source separation in the audio domain. AudioSlots is built using
  permutation-equivariant encoder and decoder networks. The encoder network based
  on the Transformer architecture learns to map a mixed audio spectrogram to an
  unordered set of independent source embeddings. The spatial broadcast decoder
  network learns to generate the source spectrograms from the source embeddings.
  We train the model in an end-to-end manner using a permutation invariant loss
  function. Our results on Libri2Mix speech separation constitute a proof of
  concept that this approach shows promise. We discuss the results and
  limitations of our approach in detail, and further outline potential ways to
  overcome the limitations and directions for future work.
paper_authors._id: 645b1567e68c2d8346169965
paper_authors.name: Pradyumna Reddy
paper_authors._id: 645b1567e68c2d8346169966
paper_authors.name: Scott Wisdom
paper_authors._id: 645b1567e68c2d8346169967
paper_authors.name: Klaus Greff
paper_authors._id: 645b1567e68c2d8346169968
paper_authors.name: John R. Hershey
paper_authors._id: 645b1567e68c2d8346169969
paper_authors.name: Thomas Kipf
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-24T09:22:03.577Z
paper_authors._id: 6413572b6cd62eb3ba2024e9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6413572b6cd62eb3ba2024e9/UCtMdeV8_N6w-SBnTDXBy.jpeg
paper_authors.fullname: Thomas Kipf
paper_authors.type: user
paper_authors.user: tkipf
paper_title: AudioSlots: A slot-centric generative model for audio separation
paper_summary: In a range of recent works, object-centric architectures have been shown to
  be suitable for unsupervised scene decomposition in the vision domain. Inspired
  by these methods we present AudioSlots, a slot-centric generative model for
  blind source separation in the audio domain. AudioSlots is built using
  permutation-equivariant encoder and decoder networks. The encoder network based
  on the Transformer architecture learns to map a mixed audio spectrogram to an
  unordered set of independent source embeddings. The spatial broadcast decoder
  network learns to generate the source spectrograms from the source embeddings.
  We train the model in an end-to-end manner using a permutation invariant loss
  function. Our results on Libri2Mix speech separation constitute a proof of
  concept that this approach shows promise. We discuss the results and
  limitations of our approach in detail, and further outline potential ways to
  overcome the limitations and directions for future work.
paper_ai_keywords: permutations-equivariant slot-centric generative model blind source separation Transformer architecture mixed audio spectrogram unordered set independent source embeddings spatial broadcast decoder permutation invariant loss function Libri2Mix speech separation