title: Multitrack Music Transcription with a Time-Frequency Perceiver
summary: Multitrack music transcription aims to transcribe a music audio input into
  the musical notes of multiple instruments simultaneously. It is a very
  challenging task that typically requires a more complex model to achieve
  satisfactory result. In addition, prior works mostly focus on transcriptions of
  regular instruments, however, neglecting vocals, which are usually the most
  important signal source if present in a piece of music. In this paper, we
  propose a novel deep neural network architecture, Perceiver TF, to model the
  time-frequency representation of audio input for multitrack transcription.
  Perceiver TF augments the Perceiver architecture by introducing a hierarchical
  expansion with an additional Transformer layer to model temporal coherence.
  Accordingly, our model inherits the benefits of Perceiver that posses better
  scalability, allowing it to well handle transcriptions of many instruments in a
  single model. In experiments, we train a Perceiver TF to model 12 instrument
  classes as well as vocal in a multi-task learning manner. Our result
  demonstrates that the proposed system outperforms the state-of-the-art
  counterparts (e.g., MT3 and SpecTNT) on various public datasets.
paper_authors._id: 64929c9415180cc729d95a37
paper_authors.name: Wei-Tsung Lu
paper_authors._id: 64929c9415180cc729d95a38
paper_authors.name: Ju-Chiang Wang
paper_authors._id: 64929c9415180cc729d95a39
paper_authors.name: Yun-Ning Hung
paper_title: Multitrack Music Transcription with a Time-Frequency Perceiver
paper_summary: Multitrack music transcription aims to transcribe a music audio input into
  the musical notes of multiple instruments simultaneously. It is a very
  challenging task that typically requires a more complex model to achieve
  satisfactory result. In addition, prior works mostly focus on transcriptions of
  regular instruments, however, neglecting vocals, which are usually the most
  important signal source if present in a piece of music. In this paper, we
  propose a novel deep neural network architecture, Perceiver TF, to model the
  time-frequency representation of audio input for multitrack transcription.
  Perceiver TF augments the Perceiver architecture by introducing a hierarchical
  expansion with an additional Transformer layer to model temporal coherence.
  Accordingly, our model inherits the benefits of Perceiver that posses better
  scalability, allowing it to well handle transcriptions of many instruments in a
  single model. In experiments, we train a Perceiver TF to model 12 instrument
  classes as well as vocal in a multi-task learning manner. Our result
  demonstrates that the proposed system outperforms the state-of-the-art
  counterparts (e.g., MT3 and SpecTNT) on various public datasets.
paper_ai_summary: A novel deep neural network, Perceiver TF, improves multitrack music transcription by integrating a hierarchical expansion and Transformer layer into the Perceiver architecture, achieving better scalability and performance compared to existing methods.
paper_ai_keywords: deep neural network Perceiver TF time-frequency representation Perceiver architecture hierarchical expansion Transformer layer temporal coherence multi-task learning multitrack transcription state-of-the-art