Answers.Body: <p>These papers are also very close to what I meant in the question (too long for a comment).</p>
  <p>The following references come mostly from work on speech recognition.</p>
  <ul>
  <li><a href="https://arxiv.org/abs/1910.12638" rel="nofollow noreferrer">Mockingjay</a> In this work, they use an analogy of Bert architecture that is fed by Mel-spectrogram, with some audio segments &quot;masked&quot;.
  <ul>
  <li>The model is asked to reconstruct the masked parts. To avoid the model using local smoothness of audio-data, they always mask several subsequent frames, so that reasonably long segments are being masked.</li>
  <li>They evaluate on downstream &quot;Phoneme classification tasks&quot; using the learned features and show that these learned features are stronger than raw spectrogram, in particular if little training data is available.</li>
  </ul>
  </li>
  <li><a href="https://arxiv.org/abs/2005.08575" rel="nofollow noreferrer">Audio Albert</a> Same story, but they use shared weights in the transformer layers. This significantly reduces memory and computational requirements and it is shown that results are comparable with Mockingjay (at least for phoneme-classification tasks).</li>
  <li><a href="https://arxiv.org/abs/2007.06028" rel="nofollow noreferrer">Tera</a>; another Bert variant where instead of masking, they use various &quot;Alterations&quot; of certain audio segments.</li>
  <li>This and many more references are within <a href="https://github.com/s3prl/s3prl" rel="nofollow noreferrer">this project with code</a>.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-17T15:20:56.207Z
Answers.LastActivityDate: 2021-09-17T16:34:16.077Z
Answers.LastEditDate: 2021-09-17T16:34:16.077Z
Answers.UserDisplayName: Peter Franek
Answers.Body: <p>The reason most music-generation models use discrete representations is because the long-term structures of music are very challenging to model. Note that the MIDI data in <a href="https://magenta.tensorflow.org/datasets/maestro" rel="nofollow noreferrer">MAESTRO</a> (used in the two papers you linked) encodes performances, not scores, so they include timing and accents of real performers--but are still sequences of discrete events, not audio.</p>
  <p>There's been some work on learning discrete representations directly from audio, such as with <a href="https://arxiv.org/abs/1711.00937" rel="nofollow noreferrer">vector-quantized variational autoencoders (VQ-VAEs)</a>. Typically an autoregressive model is trained on top of the learned representation; <a href="https://arxiv.org/abs/2005.00341" rel="nofollow noreferrer">Jukebox</a> used a transformer for that. By the way, I'd highly recommend reading through the &quot;related work&quot; section of the Jukebox paper for an overview of work on the audio/speech synthesis task.</p>
  <p><a href="https://ai.facebook.com/research/publications/unsupervised-speech-recognition" rel="nofollow noreferrer">wav2vec</a> is probably closest to what you're describing. They train a transformer on raw audio, self-supervised, in order to learn good representations of human speech for the speech-to-text task.</p>
  <p>As far as training directly on spectograms, there's <a href="https://arxiv.org/abs/1906.01083" rel="nofollow noreferrer">MelNet</a>, a somewhat exotic RNN trained for a variety of audio synthesis tasks, including music.</p>
  <p>Hope this helps!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-25T09:02:04.203Z
Answers.LastActivityDate: 2021-06-22T03:09:16.313Z
Answers.LastEditDate: 2021-06-22T03:09:16.313Z
Answers.UserDisplayName: Robz
Body: <p>In music information retrieval, one usually converts an audio signal into some kind &quot;sequence of frequency-vectors&quot;, such as STFT or Mel-spectrogram.</p>
  <p>I'm wondering if it is a good idea to use the transformer architecture in a self-supervised manner -- such as auto-regressive models, or BERT in NLP -- to obtain a &quot;smarter&quot; representation of the music than the spectrogram itself. Such smart pretrained representation could be used for further downstream tasks.</p>
  <p>From my quick google search, I found several papers which do something similar, but -- to my surprise -- all use some kind of symbolic/discrete music representation such as scores. (For instance <a href="https://arxiv.org/pdf/1809.04281.pdf" rel="nofollow noreferrer">here</a> or <a href="https://arxiv.org/pdf/1912.05537.pdf" rel="nofollow noreferrer">here</a>).</p>
  <p>My question is this:</p>
  <blockquote>
  <p>Is it realistic to train such an unsupervised model directly on the
  Mel spectrogram?</p>
  </blockquote>
  <p>The loss function would not be &quot;log softmax of next word probability&quot;, but some kind of l2-distance between &quot;predicted vector of spectra&quot; and &quot;observed vector of spectra&quot;, in the next time step.</p>
  <p>Did someone try it?</p>
  
Tags: <transformer><gpt><audio-processing><embeddings><self-supervised-learning>
Title: Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?
UserDisplayName: Peter Franek
ThreadText: # Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?
  
  (Asked by: username_0 on 2021-05-24)
  
  In music information retrieval, one usually converts an audio signal into some kind "sequence of frequency-vectors", such as STFT or Mel-spectrogram.
  
  I'm wondering if it is a good idea to use the transformer architecture in a self-supervised manner -- such as auto-regressive models, or BERT in NLP -- to obtain a "smarter" representation of the music than the spectrogram itself. Such smart pretrained representation could be used for further downstream tasks.
  
  From my quick google search, I found several papers which do something similar, but -- to my surprise -- all use some kind of symbolic/discrete music representation such as scores. (For instance [here](https://arxiv.org/pdf/1809.04281.pdf) or [here](https://arxiv.org/pdf/1912.05537.pdf)).
  
  My question is this:
  
  > Is it realistic to train such an unsupervised model directly on the
  > Mel spectrogram?
  
  The loss function would not be "log softmax of next word probability", but some kind of l2-distance between "predicted vector of spectra" and "observed vector of spectra", in the next time step.
  
  Did someone try it?
  
  **Tags:** transformer, gpt, audio-processing, embeddings, self-supervised-learning
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-05-25. Score: 3 | Accepted answer
  
  The reason most music-generation models use discrete representations is because the long-term structures of music are very challenging to model. Note that the MIDI data in [MAESTRO](https://magenta.tensorflow.org/datasets/maestro) (used in the two papers you linked) encodes performances, not scores, so they include timing and accents of real performers--but are still sequences of discrete events, not audio.
  
  There's been some work on learning discrete representations directly from audio, such as with [vector-quantized variational autoencoders (VQ-VAEs)](https://arxiv.org/abs/1711.00937). Typically an autoregressive model is trained on top of the learned representation; [Jukebox](https://arxiv.org/abs/2005.00341) used a transformer for that. By the way, I'd highly recommend reading through the "related work" section of the Jukebox paper for an overview of work on the audio/speech synthesis task.
  
  [wav2vec](https://ai.facebook.com/research/publications/unsupervised-speech-recognition) is probably closest to what you're describing. They train a transformer on raw audio, self-supervised, in order to learn good representations of human speech for the speech-to-text task.
  
  As far as training directly on spectograms, there's [MelNet](https://arxiv.org/abs/1906.01083), a somewhat exotic RNN trained for a variety of audio synthesis tasks, including music.
  
  Hope this helps!
  
  
  ---
  
  ### Answer by username_0 on 2021-09-17. Score: 1
  
  These papers are also very close to what I meant in the question (too long for a comment).
  
  The following references come mostly from work on speech recognition.
  
  * [Mockingjay](https://arxiv.org/abs/1910.12638) In this work, they use an analogy of Bert architecture that is fed by Mel-spectrogram, with some audio segments "masked".
    + The model is asked to reconstruct the masked parts. To avoid the model using local smoothness of audio-data, they always mask several subsequent frames, so that reasonably long segments are being masked.
    + They evaluate on downstream "Phoneme classification tasks" using the learned features and show that these learned features are stronger than raw spectrogram, in particular if little training data is available.
  * [Audio Albert](https://arxiv.org/abs/2005.08575) Same story, but they use shared weights in the transformer layers. This significantly reduces memory and computational requirements and it is shown that results are comparable with Mockingjay (at least for phoneme-classification tasks).
  * [Tera](https://arxiv.org/abs/2007.06028); another Bert variant where instead of masking, they use various "Alterations" of certain audio segments.
  * This and many more references are within [this project with code](https://github.com/s3prl/s3prl).
  
  
  ---
  
  