Answers.Body: <p>Yes you can apply RNN to any sequence of same data type. The sequence can be in space, time, or any arbitrary ordered list. The <em>items</em> in the sequence can have any data at all, the only requirement is that each represents that same kind of thing (if you have multiple types of thing to process as a sequence, you just need to expand the definition so that the input features can represent all types unambiguously - essentially creating a "base class" that can represent them all).</p>
  
  <p>The RNN will <em>consume</em> the sequence as a time-based sequence, one item per time step of the RNN. However, you can think of that as the same as a processor clock for a computer . . . an RNN is essentially a trainable Turing machine, and in principle can learn to accumulate any data about the sequence it has seen, and output any function of that accumulated data. Although in practice this learning process might be too hard for our current systems, require immense amounts of data etc . . .</p>
  
  <p>In your case, STFT does create a time-based sequence. Each <em>item</em> in the sequence is a frequency analysis for a short period of time, and each time step of the sequence represents a fixed time difference between STFT frames (the windows usually overlap a little), where frequencies in the signal may change. Typically each STFT frame is a single time step input to a RNN. You <em>could</em> input the frequency-domain values in fixed order (e.g. low to high frequency) one at a time into a RNN too, but that would be unusual and would make most learning tasks harder.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-09T10:17:05.903Z
Answers.Text: @DuttaA: Yes I am sure of my answer. A RNN/LSTM run over a sequence of STFT or MFCC frames is a standard audio processing architecture, used in multiple projects. If you convert a piece *fully* to frequency domain, then an RNN could still process it - actually that's not so useful for many tasks, but that's not a limitation of RNNs, just a limitation of the representation, there aren't so many target features that are clear from such a conversion, it's just as hard to process as a raw waveform.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-09T10:24:56.253Z
Answers.Text: @DuttaA: In this case we don't have a fixed sized input, but a series of them.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-09T10:38:52.363Z
Answers.Text: @DuttaA: I *would* probably use a CNN (as you suggested in your answer) to process e.g. outputs of a spectrometer for chemical analysis, or other stable signal that could be converted to frequency domain, for exactly the issues you mention.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-09T08:28:18.033Z
Answers.LastActivityDate: 2018-10-09T10:35:17.490Z
Answers.LastEditDate: 2018-10-09T10:35:17.490Z
Answers.UserDisplayName: Neil Slater
Body: <p>Can the recurrent neural network input come from a short-time Fourier transform? I mean the input is not from the time-series domain.</p>
  
Tags: <recurrent-neural-networks><sequence-modeling><fourier-transform>
Title: Can the recurrent neural network's input come from a short-time Fourier transform?
UserDisplayName: user18884
ThreadText: # Can the recurrent neural network's input come from a short-time Fourier transform?
  
  (Asked by: username_0 on 2018-10-09)
  
  Can the recurrent neural network input come from a short-time Fourier transform? I mean the input is not from the time-series domain.
  
  **Tags:** recurrent-neural-networks, sequence-modeling, fourier-transform
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-10-09. Score: 2
  
  Yes you can apply RNN to any sequence of same data type. The sequence can be in space, time, or any arbitrary ordered list. The *items* in the sequence can have any data at all, the only requirement is that each represents that same kind of thing (if you have multiple types of thing to process as a sequence, you just need to expand the definition so that the input features can represent all types unambiguously - essentially creating a "base class" that can represent them all).
  
  The RNN will *consume* the sequence as a time-based sequence, one item per time step of the RNN. However, you can think of that as the same as a processor clock for a computer . . . an RNN is essentially a trainable Turing machine, and in principle can learn to accumulate any data about the sequence it has seen, and output any function of that accumulated data. Although in practice this learning process might be too hard for our current systems, require immense amounts of data etc . . .
  
  In your case, STFT does create a time-based sequence. Each *item* in the sequence is a frequency analysis for a short period of time, and each time step of the sequence represents a fixed time difference between STFT frames (the windows usually overlap a little), where frequencies in the signal may change. Typically each STFT frame is a single time step input to a RNN. You *could* input the frequency-domain values in fixed order (e.g. low to high frequency) one at a time into a RNN too, but that would be unusual and would make most learning tasks harder.
  
  > **Comments:**
  > **username_1**: @DuttaA: Yes I am sure of my answer. A RNN/LSTM run over a sequence of STFT or MFCC frames is a standard audio processing architecture, used in multiple projects. If you convert a piece *fully* to frequency domain, then an RNN could still process it - actually that's not so useful for many tasks, but that's not a limitation of RNNs, just a limitation of the representation, there aren't so many target features that are clear from such a conversion, it's just as hard to process as a raw waveform.  
  > **username_1**: @DuttaA: In this case we don't have a fixed sized input, but a series of them.  
  > **username_1**: @DuttaA: I *would* probably use a CNN (as you suggested in your answer) to process e.g. outputs of a spectrometer for chemical analysis, or other stable signal that could be converted to frequency domain, for exactly the issues you mention.  
  
  ---
  
  