Answers.Body: <p>LSTM is a neural network which learns for an input x an output y. In additional to CNNs or MLPs it considers a hiddenstate h (which is influenced by prvious inputs) when your next input x is feed into the network. </p>
  
  <p>Augmenting the feature Space is a technique which you do previous training your LSTM (to augment your data set in order to generatre more data and let the LSTM better generalize to new data). In the field of image recognition you can rotate your images by 40 degree to generate a new one. This process is known as data augmentation. Such methods also appliable to time series.</p>
  
  <p>In summary: first, you start with augmenting your input feature space in order to improve prediction accuracy and then training your LSTM with the augmeneted training data set.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-09T19:20:34.927Z
Answers.Text: I don't think this answers the question.
Answers.UserDisplayName: echo
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-10T07:16:12.900Z
Answers.Text: This answer first distinguish and relates the two terms LSTM and feature space augmentation. For a more detailed answer please give more information (for instance a paper to each term).
Answers.UserDisplayName: user3352632
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-11T07:01:19.557Z
Answers.Text: Everything @user3352632 says is correct and makes sense though :p
Answers.UserDisplayName: Andreas Storvik Strauman
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-09T16:11:13.527Z
Answers.LastActivityDate: 2018-04-09T16:11:13.527Z
Answers.UserDisplayName: user3352632
Answers.Body: <p>I just read this in a recent Bengio paper and it's pretty obvious. He says that there are zero differences between a short-term memory and an augmented feature space. However, if you want to capture long-term dependencies without blowing up the feature space, you'd want to use an LSTM because traditional approaches can't dynamically learn what to "remember".</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-08T21:12:16.867Z
Answers.Text: Correct! However, the Elman-RNN (vanilla) can in theory learn long term dependencies as well as an LSTM, as Bengio also has a paper on. The problem is that it is very sensitive to the hyper parameters and thus they are very hard to tune.
Answers.UserDisplayName: Andreas Storvik Strauman
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-08T21:13:52.190Z
Answers.Text: No, wait what? What do you mean by "augmented feature space"?
Answers.UserDisplayName: Andreas Storvik Strauman
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-08T21:49:41.627Z
Answers.Text: Vector concatenation: <Vector T>.<Vector T-1>.<Vector T-2>
Answers.UserDisplayName: echo
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-11T01:27:58.513Z
Answers.Text: Does that make sense?
Answers.UserDisplayName: echo
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-11T06:02:30.237Z
Answers.Text: No. Maybe. I don't understand how you can compare concatenating vectors and LSTM?
Answers.UserDisplayName: Andreas Storvik Strauman
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-11T07:02:08.780Z
Answers.Text: By LSTM you mean long short term memory recurrent neural network, right?
Answers.UserDisplayName: Andreas Storvik Strauman
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-12T02:39:46.533Z
Answers.Text: Yes, that's what I mean. I'm just concatenating them temporally. We have a stream of input vectors and I maintain a sliding window of them.
Answers.UserDisplayName: echo
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-12T06:58:52.120Z
Answers.Text: Are you wondering of the differences in effect of the LSTM when input vectors are concatenated along the feature axis?
Answers.UserDisplayName: Andreas Storvik Strauman
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-12T14:29:46.897Z
Answers.Text: No, the difference in how to model the temporal nature of the stream. Through an LSTM or through vector concatenation.
Answers.UserDisplayName: echo
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-12T16:41:12.787Z
Answers.Text: But these are two wildly different things? How are LSTM-RNN networks an alternative to concatenate vectors?! Vector concatenation is like a mathematical operation and an LSTM is a machine learning structure, that uses time series to learn and predict. It's is like asking "Should I use addition or a neural net to figure out if a number is big enough?" It just doesn't make sense.
Answers.UserDisplayName: Andreas Storvik Strauman
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-12T16:58:16.860Z
Answers.Text: They are an alternative for temporal modelling. Vector concatenation is a widely used technique for sequence modelling.
Answers.UserDisplayName: echo
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-12T17:20:39.570Z
Answers.Text: Either you don’t mean what I think you mean or you don’t know what you mean. I give up x)
Answers.UserDisplayName: Andreas Storvik Strauman
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-08T14:47:33.303Z
Answers.LastActivityDate: 2018-04-08T14:47:33.303Z
Answers.UserDisplayName: echo
Body: <p>In time Series prediction, we have a stream of vectors. There are different approaches for accounting for the temporal patterns between these vectors.</p>
  
  <p>There's two that I'm considering. An LSTM or augmenting the feature space. What's the difference between the two? The most obvious to me is that an LSTM is more expressive and can get superior accuracy if modelled properly. </p>
  
Tags: <machine-learning><recurrent-neural-networks><long-short-term-memory>
Title: Time Series: LSTM or Augmented Vector Space?
UserDisplayName: echo
ThreadText: # Time Series: LSTM or Augmented Vector Space?
  
  (Asked by: username_0 on 2018-04-08)
  
  In time Series prediction, we have a stream of vectors. There are different approaches for accounting for the temporal patterns between these vectors.
  
  There's two that I'm considering. An LSTM or augmenting the feature space. What's the difference between the two? The most obvious to me is that an LSTM is more expressive and can get superior accuracy if modelled properly.
  
  **Tags:** machine-learning, recurrent-neural-networks, long-short-term-memory
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-04-09. Score: 1
  
  LSTM is a neural network which learns for an input x an output y. In additional to CNNs or MLPs it considers a hiddenstate h (which is influenced by prvious inputs) when your next input x is feed into the network.
  
  Augmenting the feature Space is a technique which you do previous training your LSTM (to augment your data set in order to generatre more data and let the LSTM better generalize to new data). In the field of image recognition you can rotate your images by 40 degree to generate a new one. This process is known as data augmentation. Such methods also appliable to time series.
  
  In summary: first, you start with augmenting your input feature space in order to improve prediction accuracy and then training your LSTM with the augmeneted training data set.
  
  > **Comments:**
  > **username_0**: I don't think this answers the question.  
  > **username_1**: This answer first distinguish and relates the two terms LSTM and feature space augmentation. For a more detailed answer please give more information (for instance a paper to each term).  
  > **username_2**: Everything @username_1 says is correct and makes sense though :p  
  
  ---
  
  ### Answer by username_0 on 2018-04-08. Score: 0
  
  I just read this in a recent Bengio paper and it's pretty obvious. He says that there are zero differences between a short-term memory and an augmented feature space. However, if you want to capture long-term dependencies without blowing up the feature space, you'd want to use an LSTM because traditional approaches can't dynamically learn what to "remember".
  
  > **Comments:**
  > **username_2**: Correct! However, the Elman-RNN (vanilla) can in theory learn long term dependencies as well as an LSTM, as Bengio also has a paper on. The problem is that it is very sensitive to the hyper parameters and thus they are very hard to tune.  
  > **username_2**: No, wait what? What do you mean by "augmented feature space"?  
  > **username_0**: Vector concatenation: <Vector T>.<Vector T-1>.<Vector T-2>  
  > **username_0**: Does that make sense?  
  > **username_2**: No. Maybe. I don't understand how you can compare concatenating vectors and LSTM?  
  > **username_2**: By LSTM you mean long short term memory recurrent neural network, right?  
  > **username_0**: Yes, that's what I mean. I'm just concatenating them temporally. We have a stream of input vectors and I maintain a sliding window of them.  
  > **username_2**: Are you wondering of the differences in effect of the LSTM when input vectors are concatenated along the feature axis?  
  > **username_0**: No, the difference in how to model the temporal nature of the stream. Through an LSTM or through vector concatenation.  
  > **username_2**: But these are two wildly different things? How are LSTM-RNN networks an alternative to concatenate vectors?! Vector concatenation is like a mathematical operation and an LSTM is a machine learning structure, that uses time series to learn and predict. It's is like asking "Should I use addition or a neural net to figure out if a number is big enough?" It just doesn't make sense.  
  > **username_0**: They are an alternative for temporal modelling. Vector concatenation is a widely used technique for sequence modelling.  
  > **username_2**: Either you don’t mean what I think you mean or you don’t know what you mean. I give up x)  
  
  ---
  
  