Answers.Body: <p>One thing I might add is not to expect the ML model to take care of everything, that is where ML engineering comes into play. One suggestion I have, without knowing much about your data stream, is that you need to implement a real-time data ingestion/pipeline, like <a href="https://kafka.apache.org/documentation/streams/" rel="nofollow noreferrer">Apache Kafka Stream</a> (so similar implementation), where you can decompose your stream into output that your model was trained for. You could split by dictionary, pattern or whatever you need, then push the split output into another real-time stream that can be ingested by your model. Not a lot to go for, but hopefully it helps a little. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-24T16:34:52.893Z
Answers.Text: Hi, thanks. The quetion is still what method to use to decompose the signal. I was thinkig of some sort of convolution, but I have no idea yet how exactly it would help me.
Answers.UserDisplayName: MScott
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-24T16:28:32.230Z
Answers.LastActivityDate: 2020-01-24T16:28:32.230Z
Answers.UserDisplayName: wyrml4
Body: <p>How can a data stream for a RNN (LSTM) be handled, when the stream contains data sets belonging to different prediction classes?</p>
  
  <p>Training phase:
  I have trained a LSTM to predict a class out of a sequence of Letters. For the training phase I used a fixed data array where the beginning an the ending of a sequence belonged to a class. Of course there is a little noise but the whole data set was labled with a class. E.g:</p>
  
  <pre><code>Seq.    is  Class
  ABC     is  One
  CBA     is  Two
  ABD     is  Three
  </code></pre>
  
  <p>The network predicts well when it sees a static data array.</p>
  
  <p>Problem in Prediction Phase:
  During prediction the LSTM will receive a data stream where there is a sequence off arrays but there is no delimiter. The data set can not be distinguished or separated. I am not sure how it would perform when I have a data stream for different classes like 
      ABCABCCBAABD. </p>
  
  <p>I guess in speech recognition one must face similiar problems.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-18T10:48:56.093Z
Comments.Text: u need to do some feature engineering possibly
Comments.UserDisplayName: Dan D.
Tags: <deep-learning><recurrent-neural-networks><long-short-term-memory>
Title: How to process data in a data stream for a LSTM
UserDisplayName: MScott
ThreadText: # How to process data in a data stream for a LSTM
  
  (Asked by: username_0 on 2020-01-24)
  
  How can a data stream for a RNN (LSTM) be handled, when the stream contains data sets belonging to different prediction classes?
  Training phase:
  I have trained a LSTM to predict a class out of a sequence of Letters. For the training phase I used a fixed data array where the beginning an the ending of a sequence belonged to a class. Of course there is a little noise but the whole data set was labled with a class. E.g:
  Seq.    is  Class
  ABC     is  One
  CBA     is  Two
  ABD     is  Three
  
  The network predicts well when it sees a static data array.
  Problem in Prediction Phase:
  During prediction the LSTM will receive a data stream where there is a sequence off arrays but there is no delimiter. The data set can not be distinguished or separated. I am not sure how it would perform when I have a data stream for different classes like 
      ABCABCCBAABD. 
  I guess in speech recognition one must face similiar problems.
  
  **Tags:** deep-learning, recurrent-neural-networks, long-short-term-memory
  > **Comments:**
  > **username_1**: u need to do some feature engineering possibly  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-01-24. Score: 1
  
  One thing I might add is not to expect the ML model to take care of everything, that is where ML engineering comes into play. One suggestion I have, without knowing much about your data stream, is that you need to implement a real-time data ingestion/pipeline, like [Apache Kafka Stream](https://kafka.apache.org/documentation/streams/) (so similar implementation), where you can decompose your stream into output that your model was trained for. You could split by dictionary, pattern or whatever you need, then push the split output into another real-time stream that can be ingested by your model. Not a lot to go for, but hopefully it helps a little.
  
  > **Comments:**
  > **username_0**: Hi, thanks. The quetion is still what method to use to decompose the signal. I was thinkig of some sort of convolution, but I have no idea yet how exactly it would help me.  
  
  ---
  
  