Answers.Body: <p>Your idea is a good one.</p>
  <p>Another idea is to upsample or aggregate your data.  For example, average by week if you generally have a couple of missing days in every week.</p>
  <p>A similar question on Stack Exchange:
  <a href="https://stats.stackexchange.com/questions/374935/how-to-deal-with-really-sparse-time-series-data-for-a-binary-classification-task">https://stats.stackexchange.com/questions/374935/how-to-deal-with-really-sparse-time-series-data-for-a-binary-classification-task</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-06T13:28:38.153Z
Answers.LastActivityDate: 2020-07-06T13:28:38.153Z
Answers.UserDisplayName: Brian O'Donnell
Body: <p>I am using LSTM network for predicting IOT time-series data receiving from un-reliable devices and networks.<br>
  This results in several multiple sections [continuous streak of bad data for several days until the problem is fixed].<br>
  I need to exclude this bad data section before feeding it to model training.<br>
  Since I am using LSTM-RNN network, it requires to do an un-roll data based on the previous records.   </p>
  
  <p>How can I properly exclude this bad data?<br>
  I thought of an approach as training model separately using each batch of good data, and use subsequent good-data batch for fine-tuning the model.<br>
  Please let me know if this is a good approach? or is there a better method?</p>
  
  <pre><code>example data:
  "1-01",266.0
  "1-02",145.9
  "1-03",183.1
  "1-04",0  [bad data]
  "1-05",0  [bad data]
  "1-06",0  [bad data]
  "1-07",0  [bad data]
  "1-08",224.5
  "1-09",192.8
  "1-10",122.9
  "1-11",0  [bad data]
  "1-12",0  [bad data]
  "2-01",194.3
  "2-02",149.5
  </code></pre>
  
Tags: <recurrent-neural-networks><long-short-term-memory><time-series>
Title: How to exclude sections of bad data from time-series data before training an LSTM network
UserDisplayName: Remis Haroon - رامز
ThreadText: # How to exclude sections of bad data from time-series data before training an LSTM network
  
  (Asked by: username_0 on 2020-06-01)
  
  I am using LSTM network for predicting IOT time-series data receiving from un-reliable devices and networks.  
  This results in several multiple sections [continuous streak of bad data for several days until the problem is fixed].  
  I need to exclude this bad data section before feeding it to model training.  
  Since I am using LSTM-RNN network, it requires to do an un-roll data based on the previous records.
  
  How can I properly exclude this bad data?  
  I thought of an approach as training model separately using each batch of good data, and use subsequent good-data batch for fine-tuning the model.  
  Please let me know if this is a good approach? or is there a better method?
  
  ```
  example data:
  "1-01",266.0
  "1-02",145.9
  "1-03",183.1
  "1-04",0  [bad data]
  "1-05",0  [bad data]
  "1-06",0  [bad data]
  "1-07",0  [bad data]
  "1-08",224.5
  "1-09",192.8
  "1-10",122.9
  "1-11",0  [bad data]
  "1-12",0  [bad data]
  "2-01",194.3
  "2-02",149.5
  
  ```
  
  **Tags:** recurrent-neural-networks, long-short-term-memory, time-series
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-06. Score: 1
  
  Your idea is a good one.
  
  Another idea is to upsample or aggregate your data. For example, average by week if you generally have a couple of missing days in every week.
  
  A similar question on Stack Exchange:
  <https://stats.stackexchange.com/questions/374935/how-to-deal-with-really-sparse-time-series-data-for-a-binary-classification-task>
  
  
  ---
  
  