<p>Your idea is a good one.</p>
<p>Another idea is to upsample or aggregate your data.  For example, average by week if you generally have a couple of missing days in every week.</p>
<p>A similar question on Stack Exchange:
<a href="https://stats.stackexchange.com/questions/374935/how-to-deal-with-really-sparse-time-series-data-for-a-binary-classification-task">https://stats.stackexchange.com/questions/374935/how-to-deal-with-really-sparse-time-series-data-for-a-binary-classification-task</a></p>

CC BY-SA 4.0
2020-07-06T13:28:38.153Z
2020-07-06T13:28:38.153Z
Brian O'Donnell
<p>I am using LSTM network for predicting IOT time-series data receiving from un-reliable devices and networks.<br>
This results in several multiple sections [continuous streak of bad data for several days until the problem is fixed].<br>
I need to exclude this bad data section before feeding it to model training.<br>
Since I am using LSTM-RNN network, it requires to do an un-roll data based on the previous records.   </p>

<p>How can I properly exclude this bad data?<br>
I thought of an approach as training model separately using each batch of good data, and use subsequent good-data batch for fine-tuning the model.<br>
Please let me know if this is a good approach? or is there a better method?</p>

<pre><code>example data:
"1-01",266.0
"1-02",145.9
"1-03",183.1
"1-04",0  [bad data]
"1-05",0  [bad data]
"1-06",0  [bad data]
"1-07",0  [bad data]
"1-08",224.5
"1-09",192.8
"1-10",122.9
"1-11",0  [bad data]
"1-12",0  [bad data]
"2-01",194.3
"2-02",149.5
</code></pre>

<recurrent-neural-networks><long-short-term-memory><time-series>
How to exclude sections of bad data from time-series data before training an LSTM network
Remis Haroon - رامز
# How to exclude sections of bad data from time-series data before training an LSTM network

(Asked by: username_0 on 2020-06-01)

I am using LSTM network for predicting IOT time-series data receiving from un-reliable devices and networks.
This results in several multiple sections [continuous streak of bad data for several days until the problem is fixed].
I need to exclude this bad data section before feeding it to model training.
Since I am using LSTM-RNN network, it requires to do an un-roll data based on the previous records.   
How can I properly exclude this bad data?
I thought of an approach as training model separately using each batch of good data, and use subsequent good-data batch for fine-tuning the model.
Please let me know if this is a good approach? or is there a better method?
example data:
"1-01",266.0
"1-02",145.9
"1-03",183.1
"1-04",0  [bad data]
"1-05",0  [bad data]
"1-06",0  [bad data]
"1-07",0  [bad data]
"1-08",224.5
"1-09",192.8
"1-10",122.9
"1-11",0  [bad data]
"1-12",0  [bad data]
"2-01",194.3
"2-02",149.5

**Tags:** recurrent-neural-networks, long-short-term-memory, time-series


---

## 1 answers

### Answer by username_1 on 2020-07-06. Score: 1

Your idea is a good one.

Another idea is to upsample or aggregate your data. For example, average by week if you generally have a couple of missing days in every week.

A similar question on Stack Exchange:
<https://stats.stackexchange.com/questions/374935/how-to-deal-with-really-sparse-time-series-data-for-a-binary-classification-task>


---

