Answers.Body: <p>You can claim to use a real-world dataset, you would just need to specify that some values were interpolated.</p>
  <p>Do you have to have the inter-mediate values though? By the looks of it, each &quot;region&quot; was only measured every 2 hours, so I would just keep it that way and just have the resolution be 2 hours. It doesn't have to be hourly, and probably shouldn't since that isn't the resolution of the data by the looks of it.</p>
  <p>If it does need to be hourly then it is fine to just linearly interpolate the data. Additionally, you can try and train the network to accept empty inputs (though It'd definitely be easier to just interpolate your dataset)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-13T01:39:56.953Z
Answers.Text: Thank you for your explanation. According to your advice, Is it ok that I use my dataset with missing value estimation to submit a paper?
  
  Can I say that this is not data manipulation?
Answers.UserDisplayName: Dae-Young Park
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-13T02:15:53.503Z
Answers.Text: @Dae-YoungPark I can't say for sure because I don't have the full dataset, but it doesn't look like there is any missing data. It just looks like the measurements were taken every 2 hours (for each region), which is fine. It's the same as measurements being taken every minute, just because there isn't data at every second doesn't mean there's missing data - it just means the resolution is minutely.
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-13T02:34:34.207Z
Answers.Text: The measurements is not taken every 2 hours. Actually, About Region1, there is a row with Hour=3. Thus, I think there are some missing values. In such a case, can I claim to use a real-world dataset even though some data are interpolated in data science area?
Answers.UserDisplayName: Dae-Young Park
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-13T07:25:01.180Z
Answers.Text: If you're using real-world data then it is a real-world dataset. If you can still draw representative conclusions about the data, what difference does it make?
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-13T01:18:24.980Z
Answers.LastActivityDate: 2021-07-13T01:18:24.980Z
Answers.UserDisplayName: Recessive
Body: <p>I'm a student and writing my first paper for submission on conference. I have a question</p>
  <p>there is a dataset below. this is temporal-spatial dataset.</p>
  <pre><code>Date         Hour   City       Sensor1  Sensor2  Sensor3 Sensor4 ...
  21-06-10     0      Region1      0.12     0.52    0.33     0.44  ...
  21-06-10     1      Region2      0.16     0.83    0.34     0.49  ...
  21-06-10     2      Region1      0.21     0.44    0.57     0.5   ...
  ...
  </code></pre>
  <p>My Task is anomaly detection for each region</p>
  <p>I want to use LSTM. So, I represent the temporal-spatial data to two time-series data. my dataset can be represented below.</p>
  <pre><code>City       Date       Hour     Sensor1  Sensor2  Sensor3 Sensor4 ...
  Region1   21-06-10     0         0.12     0.52    0.33     0.44  ...
  Region1   21-06-10     2         0.21     0.44    0.57     0.5   ...
  ...
  
  
  City       Date       Hour     Sensor1  Sensor2  Sensor3 Sensor4 ...
  Region2   21-06-10     1         0.16     0.83    0.34     0.49  ...
  ...
  </code></pre>
  <p>However, then, there is no a row with attribute '<em>Hour=1</em>' in Region1 dataset
  (you can see the table below)</p>
  <pre><code>City       Date       Hour     Sensor1  Sensor2  Sensor3 Sensor4 ...
  Region1   21-06-10     0         0.12     0.52    0.33     0.44  ...
  Region1   21-06-10     1         NaN      NaN     NaN      NaN   ...
  Region1   21-06-10     2         0.21     0.44    0.57     0.5   ...
  ...
  </code></pre>
  <p>Can I insert estimated values into the row with attribute '<em>Hour=1</em>' in Region1 dataset? (for example, I want to insert average between the first row and the third row)</p>
  <p>Can I claim to have utilized a real world dataset even with this missing value estimation?</p>
  
Tags: <machine-learning><long-short-term-memory><datasets><data-preprocessing><data-science>
Title: How can I address missing values for LSTM?
UserDisplayName: Dae-Young Park
ThreadText: # How can I address missing values for LSTM?
  
  (Asked by: username_0 on 2021-07-13)
  
  I'm a student and writing my first paper for submission on conference. I have a question
  there is a dataset below. this is temporal-spatial dataset.
  Date         Hour   City       Sensor1  Sensor2  Sensor3 Sensor4 ...
  21-06-10     0      Region1      0.12     0.52    0.33     0.44  ...
  21-06-10     1      Region2      0.16     0.83    0.34     0.49  ...
  21-06-10     2      Region1      0.21     0.44    0.57     0.5   ...
  ...
  
  My Task is anomaly detection for each region
  I want to use LSTM. So, I represent the temporal-spatial data to two time-series data. my dataset can be represented below.
  City       Date       Hour     Sensor1  Sensor2  Sensor3 Sensor4 ...
  Region1   21-06-10     0         0.12     0.52    0.33     0.44  ...
  Region1   21-06-10     2         0.21     0.44    0.57     0.5   ...
  ...
  
  
  City       Date       Hour     Sensor1  Sensor2  Sensor3 Sensor4 ...
  Region2   21-06-10     1         0.16     0.83    0.34     0.49  ...
  ...
  
  However, then, there is no a row with attribute 'Hour=1' in Region1 dataset
  (you can see the table below)
  City       Date       Hour     Sensor1  Sensor2  Sensor3 Sensor4 ...
  Region1   21-06-10     0         0.12     0.52    0.33     0.44  ...
  Region1   21-06-10     1         NaN      NaN     NaN      NaN   ...
  Region1   21-06-10     2         0.21     0.44    0.57     0.5   ...
  ...
  
  Can I insert estimated values into the row with attribute 'Hour=1' in Region1 dataset? (for example, I want to insert average between the first row and the third row)
  Can I claim to have utilized a real world dataset even with this missing value estimation?
  
  **Tags:** machine-learning, long-short-term-memory, datasets, data-preprocessing, data-science
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-07-13. Score: 2 | Accepted answer
  
  You can claim to use a real-world dataset, you would just need to specify that some values were interpolated.
  
  Do you have to have the inter-mediate values though? By the looks of it, each "region" was only measured every 2 hours, so I would just keep it that way and just have the resolution be 2 hours. It doesn't have to be hourly, and probably shouldn't since that isn't the resolution of the data by the looks of it.
  
  If it does need to be hourly then it is fine to just linearly interpolate the data. Additionally, you can try and train the network to accept empty inputs (though It'd definitely be easier to just interpolate your dataset)
  
  > **Comments:**
  > **username_0**: Thank you for your explanation. According to your advice, Is it ok that I use my dataset with missing value estimation to submit a paper?  Can I say that this is not data manipulation?  
  > **username_1**: @Dae-YoungPark I can't say for sure because I don't have the full dataset, but it doesn't look like there is any missing data. It just looks like the measurements were taken every 2 hours (for each region), which is fine. It's the same as measurements being taken every minute, just because there isn't data at every second doesn't mean there's missing data - it just means the resolution is minutely.  
  > **username_0**: The measurements is not taken every 2 hours. Actually, About Region1, there is a row with Hour=3. Thus, I think there are some missing values. In such a case, can I claim to use a real-world dataset even though some data are interpolated in data science area?  
  > **username_1**: If you're using real-world data then it is a real-world dataset. If you can still draw representative conclusions about the data, what difference does it make?  
  
  ---
  
  