Answers.Body: <p>First, if your data has a minimum of 0 and maximum of 5000, 1000 will get rescaled to .2 and 5000 will get rescaled to 1. So it's not a .001 difference as you suggest.</p>
  <p>If you just used a regular loss function (e.g. mean squared error), I'm not sure you'd be able to achieve good predictions. In the literature this type of data might be called &quot;sharp&quot;. There are some special loss functions such as DILATE <a href="https://papers.nips.cc/paper/2019/file/466accbac9a66b805ba50e42ad715740-Paper.pdf" rel="nofollow noreferrer">(link)</a> or soft-DTW <a href="http://proceedings.mlr.press/v70/cuturi17a/cuturi17a.pdf" rel="nofollow noreferrer">(link)</a> which are specially designed for time series like the one you show.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-21T19:56:37.813Z
Answers.Text: oh thanks so much for pointing these papers out to me. I have seen the work of Cuturi before, especially with respect to optimal transport, but not with respect to time series. So that is excellent. I will check out these alternative loss functions. Thanks again.
Answers.UserDisplayName: krishnab
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-21T14:57:06.050Z
Answers.LastActivityDate: 2021-05-21T14:57:06.050Z
Answers.UserDisplayName: Taw
Body: <p>I am working with some time-series hydrology data. Our goal is to forecast the time series forward, meaning predicting the data 1 month, 3 months ,6 months into the future. The data itself(image below) is characterized by mostly 0 or very small rates of flow expect for brief periods that are characterized by high flow. So I get this crazy spiky pattern where the median is around 0 or 1-2 meters^3/min, but at the same time there are periods of 5000 meters^3/minute, etc. I am not sure of the exact scale dimensions, but the picture below tells the tale.</p>
  <p><a href="https://i.sstatic.net/8KgN8.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/8KgN8.png" alt="enter image description here" /></a></p>
  <p>So I was trying to figure out a good way to scale this type of spiky data. I have been using a MinMaxScaler just to start with, to rescale the values between (-1, 1). But that approach is not going to work well, especially because at the top ends of the range, the difference between 1000 m^3/min and 5000 m^3/min will be like 0.001 difference.</p>
  <p>Does anyone have a good suggestion of how to rescale data like this for time-series analysis in an LSTM or RNN network?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-19T18:00:14.347Z
Comments.Text: What is your objective? prediction or detect the spikes?
Comments.UserDisplayName: xsari3x
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-19T18:02:00.740Z
Comments.Text: I want to get accurate predictions of future values. Sorry was I unclear about that?
Comments.UserDisplayName: krishnab
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-21T13:15:54.033Z
Comments.Text: And what is your input?
Comments.UserDisplayName: xsari3x
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-21T13:48:23.770Z
Comments.Text: @xsari3x our input is the image above, meaning it is the daily hydrological flow data. There are daily measurements.
Comments.UserDisplayName: krishnab
Tags: <recurrent-neural-networks><long-short-term-memory><data-preprocessing><time-series>
Title: Rescaling time-series data with very spiky pattern for training data in LSTM network
UserDisplayName: krishnab
ThreadText: # Rescaling time-series data with very spiky pattern for training data in LSTM network
  
  (Asked by: username_0 on 2021-05-19)
  
  I am working with some time-series hydrology data. Our goal is to forecast the time series forward, meaning predicting the data 1 month, 3 months ,6 months into the future. The data itself(image below) is characterized by mostly 0 or very small rates of flow expect for brief periods that are characterized by high flow. So I get this crazy spiky pattern where the median is around 0 or 1-2 meters^3/min, but at the same time there are periods of 5000 meters^3/minute, etc. I am not sure of the exact scale dimensions, but the picture below tells the tale.
  
  [![enter image description here](https://i.sstatic.net/8KgN8.png)](https://i.sstatic.net/8KgN8.png)
  
  So I was trying to figure out a good way to scale this type of spiky data. I have been using a MinMaxScaler just to start with, to rescale the values between (-1, 1). But that approach is not going to work well, especially because at the top ends of the range, the difference between 1000 m^3/min and 5000 m^3/min will be like 0.001 difference.
  
  Does anyone have a good suggestion of how to rescale data like this for time-series analysis in an LSTM or RNN network?
  
  **Tags:** recurrent-neural-networks, long-short-term-memory, data-preprocessing, time-series
  
  > **Comments:**
  > **username_1**: What is your objective? prediction or detect the spikes?  
  > **username_0**: I want to get accurate predictions of future values. Sorry was I unclear about that?  
  > **username_1**: And what is your input?  
  > **username_0**: @username_1 our input is the image above, meaning it is the daily hydrological flow data. There are daily measurements.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-05-21. Score: 3 | Accepted answer
  
  First, if your data has a minimum of 0 and maximum of 5000, 1000 will get rescaled to .2 and 5000 will get rescaled to 1. So it's not a .001 difference as you suggest.
  
  If you just used a regular loss function (e.g. mean squared error), I'm not sure you'd be able to achieve good predictions. In the literature this type of data might be called "sharp". There are some special loss functions such as DILATE [(link)](https://papers.nips.cc/paper/2019/file/466accbac9a66b805ba50e42ad715740-Paper.pdf) or soft-DTW [(link)](http://proceedings.mlr.press/v70/cuturi17a/cuturi17a.pdf) which are specially designed for time series like the one you show.
  
  > **Comments:**
  > **username_0**: oh thanks so much for pointing these papers out to me. I have seen the work of Cuturi before, especially with respect to optimal transport, but not with respect to time series. So that is excellent. I will check out these alternative loss functions. Thanks again.  
  
  ---
  
  