Answers.Body: <p>Mathematically speaking when dealing with <a href="https://en.wikipedia.org/wiki/Fat-tailed_distribution" rel="nofollow noreferrer">fat tailed distributions</a> or values that varies in a range of several order of magnitude the most common and simplest solution is to move to logarithmic space. Of course that requires positive values before scaling the data, and the log of a number can be negative as well, so you might want to normalize your data before and after. Something like:</p>
  <ul>
  <li>normalize data to be positive</li>
  <li>move to log space to reduce skewness and magnitude range of the data</li>
  <li>normalize again in the proper input range for your model</li>
  </ul>
  <p>regarding your ideas:</p>
  <ul>
  <li>applying moving average is always a good preprocessing step when dealing with temporal data, but that's more noise cleaning not really data scaling/normalization</li>
  <li>splitting mantissa and exponent sounds like a too easy trick to work, plus it doesn't really mathematically tackle the skewness issue.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-29T07:42:27.010Z
Answers.Text: Log is a great idea! But after log is applied, the small changes in value will become even smaller. How many training parameters on the 1th (and may be 2th) layers do I need to make the network able to sence small changes? Also, my data goes straight into LSTM. Would it be better to introduce a couple of dense layers (operating on each single LSTM input element/row) as a data preprocessor? Thank you!
Answers.UserDisplayName: Boris Brodski
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-29T09:20:47.660Z
Answers.Text: I wouldn't advice to add dense layers, time series are really specific type of data and there are many well known practices to preprocess them, so I would look into them first. For example identifying the trend-seasonality-rest components, investigating and removing collinearity if you're using multiple variables and so on.
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-29T09:28:45.003Z
Answers.Text: if you have reasons to focus the model on really small changes (which be aware to me sounds like overfitting noise rather than predicting a trend) you could play with the forget gate of the LSTM. Usually a normal sigmoid $1 / 1 + e^{-x}$ is applied to the input and previous cell data. By adding the temperature parameter you can tune how much is forget and how much is retained from past cells $1 / 1 + e^{-x/t}$. Specifically, you want a t value smaller than 1 (so the sigmoid get squeezed close to the y axis, which means more values will be pushed to zero and forget).
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-26T09:08:23.183Z
Answers.LastActivityDate: 2022-04-26T09:08:23.183Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I have a bunch of arbitrary float numbers (asset prices), that I have to feed into a neural network.</p>
  <ul>
  <li>In the data set: values are between 1E-10 and 1E6</li>
  <li>In a single sample: values may differ by a factor 100</li>
  <li>Even slight changes (price movements) are important</li>
  </ul>
  <p>I tried a naive &quot;min/max&quot; normalization to the range [0..1], but the performance of the model degrades quickly if the values within a single sample are changing more than by a factor ~5. (The performance is determined by using predictions in a trading-simulation)</p>
  <p>My own ideas:</p>
  <ul>
  <li>Provide diffs from one float to the next one (change in price from <code>t</code> to <code>t+1</code>).</li>
  <li>Split each float number into [0..1] mantissa and exponent
  <ul>
  <li>Exponent as one-hot-vector mapping values from -10 to 6</li>
  <li>Exponent min/max normalized to [0..1]</li>
  </ul>
  </li>
  </ul>
  <p><strong>How should I correctly normalize such a data to feed into AI?</strong></p>
  <p><em>Requested details of the network</em>:</p>
  <ul>
  <li>What are you trying to predict?
  <ul>
  <li>Next candle</li>
  <li>Expected gain of a long position / raise in price in the next ~5 candles</li>
  </ul>
  </li>
  <li>How are you assessing the performance of the model?
  <ul>
  <li>I'm using a trading simulation with predictions on fresh data</li>
  </ul>
  </li>
  <li>Which model?
  <ul>
  <li>Multiple parallel 3 layers-LSTMs</li>
  <li>Each LSTM process separate consecutive chunk of candles</li>
  </ul>
  </li>
  <li>A regular feedforward neural network?
  <ul>
  <li>Yes, multiple inputs goes though LSTMs</li>
  <li>Then concatenate to single vector</li>
  <li>Plus a bunch of dense layers</li>
  </ul>
  </li>
  <li>With how many layers and which activation functions?
  <ul>
  <li>activation functions: <code>relu</code>, <code>sigmoid</code> (for output)</li>
  <li>I played with different configurations, e.g.
  <ul>
  <li>~ 15 layers total</li>
  <li>500.000 trainable params</li>
  </ul>
  </li>
  </ul>
  </li>
  <li>Which loss function are you using?
  <ul>
  <li>mean_squared_error</li>
  </ul>
  </li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-23T09:12:14.127Z
Comments.Text: What do you mean by "it performs ok" or not, in practice? What are you trying to predict? How are you assessing the performance of the model? Which model? A regular feedforward neural network? With how many layers and which activation functions? Which loss function are you using? Edit your post to include these details, which could be useful.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-25T06:55:25.653Z
Comments.Text: Added all requested information.
Comments.UserDisplayName: Boris Brodski
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-25T11:00:20.010Z
Comments.Text: @nbro please, let me know, if more information is needed.
Comments.UserDisplayName: Boris Brodski
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-25T13:05:15.227Z
Comments.Text: Thanks for the details. I am not sure I can help you in this case, because I don't remember having solved a similar problem. However, the range of your data is really wide, and this can indeed make the task more difficult. You could try using a different configuration of the neural network. You have 15 layers, but "only" 500000 parameters. That doesn't seem right. A model with 15 layers can quickly have more parameters. Well, it also depends on the hyper-parameters of these layers (like width).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-25T15:19:51.777Z
Comments.Text: @nbro Ah! You mean, the small amount of parameters per layer may cause problems by sensing small changes in the input data? Should I invest time and try to go with the `Split each float number into [0..1] mantissa and exponent` approach? Thank you!
Comments.UserDisplayName: Boris Brodski
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-25T15:48:20.480Z
Comments.Text: No, I didn't mean that. I meant that, if you have 15 layers, I would expect that the total number of parameters to be higher. Are you sure that the total number of parameters is just 500000? That was my question. Anyway, having 15 layers seems a big number of layers, and it can lead to vanishing/exploding gradients, although if you're using LSTMs, these issues should be mitigated to some extent, at least, compared to vanilla RNNs. Regarding your 2nd approach, I don't really know. Your first one looks more promising to me, but this is just a guess.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-29T07:31:26.360Z
Comments.Text: @nbro Yes, currently ~500,000. But I can easily increase it. Also my network is only 12 layers deep becaues of 2 output vectors. (Last 3 layers goes in parallel). Which number of training parameters would sound appropriate to you?  Thank you!
Comments.UserDisplayName: Boris Brodski
Tags: <data-preprocessing><batch-normalization>
Title: Normalizing float prices with movements up to a factor of 100
UserDisplayName: Boris Brodski
ThreadText: # Normalizing float prices with movements up to a factor of 100
  
  (Asked by: username_0 on 2022-04-16)
  
  I have a bunch of arbitrary float numbers (asset prices), that I have to feed into a neural network.
  
  * In the data set: values are between 1E-10 and 1E6
  * In a single sample: values may differ by a factor 100
  * Even slight changes (price movements) are important
  
  I tried a naive "min/max" normalization to the range [0..1], but the performance of the model degrades quickly if the values within a single sample are changing more than by a factor ~5. (The performance is determined by using predictions in a trading-simulation)
  
  My own ideas:
  
  * Provide diffs from one float to the next one (change in price from `t` to `t+1`).
  * Split each float number into [0..1] mantissa and exponent
    + Exponent as one-hot-vector mapping values from -10 to 6
    + Exponent min/max normalized to [0..1]
  
  **How should I correctly normalize such a data to feed into AI?**
  
  *Requested details of the network*:
  
  * What are you trying to predict?
    + Next candle
    + Expected gain of a long position / raise in price in the next ~5 candles
  * How are you assessing the performance of the model?
    + I'm using a trading simulation with predictions on fresh data
  * Which model?
    + Multiple parallel 3 layers-LSTMs
    + Each LSTM process separate consecutive chunk of candles
  * A regular feedforward neural network?
    + Yes, multiple inputs goes though LSTMs
    + Then concatenate to single vector
    + Plus a bunch of dense layers
  * With how many layers and which activation functions?
    + activation functions: `relu`, `sigmoid` (for output)
    + I played with different configurations, e.g.
      - ~ 15 layers total
      - 500.000 trainable params
  * Which loss function are you using?
    + mean\_squared\_error
  
  **Tags:** data-preprocessing, batch-normalization
  > **Comments:**
  > **username_1**: What do you mean by "it performs ok" or not, in practice? What are you trying to predict? How are you assessing the performance of the model? Which model? A regular feedforward neural network? With how many layers and which activation functions? Which loss function are you using? Edit your post to include these details, which could be useful.  
  > **username_0**: Added all requested information.  
  > **username_0**: @nbro please, let me know, if more information is needed.  
  > **username_1**: Thanks for the details. I am not sure I can help you in this case, because I don't remember having solved a similar problem. However, the range of your data is really wide, and this can indeed make the task more difficult. You could try using a different configuration of the neural network. You have 15 layers, but "only" 500000 parameters. That doesn't seem right. A model with 15 layers can quickly have more parameters. Well, it also depends on the hyper-parameters of these layers (like width).  
  > **username_0**: @nbro Ah! You mean, the small amount of parameters per layer may cause problems by sensing small changes in the input data? Should I invest time and try to go with the `Split each float number into [0..1] mantissa and exponent` approach? Thank you!  
  > **username_1**: No, I didn't mean that. I meant that, if you have 15 layers, I would expect that the total number of parameters to be higher. Are you sure that the total number of parameters is just 500000? That was my question. Anyway, having 15 layers seems a big number of layers, and it can lead to vanishing/exploding gradients, although if you're using LSTMs, these issues should be mitigated to some extent, at least, compared to vanilla RNNs. Regarding your 2nd approach, I don't really know. Your first one looks more promising to me, but this is just a guess.  
  > **username_0**: @nbro Yes, currently ~500,000. But I can easily increase it. Also my network is only 12 layers deep becaues of 2 output vectors. (Last 3 layers goes in parallel). Which number of training parameters would sound appropriate to you?  Thank you!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-04-26. Score: 2 | Accepted answer
  
  Mathematically speaking when dealing with [fat tailed distributions](https://en.wikipedia.org/wiki/Fat-tailed_distribution) or values that varies in a range of several order of magnitude the most common and simplest solution is to move to logarithmic space. Of course that requires positive values before scaling the data, and the log of a number can be negative as well, so you might want to normalize your data before and after. Something like:
  
  * normalize data to be positive
  * move to log space to reduce skewness and magnitude range of the data
  * normalize again in the proper input range for your model
  
  regarding your ideas:
  
  * applying moving average is always a good preprocessing step when dealing with temporal data, but that's more noise cleaning not really data scaling/normalization
  * splitting mantissa and exponent sounds like a too easy trick to work, plus it doesn't really mathematically tackle the skewness issue.
  
  > **Comments:**
  > **username_0**: Log is a great idea! But after log is applied, the small changes in value will become even smaller. How many training parameters on the 1th (and may be 2th) layers do I need to make the network able to sence small changes? Also, my data goes straight into LSTM. Would it be better to introduce a couple of dense layers (operating on each single LSTM input element/row) as a data preprocessor? Thank you!  
  > **username_2**: I wouldn't advice to add dense layers, time series are really specific type of data and there are many well known practices to preprocess them, so I would look into them first. For example identifying the trend-seasonality-rest components, investigating and removing collinearity if you're using multiple variables and so on.  
  > **username_2**: if you have reasons to focus the model on really small changes (which be aware to me sounds like overfitting noise rather than predicting a trend) you could play with the forget gate of the LSTM. Usually a normal sigmoid $1 / 1 + e^{-x}$ is applied to the input and previous cell data. By adding the temperature parameter you can tune how much is forget and how much is retained from past cells $1 / 1 + e^{-x/t}$. Specifically, you want a t value smaller than 1 (so the sigmoid get squeezed close to the y axis, which means more values will be pushed to zero and forget).  
  
  ---
  
  