Answers.Body: <p>This is a good question which I wrestled with myself when first trying to code an ANN. </p>
  
  <p>Below is a good general-purpose solution, and it's the one I implemented in my code for trying to predict well-behaved numerical data. If your data is not well-behaved (i.e. fraught with outliers) then you may need to do more work normalizing the inputs and outputs. Some of the more advanced methods are described <a href="https://www.cs.ccu.edu.tw/~wylin/BA/Fusion_of_Biometrics_II.ppt" rel="noreferrer">here</a>.</p>
  
  <p><em>Note: I will assume that you are using f(x) = tanh(x) as your activation function. If you aren't, you should still be able to reason through how to normalize your data after reading this.</em></p>
  
  <p><strong>How to prepare the input data:</strong></p>
  
  <p>The basic idea is that you want a significant variation in each input parameter to be reflected by a significant variation in the activation of the neuron those inputs are being fed into. By looking at a plot of the derivative of the tanh(x) actiavtion function, you'll see that the region of significant slope is within a distance of one or two from the origin. This means that whether the input to the activation function is 2000 or 3000 (values of x for which the derivative is negligibly small), the output of the activation will be almost identical...so your neuron's state will be independent of the difference between 2000 and 3000, and your network will never produce any predictive power from values in that range. </p>
  
  <p>So if you want to input the square footage of the house into a neuron, you need to <em>normalize</em> the square footage so that the network can tell the difference between 2000 and 3000. One way to do this so that all of the significant variations in your data are 'noticed' by the neuron is to <strong>z-score-normalize the inputs</strong>. </p>
  
  <ul>
  <li><p>Gather all of your square footage values (from your training set) and calculate the mean and standard deviation. <em>Store the mean and standard deviation</em>---you'll need this information to normalize new square footage values when testing. </p></li>
  <li><p><strong>Normalize the vector of square footage values by subtracting the mean and then dividing the result by the standard deviation</strong> (all operations element-wise of course). Subtracting the mean centers your data at the origin, and dividing by the standard deviation makes sure most of it is between -1 and 1, where the neuron's output is most sensitive to its input. This is called z-score normalization because each input value is replaced by its <a href="https://en.wikipedia.org/wiki/Standard_score" rel="noreferrer">z-score</a>.</p></li>
  <li><p><strong>Do the above for each input variable.</strong></p></li>
  </ul>
  
  <p>Now, when you put each input value through a neuron, the output of the neuron is an activation between -1 and 1 (look at the image of tanh(x)). Since this is already in the 'sensitive' range of the activation function, you don't need to worry about altering the output of the input-layer neurons before sending them to the first hidden layer. Just <strong>give any hidden layer neurons the outputs of the previous layer directly</strong>---they will be able to handle them just fine. </p>
  
  <p><strong>When you reach the last layer (the output neuron(s)), what you get is again another activation between -1 and 1. You have to convert this back into a value for the house in question</strong>, whether that value will be used as a prediction in a test set or to calculate error during training. However you do this, you just have to be consistent and use the same de-normalization procedure in training and testing. One way to think about it is: when the output neuron(s) returns 1, that means the network is returning the <em>maximum possible house value</em> as its prediction. <em>What should the highest value the network can estimate be?</em> The right approach here simply depends on your application. This is what I did:</p>
  
  <ul>
  <li>Calculate the mean of [the/each] output variable and store it.</li>
  <li>Calculate the maximum deviation of the output variable from the mean. 
  Python: <code>MaxDev = max([abs(DataPoint-numpy.mean(TrainingData)) for DataPoint in TrainingData])</code></li>
  <li>When the network returns output(s) between -1 and 1, multiply the output by <code>MaxDev</code> and add it to the mean.</li>
  </ul>
  
  <p>Two basic quick checks you can do to see if your normalization-renormalization scheme is suitable (these are necessary, but perhaps not sufficient conditions):</p>
  
  <ol>
  <li>If all the input values are average (e.g. average no. of bedrooms, average sq.feet, etc), is the network's output equal to the average of the output variable (e.g. house value) as well? (It should be.)</li>
  <li>If all the input values are unusually high/low, is the network's output unusually high/low as well? (This only works if all the inputs are positively related to the output...if some of them are inversely related related, you will have to think a bit more).</li>
  </ol>
  
  <p>Observe that the scheme presented here satisfies these two conditions.</p>
  
  <p>Notice that this scheme would allow your network to only predict house values <em>inside</em> the range of house values in your training data set. Depending on the application, this behavior can be desirable or undesirable.</p>
  
  <p>For example: you may want to make it impossible for your network to predict negative house values. Think about how you would do this. De-normalize the output so that -1 is mapped to 0.</p>
  
  <p>If you want to set no limit on the values your network can predict, then you can run the network's output through a function that maps the [-1,1] range to all real numbers...like arctanh(x)! As long as you do this during training your network will adjust its weights to accommodate this.</p>
  
  <p>I hope this was helpful. Let me know if you have further questions. My ANN module is in Python, by the way, so I might have language-specific advice.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-27T19:17:30.673Z
Answers.Text: "By looking at a plot of the derivative of the tanh(x)"  Dont understund, why should I be lookning at derivative of base function, when we input is mulitplied by by weight and then passed to base funcion eg. RELU. Could u explain?
Answers.UserDisplayName: Phate P
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-28T20:29:52.527Z
Answers.Text: Because the weight only scales the derivative by a constant value, it doesn’t affect the insight that was sought here
Answers.UserDisplayName: Marko Bakić
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-02-08T09:07:09.670Z
Answers.Text: This was very useful! Every blog/tutorial I come across seems to avoid (almost deliberately) describing this process, but yes that all makes sense. It'll take a while for me to digest properly but I'll be back if I have any follow up questions. Much obliged!
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-02-08T11:32:33.740Z
Answers.Text: So couple of questions. If my Sq. Feet training data was { 2000, 800, 850, 550, 2000 } then my z-score inputs for { 1900, 1500, 600 } would be (if I've calculated correctly) { 1.0496, 0.4134, -1.0177 }. So one of those values is > 1 and one is < -1, what would I do with those? Input them into the input layer nodes regardless or round them to 1 & -1? Why does 1900 & 600 produce those values when they are within the 550 - 2000 range? Is this just a trick of the data because there is such a small data set?
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-02-08T11:37:06.107Z
Answers.Text: With regards to output layer renormalization, do I have it correctly that you'd plot the output to the min and max values? So if the minimum value was $0 and the maximum $100 and the output was zero (assuming -1 to 1) then that would translate as $50?
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-02-08T18:02:47.717Z
Answers.Text: Remember that the inputs don't need to be strictly between 1 and -1. All you need for the inputs is that most of the data is in that range. A value greater or less than one means that point is more than one standard deviation away from the mean, so that point is closer to the higher end of the data. It should be a bit rare for your data to go outside of [-1, 1], even more rare for it to go outside of [-2, 2] and extremely rare to go outside of [-3, 3]. Look at tanh(x) and you will see the sensitive range isn't just strictly between -1 and 1, but goes a bit further out than that.
Answers.UserDisplayName: Marko Bakić
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-02-08T18:05:01.273Z
Answers.Text: Regarding the output denormalization, that min-max denormalization is what I did in my implementation, and your interpretation is correct, but you don't necessarily have to do that. You could make it so that 1 corresponds to twice the maximum house value--that way your network would be able to predict house values above what you trained it on.
Answers.UserDisplayName: Marko Bakić
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-02-08T08:45:44.100Z
Answers.LastActivityDate: 2017-02-08T08:45:44.100Z
Answers.UserDisplayName: Marko Bakić
Body: <p>So I've been trying to understand neural networks ever since I came across <a href="https://medium.com/@ageitgey/machine-learning-is-fun-80ea3ec3c471" rel="noreferrer">Adam Geitgey's</a> blog on machine learning. I've read as much as I can on the subject (that I can grasp) and believe I understand all the broad concepts and some of the workings (despite being very weak in maths), neurons, synapses, weights, cost functions, backpropagation etc. However, I've not been able to figure out how to translate real world problems into a neural network solution. </p>
  
  <p>Case in point, Adam Geitgey gives as an example usage, a house price prediction system where given a data set containing <strong>No. of bedrooms</strong>, <strong>Sq. feet</strong>, <strong>Neighborhood</strong> and <strong>Sale price</strong> you can train a neural network to be able to predict the price of a house. However he stops short of actually implementing a possible solution in code. The closest he gets, by way of an example, is basic a function demonstrating how you'd implement weights:</p>
  
  <pre><code>def estimate_house_sales_price(num_of_bedrooms, sqft, neighborhood):
    price = 0
  
    # a little pinch of this
    price += num_of_bedrooms * 1.0
  
    # and a big pinch of that
    price += sqft * 1.0
  
    # maybe a handful of this
    price += neighborhood * 1.0
  
    # and finally, just a little extra salt for good measure
    price += 1.0
  
    return price 
  </code></pre>
  
  <p>Other resources seem to focus more heavily on the maths and the only basic code example I could find that I understand (i.e. that isn't some all singing, all dancing image classification codebase) is an implementation that trains a neural network to be an XOR gate that deals only in 1's and 0's.  </p>
  
  <p>So there's a gap in my knowledge that I just can't seem to bridge. If we return to the <strong>house price prediction</strong> problem, hows does one make the data suitable for feeding into a neural network? For example:</p>
  
  <ul>
  <li>No. of bedrooms: 3</li>
  <li>Sq. feet: 2000</li>
  <li>Neighborhood: Normaltown</li>
  <li>Sale price: $250,000</li>
  </ul>
  
  <p>Can you just feed <strong>3</strong> and <strong>2000</strong> directly into the neural network because they are numbers? Or do you need to transform them into something else? Similarly what about the <strong>Normaltown</strong> value, that's a string, how do you go about translating it into a value a neural network can understand? Can you just pick a number, like an index, so long as it's consistent throughout the data?</p>
  
  <p>Most of the neural network examples I've seen the numbers passing between layers are either 0 to 1 or -1 to 1. So at the end of processing, how do you transform the output value to something usable like <strong>$185,000</strong>?</p>
  
  <p>I know the house price prediction example probably isn't a particularly useful problem given that it's been massively oversimplified to just three data points. But I just feel that if I could get over this hurdle and write an extremely basic app that trains using pseudo real-life data and spits out a pseudo real-life answer than I'll have broken the back of it and be able to kick on and delve further into machine learning.</p>
  
Tags: <neural-networks><machine-learning>
Title: How to transform inputs and extract useful outputs in a Neural Network?
UserDisplayName: David
ThreadText: # How to transform inputs and extract useful outputs in a Neural Network?
  
  (Asked by: username_0 on 2017-02-07)
  
  So I've been trying to understand neural networks ever since I came across Adam Geitgey's blog on machine learning. I've read as much as I can on the subject (that I can grasp) and believe I understand all the broad concepts and some of the workings (despite being very weak in maths), neurons, synapses, weights, cost functions, backpropagation etc. However, I've not been able to figure out how to translate real world problems into a neural network solution. 
  Case in point, Adam Geitgey gives as an example usage, a house price prediction system where given a data set containing No. of bedrooms, Sq. feet, Neighborhood and Sale price you can train a neural network to be able to predict the price of a house. However he stops short of actually implementing a possible solution in code. The closest he gets, by way of an example, is basic a function demonstrating how you'd implement weights:
  def estimate_house_sales_price(num_of_bedrooms, sqft, neighborhood):
    price = 0
  
    # a little pinch of this
    price += num_of_bedrooms * 1.0
  
    # and a big pinch of that
    price += sqft * 1.0
  
    # maybe a handful of this
    price += neighborhood * 1.0
  
    # and finally, just a little extra salt for good measure
    price += 1.0
  
    return price 
  
  Other resources seem to focus more heavily on the maths and the only basic code example I could find that I understand (i.e. that isn't some all singing, all dancing image classification codebase) is an implementation that trains a neural network to be an XOR gate that deals only in 1's and 0's.  
  So there's a gap in my knowledge that I just can't seem to bridge. If we return to the house price prediction problem, hows does one make the data suitable for feeding into a neural network? For example:
  
  No. of bedrooms: 3
  Sq. feet: 2000
  Neighborhood: Normaltown
  Sale price: $250,000
  
  Can you just feed 3 and 2000 directly into the neural network because they are numbers? Or do you need to transform them into something else? Similarly what about the Normaltown value, that's a string, how do you go about translating it into a value a neural network can understand? Can you just pick a number, like an index, so long as it's consistent throughout the data?
  Most of the neural network examples I've seen the numbers passing between layers are either 0 to 1 or -1 to 1. So at the end of processing, how do you transform the output value to something usable like $185,000?
  I know the house price prediction example probably isn't a particularly useful problem given that it's been massively oversimplified to just three data points. But I just feel that if I could get over this hurdle and write an extremely basic app that trains using pseudo real-life data and spits out a pseudo real-life answer than I'll have broken the back of it and be able to kick on and delve further into machine learning.
  
  **Tags:** neural-networks, machine-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2017-02-08. Score: 11 | Accepted answer
  
  This is a good question which I wrestled with myself when first trying to code an ANN.
  
  Below is a good general-purpose solution, and it's the one I implemented in my code for trying to predict well-behaved numerical data. If your data is not well-behaved (i.e. fraught with outliers) then you may need to do more work normalizing the inputs and outputs. Some of the more advanced methods are described [here](https://www.cs.ccu.edu.tw/~wylin/BA/Fusion_of_Biometrics_II.ppt).
  
  *Note: I will assume that you are using f(x) = tanh(x) as your activation function. If you aren't, you should still be able to reason through how to normalize your data after reading this.*
  
  **How to prepare the input data:**
  
  The basic idea is that you want a significant variation in each input parameter to be reflected by a significant variation in the activation of the neuron those inputs are being fed into. By looking at a plot of the derivative of the tanh(x) actiavtion function, you'll see that the region of significant slope is within a distance of one or two from the origin. This means that whether the input to the activation function is 2000 or 3000 (values of x for which the derivative is negligibly small), the output of the activation will be almost identical...so your neuron's state will be independent of the difference between 2000 and 3000, and your network will never produce any predictive power from values in that range.
  
  So if you want to input the square footage of the house into a neuron, you need to *normalize* the square footage so that the network can tell the difference between 2000 and 3000. One way to do this so that all of the significant variations in your data are 'noticed' by the neuron is to **z-score-normalize the inputs**.
  
  * Gather all of your square footage values (from your training set) and calculate the mean and standard deviation. *Store the mean and standard deviation*---you'll need this information to normalize new square footage values when testing.
  * **Normalize the vector of square footage values by subtracting the mean and then dividing the result by the standard deviation** (all operations element-wise of course). Subtracting the mean centers your data at the origin, and dividing by the standard deviation makes sure most of it is between -1 and 1, where the neuron's output is most sensitive to its input. This is called z-score normalization because each input value is replaced by its [z-score](https://en.wikipedia.org/wiki/Standard_score).
  * **Do the above for each input variable.**
  
  Now, when you put each input value through a neuron, the output of the neuron is an activation between -1 and 1 (look at the image of tanh(x)). Since this is already in the 'sensitive' range of the activation function, you don't need to worry about altering the output of the input-layer neurons before sending them to the first hidden layer. Just **give any hidden layer neurons the outputs of the previous layer directly**---they will be able to handle them just fine.
  
  **When you reach the last layer (the output neuron(s)), what you get is again another activation between -1 and 1. You have to convert this back into a value for the house in question**, whether that value will be used as a prediction in a test set or to calculate error during training. However you do this, you just have to be consistent and use the same de-normalization procedure in training and testing. One way to think about it is: when the output neuron(s) returns 1, that means the network is returning the *maximum possible house value* as its prediction. *What should the highest value the network can estimate be?* The right approach here simply depends on your application. This is what I did:
  
  * Calculate the mean of [the/each] output variable and store it.
  * Calculate the maximum deviation of the output variable from the mean.
    Python: `MaxDev = max([abs(DataPoint-numpy.mean(TrainingData)) for DataPoint in TrainingData])`
  * When the network returns output(s) between -1 and 1, multiply the output by `MaxDev` and add it to the mean.
  
  Two basic quick checks you can do to see if your normalization-renormalization scheme is suitable (these are necessary, but perhaps not sufficient conditions):
  
  1. If all the input values are average (e.g. average no. of bedrooms, average sq.feet, etc), is the network's output equal to the average of the output variable (e.g. house value) as well? (It should be.)
  2. If all the input values are unusually high/low, is the network's output unusually high/low as well? (This only works if all the inputs are positively related to the output...if some of them are inversely related related, you will have to think a bit more).
  
  Observe that the scheme presented here satisfies these two conditions.
  
  Notice that this scheme would allow your network to only predict house values *inside* the range of house values in your training data set. Depending on the application, this behavior can be desirable or undesirable.
  
  For example: you may want to make it impossible for your network to predict negative house values. Think about how you would do this. De-normalize the output so that -1 is mapped to 0.
  
  If you want to set no limit on the values your network can predict, then you can run the network's output through a function that maps the [-1,1] range to all real numbers...like arctanh(x)! As long as you do this during training your network will adjust its weights to accommodate this.
  
  I hope this was helpful. Let me know if you have further questions. My ANN module is in Python, by the way, so I might have language-specific advice.
  
  > **Comments:**
  > **username_0**: This was very useful! Every blog/tutorial I come across seems to avoid (almost deliberately) describing this process, but yes that all makes sense. It'll take a while for me to digest properly but I'll be back if I have any follow up questions. Much obliged!  
  > **username_0**: So couple of questions. If my Sq. Feet training data was { 2000, 800, 850, 550, 2000 } then my z-score inputs for { 1900, 1500, 600 } would be (if I've calculated correctly) { 1.0496, 0.4134, -1.0177 }. So one of those values is > 1 and one is < -1, what would I do with those? Input them into the input layer nodes regardless or round them to 1 & -1? Why does 1900 & 600 produce those values when they are within the 550 - 2000 range? Is this just a trick of the data because there is such a small data set?  
  > **username_0**: With regards to output layer renormalization, do I have it correctly that you'd plot the output to the min and max values? So if the minimum value was $0 and the maximum $100 and the output was zero (assuming -1 to 1) then that would translate as $50?  
  > **username_1**: Remember that the inputs don't need to be strictly between 1 and -1. All you need for the inputs is that most of the data is in that range. A value greater or less than one means that point is more than one standard deviation away from the mean, so that point is closer to the higher end of the data. It should be a bit rare for your data to go outside of [-1, 1], even more rare for it to go outside of [-2, 2] and extremely rare to go outside of [-3, 3]. Look at tanh(x) and you will see the sensitive range isn't just strictly between -1 and 1, but goes a bit further out than that.  
  > **username_1**: Regarding the output denormalization, that min-max denormalization is what I did in my implementation, and your interpretation is correct, but you don't necessarily have to do that. You could make it so that 1 corresponds to twice the maximum house value--that way your network would be able to predict house values above what you trained it on.  
  > **username_2**: "By looking at a plot of the derivative of the tanh(x)"  Dont understund, why should I be lookning at derivative of base function, when we input is mulitplied by by weight and then passed to base funcion eg. RELU. Could u explain?  
  > **username_1**: Because the weight only scales the derivative by a constant value, it doesn’t affect the insight that was sought here  
  
  ---
  
  