Answers.Body: <p>One of the factors that rescaling gives a better result is on your weight initialization of the model. Your experiments run on the same model that using the same weight initialization (<code>tf.keras.dense</code> using <a href="https://keras.io/layers/core/#dense" rel="nofollow noreferrer">Glorot Uniform</a>). If your input is too big or too small the model needs time to adjust the weight.</p>
  
  <blockquote>
    <p>Why is it necessary to rescale? I understand rescaling when different features are of different scales - that will lead to a skewed surface of the cost function in parameter space. And even then, as I understand from the linked video, the problem has to do with slow learning (convergence) and not high loss/inaccuracy. In this case, ALL the input features had the same scale. I'd assume the model would automatically adjust the scale of the weights and there would be no adverse effect on the loss. So why is the loss so high for the non-scaled case?</p>
  </blockquote>
  
  <p>Slow learning means you can't compare the loss of two models even at the same epoch. As the model using the same initial value of weights, the second model may need time (epoch) to adjust its weight to get the same result with the first model.</p>
  
  <blockquote>
    <p>If the answer has anything to do with the magnitude of the inputs, why does further scaling down of the inputs lead to worse performance?</p>
  </blockquote>
  
  <p>It has the same reason, very small values of the inputs also lead to the difficulty to adjust the weight.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-02T12:05:56.047Z
Answers.Text: Thanks so much! I need just one clarification. The reason for the slow convergence as given in the video is a skewed loss function surface in the parameter space. But in this case, since all the inputs are of the same scale, wouldn't the loss function surface be more or less symmetric? Why then is the rate of convergence slow in this case?
Answers.UserDisplayName: Shirish
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-02T06:58:47.267Z
Answers.LastActivityDate: 2019-08-02T06:58:47.267Z
Answers.UserDisplayName: malioboro
Body: <p>I've been trying out a simple neural network on the fashion_mnist dataset using keras. Regarding normalization, I've watched <a href="https://www.youtube.com/watch?reload=9&amp;v=UIp2CMI0748" rel="nofollow noreferrer">this video</a> explaining why it's necessary to normalize input features, but the explanation covers the case <strong>when input features have different scales</strong>. The logic is, say there are only two features - then if the range of one of them is much larger than that of the other, the gradient descent steps will stagger along slowly towards the minimum.</p>
  
  <p>Now I'm doing a different course on implementing neural networks and am currently studying the following example - the input features are pixel values ranging from 0 to 255, the total number of features (pixels) is 576 and we're supposed to classify images into one of ten classes. Here's the code:</p>
  
  <pre><code>import tensorflow as tf
  
  (Xtrain, ytrain) ,  (Xtest, ytest) = tf.keras.datasets.fashion_mnist.load_data()
  
  Xtrain_norm = Xtrain.copy()/255.0
  Xtest_norm = Xtest.copy()/255.0
  
  model = tf.keras.models.Sequential([tf.keras.layers.Flatten(),
                                      tf.keras.layers.Dense(128, activation="relu"),
                                      tf.keras.layers.Dense(10, activation="softmax")])
  
  model.compile(optimizer = "adam", loss = "sparse_categorical_crossentropy")
  model.fit(Xtrain_norm, ytrain, epochs=5)
  model.evaluate(Xtest_norm, ytest)
  ------------------------------------OUTPUT------------------------------------
  Epoch 1/5
  60000/60000 [==============================] - 9s 145us/sample - loss: 0.5012
  Epoch 2/5
  60000/60000 [==============================] - 7s 123us/sample - loss: 0.3798
  Epoch 3/5
  60000/60000 [==============================] - 7s 123us/sample - loss: 0.3412
  Epoch 4/5
  60000/60000 [==============================] - 7s 123us/sample - loss: 0.3182
  Epoch 5/5
  60000/60000 [==============================] - 7s 124us/sample - loss: 0.2966
  10000/10000 [==============================] - 1s 109us/sample - loss: 0.3385
  0.3384787309527397
  </code></pre>
  
  <p>So far, so good. Note that, as advised in the course, I've rescaled all inputs by dividing by 255. Next, I ran without any rescaling:</p>
  
  <pre><code>import tensorflow as tf
  
  (Xtrain, ytrain) ,  (Xtest, ytest) = tf.keras.datasets.fashion_mnist.load_data()
  
  model2 = tf.keras.models.Sequential([tf.keras.layers.Flatten(),
                                      tf.keras.layers.Dense(128, activation="relu"),
                                      tf.keras.layers.Dense(10, activation="softmax")])
  
  model2.compile(optimizer = "adam", loss = "sparse_categorical_crossentropy")
  model2.fit(Xtrain, ytrain, epochs=5)
  model2.evaluate(Xtest, ytest)
  ------------------------------------OUTPUT------------------------------------
  Epoch 1/5
  60000/60000 [==============================] - 9s 158us/sample - loss: 13.0456
  Epoch 2/5
  60000/60000 [==============================] - 8s 137us/sample - loss: 13.0127
  Epoch 3/5
  60000/60000 [==============================] - 8s 140us/sample - loss: 12.9553
  Epoch 4/5
  60000/60000 [==============================] - 9s 144us/sample - loss: 12.9172
  Epoch 5/5
  60000/60000 [==============================] - 9s 142us/sample - loss: 12.9154
  10000/10000 [==============================] - 1s 121us/sample - loss: 12.9235
  12.923488986206054
  </code></pre>
  
  <p>So somehow rescaling does make a difference? Does that mean if I further reduce the scale, the performance will improve? Worth trying out:</p>
  
  <pre><code>import tensorflow as tf
  
  (Xtrain, ytrain) ,  (Xtest, ytest) = tf.keras.datasets.fashion_mnist.load_data()
  
  Xtrain_norm = Xtrain.copy()/1000.0
  Xtest_norm = Xtest.copy()/1000.0
  
  model3 = tf.keras.models.Sequential([tf.keras.layers.Flatten(),
                                      tf.keras.layers.Dense(128, activation="relu"),
                                      tf.keras.layers.Dense(10, activation="softmax")])
  
  model3.compile(optimizer = "adam", loss = "sparse_categorical_crossentropy")
  model3.fit(Xtrain_norm, ytrain, epochs=5)
  model3.evaluate(Xtest_norm, ytest)
  ------------------------------------OUTPUT------------------------------------
  Epoch 1/5
  60000/60000 [==============================] - 9s 158us/sample - loss: 0.5428
  Epoch 2/5
  60000/60000 [==============================] - 9s 147us/sample - loss: 0.4010
  Epoch 3/5
  60000/60000 [==============================] - 8s 141us/sample - loss: 0.3587
  Epoch 4/5
  60000/60000 [==============================] - 9s 144us/sample - loss: 0.3322
  Epoch 5/5
  60000/60000 [==============================] - 8s 138us/sample - loss: 0.3120
  10000/10000 [==============================] - 1s 133us/sample - loss: 0.3718
  0.37176641924381254
  </code></pre>
  
  <p>Nope. I divided by 1000 this time and the performance seems worse than the first model. So I have a few questions:</p>
  
  <ol>
  <li><p>Why is it necessary to rescale? I understand rescaling when different features are of different scales - that will lead to a skewed surface of the cost function in parameter space. And <em>even then</em>, as I understand from the linked video, the problem has to do with <strong>slow learning (convergence)</strong> and <strong>not high loss/inaccuracy</strong>.  In this case, ALL the input features had the same scale. I'd assume the model would automatically adjust the scale of the weights and there would be no adverse effect on the loss. So why is the loss so high for the non-scaled case?</p></li>
  <li><p>If the answer has anything to do with the magnitude of the inputs, why does further scaling down of the inputs lead to worse performance?</p></li>
  </ol>
  
  <p>Does any of this have anything to do with the nature of the sparse categorical crossentropy loss, or the ReLU activation function? I'm very confused.</p>
  
Tags: <neural-networks>
Title: Effect of rescaling of inputs on loss for a simple neural network
UserDisplayName: Shirish
ThreadText: # Effect of rescaling of inputs on loss for a simple neural network
  
  (Asked by: username_0 on 2019-08-01)
  
  I've been trying out a simple neural network on the fashion_mnist dataset using keras. Regarding normalization, I've watched this video explaining why it's necessary to normalize input features, but the explanation covers the case when input features have different scales. The logic is, say there are only two features - then if the range of one of them is much larger than that of the other, the gradient descent steps will stagger along slowly towards the minimum.
  Now I'm doing a different course on implementing neural networks and am currently studying the following example - the input features are pixel values ranging from 0 to 255, the total number of features (pixels) is 576 and we're supposed to classify images into one of ten classes. Here's the code:
  import tensorflow as tf
  
  (Xtrain, ytrain) ,  (Xtest, ytest) = tf.keras.datasets.fashion_mnist.load_data()
  
  Xtrain_norm = Xtrain.copy()/255.0
  Xtest_norm = Xtest.copy()/255.0
  
  model = tf.keras.models.Sequential([tf.keras.layers.Flatten(),
                                      tf.keras.layers.Dense(128, activation="relu"),
                                      tf.keras.layers.Dense(10, activation="softmax")])
  
  model.compile(optimizer = "adam", loss = "sparse_categorical_crossentropy")
  model.fit(Xtrain_norm, ytrain, epochs=5)
  model.evaluate(Xtest_norm, ytest)
  ------------------------------------OUTPUT------------------------------------
  Epoch 1/5
  60000/60000 [==============================] - 9s 145us/sample - loss: 0.5012
  Epoch 2/5
  60000/60000 [==============================] - 7s 123us/sample - loss: 0.3798
  Epoch 3/5
  60000/60000 [==============================] - 7s 123us/sample - loss: 0.3412
  Epoch 4/5
  60000/60000 [==============================] - 7s 123us/sample - loss: 0.3182
  Epoch 5/5
  60000/60000 [==============================] - 7s 124us/sample - loss: 0.2966
  10000/10000 [==============================] - 1s 109us/sample - loss: 0.3385
  0.3384787309527397
  
  So far, so good. Note that, as advised in the course, I've rescaled all inputs by dividing by 255. Next, I ran without any rescaling:
  import tensorflow as tf
  
  (Xtrain, ytrain) ,  (Xtest, ytest) = tf.keras.datasets.fashion_mnist.load_data()
  
  model2 = tf.keras.models.Sequential([tf.keras.layers.Flatten(),
                                      tf.keras.layers.Dense(128, activation="relu"),
                                      tf.keras.layers.Dense(10, activation="softmax")])
  
  model2.compile(optimizer = "adam", loss = "sparse_categorical_crossentropy")
  model2.fit(Xtrain, ytrain, epochs=5)
  model2.evaluate(Xtest, ytest)
  ------------------------------------OUTPUT------------------------------------
  Epoch 1/5
  60000/60000 [==============================] - 9s 158us/sample - loss: 13.0456
  Epoch 2/5
  60000/60000 [==============================] - 8s 137us/sample - loss: 13.0127
  Epoch 3/5
  60000/60000 [==============================] - 8s 140us/sample - loss: 12.9553
  Epoch 4/5
  60000/60000 [==============================] - 9s 144us/sample - loss: 12.9172
  Epoch 5/5
  60000/60000 [==============================] - 9s 142us/sample - loss: 12.9154
  10000/10000 [==============================] - 1s 121us/sample - loss: 12.9235
  12.923488986206054
  
  So somehow rescaling does make a difference? Does that mean if I further reduce the scale, the performance will improve? Worth trying out:
  import tensorflow as tf
  
  (Xtrain, ytrain) ,  (Xtest, ytest) = tf.keras.datasets.fashion_mnist.load_data()
  
  Xtrain_norm = Xtrain.copy()/1000.0
  Xtest_norm = Xtest.copy()/1000.0
  
  model3 = tf.keras.models.Sequential([tf.keras.layers.Flatten(),
                                      tf.keras.layers.Dense(128, activation="relu"),
                                      tf.keras.layers.Dense(10, activation="softmax")])
  
  model3.compile(optimizer = "adam", loss = "sparse_categorical_crossentropy")
  model3.fit(Xtrain_norm, ytrain, epochs=5)
  model3.evaluate(Xtest_norm, ytest)
  ------------------------------------OUTPUT------------------------------------
  Epoch 1/5
  60000/60000 [==============================] - 9s 158us/sample - loss: 0.5428
  Epoch 2/5
  60000/60000 [==============================] - 9s 147us/sample - loss: 0.4010
  Epoch 3/5
  60000/60000 [==============================] - 8s 141us/sample - loss: 0.3587
  Epoch 4/5
  60000/60000 [==============================] - 9s 144us/sample - loss: 0.3322
  Epoch 5/5
  60000/60000 [==============================] - 8s 138us/sample - loss: 0.3120
  10000/10000 [==============================] - 1s 133us/sample - loss: 0.3718
  0.37176641924381254
  
  Nope. I divided by 1000 this time and the performance seems worse than the first model. So I have a few questions:
  
  Why is it necessary to rescale? I understand rescaling when different features are of different scales - that will lead to a skewed surface of the cost function in parameter space. And even then, as I understand from the linked video, the problem has to do with slow learning (convergence) and not high loss/inaccuracy.  In this case, ALL the input features had the same scale. I'd assume the model would automatically adjust the scale of the weights and there would be no adverse effect on the loss. So why is the loss so high for the non-scaled case?
  If the answer has anything to do with the magnitude of the inputs, why does further scaling down of the inputs lead to worse performance?
  
  Does any of this have anything to do with the nature of the sparse categorical crossentropy loss, or the ReLU activation function? I'm very confused.
  
  **Tags:** neural-networks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-08-02. Score: 1 | Accepted answer
  
  One of the factors that rescaling gives a better result is on your weight initialization of the model. Your experiments run on the same model that using the same weight initialization (`tf.keras.dense` using [Glorot Uniform](https://keras.io/layers/core/#dense)). If your input is too big or too small the model needs time to adjust the weight.
  
  > Why is it necessary to rescale? I understand rescaling when different features are of different scales - that will lead to a skewed surface of the cost function in parameter space. And even then, as I understand from the linked video, the problem has to do with slow learning (convergence) and not high loss/inaccuracy. In this case, ALL the input features had the same scale. I'd assume the model would automatically adjust the scale of the weights and there would be no adverse effect on the loss. So why is the loss so high for the non-scaled case?
  
  Slow learning means you can't compare the loss of two models even at the same epoch. As the model using the same initial value of weights, the second model may need time (epoch) to adjust its weight to get the same result with the first model.
  
  > If the answer has anything to do with the magnitude of the inputs, why does further scaling down of the inputs lead to worse performance?
  
  It has the same reason, very small values of the inputs also lead to the difficulty to adjust the weight.
  
  > **Comments:**
  > **username_0**: Thanks so much! I need just one clarification. The reason for the slow convergence as given in the video is a skewed loss function surface in the parameter space. But in this case, since all the inputs are of the same scale, wouldn't the loss function surface be more or less symmetric? Why then is the rate of convergence slow in this case?  
  
  ---
  
  