Answers.Body: <p>The <a href="https://keras.io/api/utils/python_utils/#normalize-function" rel="nofollow noreferrer">normalize()</a> function used in the tutorial is the incorrect function to use for normalization in this context.  This normalize function is for finding L1 and L2 norms.  The correct layer/function to use for scaling between 0 and 1 would be <a href="https://keras.io/api/layers/preprocessing_layers/image_preprocessing/rescaling/" rel="nofollow noreferrer">Rescaling()</a>:</p>
  <pre><code>x_train_norm = Rescaling(scale=1./255, offset=0.0)(x_train)
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-24T04:41:03.390Z
Answers.LastActivityDate: 2022-10-24T05:03:09.883Z
Answers.LastEditDate: 2022-10-24T05:03:09.883Z
Answers.UserDisplayName: Snehal Patel
Answers.Body: <p>You could rank every pixel in terms of brightness before normalization and after to verify that the ranks are preserved. Correct normalization would preserve the  ranks. From the pictures 1 and 2, it seems that the ranks were not preserved (e.g. the top right pixels went from grey to black).</p>
  <p>The tutorial's normalization is done incorrectly: <code>x_train</code> is a an array of 2D images, but the normalization was applied along the first axis. What happened is that each column of every image was normalized relatively to itself (compare image 1 and 2 column-wise – the ranks were preserved). The normalization should have been applied along the second axis – that would normalize every pixel relative to the image it's in. In seems that incorrect normalization wasn't enough to sabotage the learning though!
  Your proposed normalization should work fine.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-31T19:48:37.557Z
Answers.LastActivityDate: 2021-01-31T22:22:05.147Z
Answers.LastEditDate: 2021-01-31T22:22:05.147Z
Answers.UserDisplayName: mark mark
Body: <p>There is <a href="https://pythonprogramming.net/introduction-deep-learning-python-tensorflow-keras/" rel="nofollow noreferrer">this video on pythonprogramming.net</a> that trains a network on the MNIST handwriting dataset. At <a href="https://youtu.be/wQ8BIBpya2k?t=539" rel="nofollow noreferrer">~9:15</a>, the author explains that the data should be normalized. </p>
  
  <p>The normalization is done with</p>
  
  <pre><code>x_train = tf.keras.utils.normalize(x_train, axis=1)
  x_test = tf.keras.utils.normalize(x_test, axis=1)
  </code></pre>
  
  <p>The explanation is that values in a range of 0 ... 1 make it easier for a network to learn. That might make sense, if we consider sigmoid functions, which would otherwise map almost all values to 1.</p>
  
  <p>I could also understand that we want black to be pure black, so we want to adjust any offset in black values. Also, we want white to be pure white and potentially stretch the data to reach the upper limit.</p>
  
  <p>However, I think the kind of normalization applied in this case is incorrect. The image before was:</p>
  
  <p><a href="https://i.sstatic.net/pjAzC.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pjAzC.png" alt="Original image"></a></p>
  
  <p>After the normalization it is</p>
  
  <p><a href="https://i.sstatic.net/0Yv01.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0Yv01.png" alt="Normalized image"></a></p>
  
  <p>As we can see, some pixels which were black before have become grey now. Columns with few black pixels before result in black pixels. Columns with many black pixels before result in lighter grey pixels.</p>
  
  <p>This can be confirmed by applying the normalization on a different axis:</p>
  
  <p><a href="https://i.sstatic.net/iMtFo.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/iMtFo.png" alt="Normalization applied to a different axis"></a></p>
  
  <p>Now, rows with few black pixels before result in black pixels. Rows with many black pixels result in lighter grey pixels.</p>
  
  <p><strong>Is normalization used the right way in this tutorial? If so, why? If not, would my normalization be correct?</strong></p>
  
  <p>What I expected was a per pixel mapping from e.g. [3 ... 253] (RGB values) to [0.0 ... 1.0]. In Python code, I think this should do:</p>
  
  <pre><code>import numpy as np
  import imageio
  image = imageio.imread("sample.png")
  image = (image - np.min(image))/np.ptp(image)
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-24T06:47:18.287Z
Comments.Text: This is one of my pet-peaves... Either using an inappropriate standardization/scaling/normalization or using the wrong word to describe what's been done. :). Good catch @thomas!
Comments.UserDisplayName: David Hoelzer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-24T13:19:36.577Z
Comments.Text: Yep, definitely a good catch.  Probably worth posting a polite correction in the comments section of the video.
Comments.UserDisplayName: Snehal Patel
Tags: <python><keras><batch-normalization><handwritten-characters>
Title: Does this tutorial use normalization the right way?
UserDisplayName: Thomas Weller
ThreadText: # Does this tutorial use normalization the right way?
  
  (Asked by: username_0 on 2020-02-11)
  
  There is this video on pythonprogramming.net that trains a network on the MNIST handwriting dataset. At ~9:15, the author explains that the data should be normalized. 
  The normalization is done with
  x_train = tf.keras.utils.normalize(x_train, axis=1)
  x_test = tf.keras.utils.normalize(x_test, axis=1)
  
  The explanation is that values in a range of 0 ... 1 make it easier for a network to learn. That might make sense, if we consider sigmoid functions, which would otherwise map almost all values to 1.
  I could also understand that we want black to be pure black, so we want to adjust any offset in black values. Also, we want white to be pure white and potentially stretch the data to reach the upper limit.
  However, I think the kind of normalization applied in this case is incorrect. The image before was:
  
  After the normalization it is
  
  As we can see, some pixels which were black before have become grey now. Columns with few black pixels before result in black pixels. Columns with many black pixels before result in lighter grey pixels.
  This can be confirmed by applying the normalization on a different axis:
  
  Now, rows with few black pixels before result in black pixels. Rows with many black pixels result in lighter grey pixels.
  Is normalization used the right way in this tutorial? If so, why? If not, would my normalization be correct?
  What I expected was a per pixel mapping from e.g. [3 ... 253] (RGB values) to [0.0 ... 1.0]. In Python code, I think this should do:
  import numpy as np
  import imageio
  image = imageio.imread("sample.png")
  image = (image - np.min(image))/np.ptp(image)
  
  **Tags:** python, keras, batch-normalization, handwritten-characters
  > **Comments:**
  > **username_1**: This is one of my pet-peaves... Either using an inappropriate standardization/scaling/normalization or using the wrong word to describe what's been done. :). Good catch @thomas!  
  > **username_2**: Yep, definitely a good catch.  Probably worth posting a polite correction in the comments section of the video.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-10-24. Score: 1
  
  The normalize() function used in the tutorial is the incorrect function to use for normalization in this context.  This normalize function is for finding L1 and L2 norms.  The correct layer/function to use for scaling between 0 and 1 would be Rescaling():
  x_train_norm = Rescaling(scale=1./255, offset=0.0)(x_train)
  
  
  ---
  
  ### Answer by username_3 on 2021-01-31. Score: 0
  
  You could rank every pixel in terms of brightness before normalization and after to verify that the ranks are preserved. Correct normalization would preserve the ranks. From the pictures 1 and 2, it seems that the ranks were not preserved (e.g. the top right pixels went from grey to black).
  
  The tutorial's normalization is done incorrectly: `x_train` is a an array of 2D images, but the normalization was applied along the first axis. What happened is that each column of every image was normalized relatively to itself (compare image 1 and 2 column-wise – the ranks were preserved). The normalization should have been applied along the second axis – that would normalize every pixel relative to the image it's in. In seems that incorrect normalization wasn't enough to sabotage the learning though!
  Your proposed normalization should work fine.
  
  
  ---
  
  