Answers.Body: <p>Generally, between -1 and 1 are ideal, though you can get away with a wider range.  For example, using the z-score as the range, you will be outside of this range, sometimes by quite a bit (say, -30 for significantly distant outliers in some cases I've seen), but even then you will be fine.</p>
  <p>Anecdotally, you might try training a toy problem like MNIST with and without feature scaling.  You might be surprised that even the range of 0 to 255 will be just fine.</p>
  <p>When you start dealing with really large numbers is when you start running into problems.  The learning rate will simply not be large enough to move the weights far enough to learn in any reasonable amount of time.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-04T11:03:59.323Z
Answers.Text: So using normalized inputs is recommended but the normalization range is not strictly enforced. Right? My voncern was rooted in the fact that many tutorials and examples used normalized inputs in range [0,1], and had ReLU as activation function. I thiught there's a relation between these two (the input range, and the activation function used). Can a NN be trained efficiently using ReLU if the inputs are in range [-1,1] ? Or it's better to use another activation function than is more symmetrical around 0? Like tanh?
Answers.UserDisplayName: Bikay
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-04T11:17:32.707Z
Answers.Text: That is correct.  Yes.  Don't forget that the bias term can move the data into a range that is useful as needed before it passes through the ReLU.
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-03T21:40:02.663Z
Answers.LastActivityDate: 2024-03-03T21:40:02.663Z
Answers.UserDisplayName: David Hoelzer
Body: <p>All features of my input dataset, which is going to be used for training a simple multi-layered neural network, are in range <span class="math-container">$[-1,+1]$</span> and the output of <span class="math-container">$NN$</span> is a single number again in range <span class="math-container">$[-1,+1]$</span>.</p>
  <p>Is it a requirement/recommendation to normalize my inputs into <span class="math-container">$[0,1]$</span> range? Or can I feed the first layer (input) straight from the <span class="math-container">$[-1,+1]$</span> values?</p>
  <p>Also, should I initialize weights and biases to <span class="math-container">$0$</span>, or to random values with standard distribution?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-03T20:39:43.230Z
Comments.Text: [0,1] and [+1,-1] won't make much difference most likely
Comments.UserDisplayName: Alberto
Tags: <machine-learning><data-preprocessing><multilayer-perceptrons><weights-initialization><normalisation>
Title: Is it a requirement/recommendation to normalize my inputs into [0,1] range?
UserDisplayName: Bikay
ThreadText: # Is it a requirement/recommendation to normalize my inputs into [0,1] range?
  
  (Asked by: username_0 on 2024-03-03)
  
  All features of my input dataset, which is going to be used for training a simple multi-layered neural network, are in range $[-1,+1]$ and the output of $NN$ is a single number again in range $[-1,+1]$.
  
  Is it a requirement/recommendation to normalize my inputs into $[0,1]$ range? Or can I feed the first layer (input) straight from the $[-1,+1]$ values?
  
  Also, should I initialize weights and biases to $0$, or to random values with standard distribution?
  
  **Tags:** machine-learning, data-preprocessing, multilayer-perceptrons, weights-initialization, normalisation
  > **Comments:**
  > **username_1**: [0,1] and [+1,-1] won't make much difference most likely  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-03-03. Score: 3 | Accepted answer
  
  Generally, between -1 and 1 are ideal, though you can get away with a wider range. For example, using the z-score as the range, you will be outside of this range, sometimes by quite a bit (say, -30 for significantly distant outliers in some cases I've seen), but even then you will be fine.
  
  Anecdotally, you might try training a toy problem like MNIST with and without feature scaling. You might be surprised that even the range of 0 to 255 will be just fine.
  
  When you start dealing with really large numbers is when you start running into problems. The learning rate will simply not be large enough to move the weights far enough to learn in any reasonable amount of time.
  
  > **Comments:**
  > **username_0**: So using normalized inputs is recommended but the normalization range is not strictly enforced. Right? My voncern was rooted in the fact that many tutorials and examples used normalized inputs in range [0,1], and had ReLU as activation function. I thiught there's a relation between these two (the input range, and the activation function used). Can a NN be trained efficiently using ReLU if the inputs are in range [-1,1] ? Or it's better to use another activation function than is more symmetrical around 0? Like tanh?  
  > **username_2**: That is correct.  Yes.  Don't forget that the bias term can move the data into a range that is useful as needed before it passes through the ReLU.  
  
  ---
  
  