Answers.Body: <blockquote>
  <p>is it common to deal with weights and biases in everyday tasks or in most of the cases existing algorithms do it well?</p>
  </blockquote>
  <p><strong>No</strong>; and it is no coincidence that you will not be able to find any reference to such a practice in any course or tutorial about neural networks. Such a practice would require a whole additional level of (business/SME) know-how in order to meaningfully apply neural networks to real-world problems, and fortunately this is not necessary.</p>
  <p>The desired situation is for both weights &amp; biases to remain in a relatively small* range around zero, among other reasons because this avoids the exploding &amp; vanishing gradient problems, which is catastrophic for learning; trying to adjust for the scale of our inputs &amp; outputs by scaling accordingly the model weights &amp; biases is the wrong approach, and never followed.</p>
  <p>[*How small? Well, see <a href="https://stackoverflow.com/questions/48865055/multilayer-perceptron-with-sigmoid-activation-produces-straight-line-on-sin2x/48866370#48866370">here</a> and <a href="https://stackoverflow.com/questions/43636321/tensorflow-training-accuracy-not-improving-in-mnist-data/43644546#43644546">here</a> for what a change in just the <em>standard deviation</em> of a zero-mean weight initialization from <code>1.0</code> and <code>0.1</code> to <code>0.01</code> can do to the model performance]</p>
  <p>It is a well-established fact by now that neural nets work with normalized inputs, and, depending on the problem, normalized <em>outputs</em> as well; which answers to your objection:</p>
  <blockquote>
  <p>One could answer that normalizing inputs solves the problem and no need to fit weights and biases, but I guess they depend also on expected output.</p>
  </blockquote>
  <p>It depends indeed, but the solution here is to scale the <em>output(s)</em> as well, and not the weights.</p>
  <p>Now, it's true that <em>de-scaling</em> the outputs back to their original range in order to be able to meaningfully compare them with the ground truth (and possibly compute more meaningful metrics, like the error on our true output range, and not the scaled one) is AFAIK seldom mentioned in introductory expositions; but this is indeed the correct thing to do (especially for regression problems, where metrics like MSE are scale-sensitive), instead of trying to manually intervene on the weights range. For details, see own answers in <a href="https://stackoverflow.com/questions/48973140/how-to-interpret-mse-in-keras-regressor/49009442#49009442">How to interpret MSE in Keras Regressor</a> and <a href="https://datascience.stackexchange.com/questions/82285/ann-regression-accuracy-and-loss-stuck/82296#82296">ANN regression accuracy and loss stuck</a>.</p>
  <hr />
  <blockquote>
  <p>what are the rules of thumb for how to decide about range and initial values of weights and biases?</p>
  </blockquote>
  <p>Range aside, the general rule, as already implied, is to initialize the biases with zeros and the weights with small random values around zero. Nevertheless, the exact details are an area of active research. Currently used initialization schemes that are already integrated into the relevant frameworks (Tensorflow, Keras, Pytorch etc) are the <a href="http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf" rel="nofollow noreferrer">Glorot (or Xavier)</a> and <a href="https://ieeexplore.ieee.org/document/7410480" rel="nofollow noreferrer">He</a> initializations (for a nice overview, see <a href="https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-954fb9b47c79" rel="nofollow noreferrer">Weight Initialization in Neural Networks: A Journey From the Basics to Kaiming</a>).</p>
  <p>Beyond these routinely-used approaches that have already reached the practitioner's workbench, and moving closer to the front of active theoretical research, the <a href="https://openreview.net/forum?id=rJl-b3RcF7" rel="nofollow noreferrer">Lottery Ticket Hypothesis</a> (finding &quot;winning&quot; weight initializations that require minimal training) is an ultra-hot topic lately.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-14T17:31:18.040Z
Answers.Text: ok no prob. Please tell me, you can add to your answer as part 3 of same question.
  What about activation functions?  same 2 questions (is it common to deal with them as a routine, and if so what are thumb rules)
Answers.UserDisplayName: Igor
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-14T17:34:14.267Z
Answers.Text: @Igor ReLU and its variants for almost all uses in the intermediate layers; output ones depend on the problem - sigmoid or softmax for classification problems, linear for regression ones.
Answers.UserDisplayName: desertnaut
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-13T23:20:01.343Z
Answers.Text: With regards to output scaling, is it required because initially (first iterations) there might be a large difference between true outcome and prediction (due to very small weights) which can cause exploding gradients or diverging behavior?
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-14T14:04:36.780Z
Answers.LastActivityDate: 2020-10-14T17:22:19.657Z
Answers.LastEditDate: 2020-10-14T17:22:19.657Z
Answers.UserDisplayName: desertnaut
Body: <p>As a <strong>routine</strong> (in typical everyday  tasks) of a data scientist, should they usually decide about weights and biases <strong>range</strong> and <strong>initial values</strong> as a function of which data they are planning to insert as an input, and which type of data they expect to get in the output? Or we usually do not deal with such fine-tuning, and let the algorithm to do it?
  One could answer that normalizing inputs solves the problem and no need to fit weights and biases, but I guess they depend also on expected output.</p>
  <p>To summarize:</p>
  <ol>
  <li><p>is it common to deal with weights and biases in everyday tasks or in most of the cases existing algorithms do it well?</p>
  </li>
  <li><p>what are the rules of thumb for how to decide about range and initial values of weights and biases?</p>
  </li>
  </ol>
  
Tags: <neural-networks><machine-learning><deep-learning><hyper-parameters><weights>
Title: Should the range and initial values of weights and biases be adjusted to fit input and output data?
UserDisplayName: Igor
ThreadText: # Should the range and initial values of weights and biases be adjusted to fit input and output data?
  
  (Asked by: username_0 on 2020-10-14)
  
  As a **routine** (in typical everyday tasks) of a data scientist, should they usually decide about weights and biases **range** and **initial values** as a function of which data they are planning to insert as an input, and which type of data they expect to get in the output? Or we usually do not deal with such fine-tuning, and let the algorithm to do it?
  One could answer that normalizing inputs solves the problem and no need to fit weights and biases, but I guess they depend also on expected output.
  
  To summarize:
  
  1. is it common to deal with weights and biases in everyday tasks or in most of the cases existing algorithms do it well?
  2. what are the rules of thumb for how to decide about range and initial values of weights and biases?
  
  **Tags:** neural-networks, machine-learning, deep-learning, hyper-parameters, weights
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-10-14. Score: 2 | Accepted answer
  
  > is it common to deal with weights and biases in everyday tasks or in most of the cases existing algorithms do it well?
  
  **No**; and it is no coincidence that you will not be able to find any reference to such a practice in any course or tutorial about neural networks. Such a practice would require a whole additional level of (business/SME) know-how in order to meaningfully apply neural networks to real-world problems, and fortunately this is not necessary.
  
  The desired situation is for both weights & biases to remain in a relatively small\* range around zero, among other reasons because this avoids the exploding & vanishing gradient problems, which is catastrophic for learning; trying to adjust for the scale of our inputs & outputs by scaling accordingly the model weights & biases is the wrong approach, and never followed.
  
  [\*How small? Well, see [here](https://stackoverflow.com/questions/48865055/multilayer-perceptron-with-sigmoid-activation-produces-straight-line-on-sin2x/48866370#48866370) and [here](https://stackoverflow.com/questions/43636321/tensorflow-training-accuracy-not-improving-in-mnist-data/43644546#43644546) for what a change in just the *standard deviation* of a zero-mean weight initialization from `1.0` and `0.1` to `0.01` can do to the model performance]
  
  It is a well-established fact by now that neural nets work with normalized inputs, and, depending on the problem, normalized *outputs* as well; which answers to your objection:
  
  > One could answer that normalizing inputs solves the problem and no need to fit weights and biases, but I guess they depend also on expected output.
  
  It depends indeed, but the solution here is to scale the *output(s)* as well, and not the weights.
  
  Now, it's true that *de-scaling* the outputs back to their original range in order to be able to meaningfully compare them with the ground truth (and possibly compute more meaningful metrics, like the error on our true output range, and not the scaled one) is AFAIK seldom mentioned in introductory expositions; but this is indeed the correct thing to do (especially for regression problems, where metrics like MSE are scale-sensitive), instead of trying to manually intervene on the weights range. For details, see own answers in [How to interpret MSE in Keras Regressor](https://stackoverflow.com/questions/48973140/how-to-interpret-mse-in-keras-regressor/49009442#49009442) and [ANN regression accuracy and loss stuck](https://datascience.stackexchange.com/questions/82285/ann-regression-accuracy-and-loss-stuck/82296#82296).
  
  ---
  
  > what are the rules of thumb for how to decide about range and initial values of weights and biases?
  
  Range aside, the general rule, as already implied, is to initialize the biases with zeros and the weights with small random values around zero. Nevertheless, the exact details are an area of active research. Currently used initialization schemes that are already integrated into the relevant frameworks (Tensorflow, Keras, Pytorch etc) are the [Glorot (or Xavier)](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf) and [He](https://ieeexplore.ieee.org/document/7410480) initializations (for a nice overview, see [Weight Initialization in Neural Networks: A Journey From the Basics to Kaiming](https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-954fb9b47c79)).
  
  Beyond these routinely-used approaches that have already reached the practitioner's workbench, and moving closer to the front of active theoretical research, the [Lottery Ticket Hypothesis](https://openreview.net/forum?id=rJl-b3RcF7) (finding "winning" weight initializations that require minimal training) is an ultra-hot topic lately.
  
  > **Comments:**
  > **username_0**: ok no prob. Please tell me, you can add to your answer as part 3 of same question. What about activation functions?  same 2 questions (is it common to deal with them as a routine, and if so what are thumb rules)  
  > **username_1**: @Igor ReLU and its variants for almost all uses in the intermediate layers; output ones depend on the problem - sigmoid or softmax for classification problems, linear for regression ones.  
  > **username_2**: With regards to output scaling, is it required because initially (first iterations) there might be a large difference between true outcome and prediction (due to very small weights) which can cause exploding gradients or diverging behavior?  
  
  ---
  
  