Answers.Body: <p>In a simple feed-forward network, each artificial neuron has a separate bias value. This allows for greater flexibility for the output layer function than if each neuron had to use a single whole-layer bias. Although not an absolute requirement, without this arrangement it may become very hard to approximate some functions. Moving from a bias vector to a single scalar bias value per layer will most of the time reduce the effectiveness of a neural network through lost flexibility in how it fits to the target function.</p>
  
  <p>Once you have <span class="math-container">$N$</span> output neurons in a layer leading to needing <span class="math-container">$N$</span> values for bias, then it is fairly straightforward to model this collection of bias values as a vector.</p>
  
  <p>Often you will see a neural network layer function written in this form or similar:</p>
  
  <p><span class="math-container">$$\mathbf{y} = f(\mathbf{W}\mathbf{x} + \mathbf{b})$$</span></p>
  
  <p>Where <span class="math-container">$f()$</span> is the activation function (applied element-wise), <span class="math-container">$\mathbf{W}$</span> the weights matrix for the layer and <span class="math-container">$\mathbf{b}$</span> is the bias. When written in this form, it is easy to see that <span class="math-container">$\mathbf{y}$</span>, <span class="math-container">$\mathbf{W}\mathbf{x}$</span> and <span class="math-container">$\mathbf{b}$</span> must all be vectors of the same size.</p>
  
  <p>This layer design has become so standard that it is possible to forget that other designs and implementations are possible for neural network parameters, and can sometimes be useful. Frameworks like TensorFlow also make it easier to take the standard approach, which is why you need a vector for bias on the example you are using. Whilst you are learning, and probably 99% of the time after that, it will be best to go with what the framework is doing here. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-24T02:34:23.523Z
Answers.Text: Makes sense, thanks! Not sure why I thought otherwise, in hindsight. Maybe one of the original tutorials I watched had it the more inflexible way.
Answers.UserDisplayName: MrMultiMediator
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-20T15:27:32.770Z
Answers.LastActivityDate: 2020-01-20T15:27:32.770Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>To emphasize (and this is not emphasized in <a href="https://ai.stackexchange.com/a/17590/2444">this answer</a>), in the case of neural networks, the biases or, more precisely, the <a href="https://stackoverflow.com/a/48243204/3924118">connections (or weights) between biases and other neurons are also <em>learnable</em> parameters</a>, so the back-propagation algorithm calculates a gradient of the loss function that contains the partial derivatives with respect to the connections between the biases and other neurons too and, in the gradient descent step, these connections can also be updated. </p>
  
  <p>Each neuron usually has its own bias. For example, in Keras, this is the case, as <a href="https://stackoverflow.com/a/59828935/3924118">you can easily verify</a>. However, in principle, you could also have a layer with a single scalar bias that is shared across all neurons of that layer, but this would probably have a different effect. The role of the bias is discussed in several places on the web. For example, in <a href="https://stackoverflow.com/q/2480650/3924118">this Stack Overflow post</a> or in <a href="https://stats.stackexchange.com/q/185911/82135">this Stats SE post</a>. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-20T18:11:15.633Z
Answers.LastActivityDate: 2020-01-20T18:17:53.593Z
Answers.LastEditDate: 2020-01-20T18:17:53.593Z
Answers.UserDisplayName: nbro
Body: <p>I am learning to use tensorflow.js. I am also using the tfvis library to print information about the neural net to the web browser. When I create a create a dense neural net with a layer with 5 neurons and another layer with 2 neurons, each layer has a bias vector of length 5 and 2 respectively. I checked the docs (<a href="https://js.tensorflow.org/api/0.6.1/#layers.dense" rel="nofollow noreferrer">https://js.tensorflow.org/api/0.6.1/#layers.dense</a>), and it says that there is indeed a bias vector for each dense layer. Isn't a vector redundant? Doesn't each layer only need a single number for the bias? See the code below:</p>
  
  
  
  <pre><code>//Create tensorflow neural net
  this.model = tf.sequential();
  
  this.model.add(tf.layers.dense({units: 5, inputShape: [1]}))
  this.model.add(tf.layers.dense({units: 2}))
  
  const surface = { name: 'Layer Summary', tab: 'Model Inspection'};
  tfvis.show.layer(surface, this.model.getLayer(undefined, 0))
  </code></pre>
  
Tags: <neural-networks><deep-learning><tensorflow><javascript>
Title: Why does the bias need to be a vector in a neural network?
UserDisplayName: MrMultiMediator
ThreadText: # Why does the bias need to be a vector in a neural network?
  
  (Asked by: username_0 on 2020-01-20)
  
  I am learning to use tensorflow.js. I am also using the tfvis library to print information about the neural net to the web browser. When I create a create a dense neural net with a layer with 5 neurons and another layer with 2 neurons, each layer has a bias vector of length 5 and 2 respectively. I checked the docs (https://js.tensorflow.org/api/0.6.1/#layers.dense), and it says that there is indeed a bias vector for each dense layer. Isn't a vector redundant? Doesn't each layer only need a single number for the bias? See the code below:
  //Create tensorflow neural net
  this.model = tf.sequential();
  
  this.model.add(tf.layers.dense({units: 5, inputShape: [1]}))
  this.model.add(tf.layers.dense({units: 2}))
  
  const surface = { name: 'Layer Summary', tab: 'Model Inspection'};
  tfvis.show.layer(surface, this.model.getLayer(undefined, 0))
  
  **Tags:** neural-networks, deep-learning, tensorflow, javascript
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-01-20. Score: 4 | Accepted answer
  
  In a simple feed-forward network, each artificial neuron has a separate bias value. This allows for greater flexibility for the output layer function than if each neuron had to use a single whole-layer bias. Although not an absolute requirement, without this arrangement it may become very hard to approximate some functions. Moving from a bias vector to a single scalar bias value per layer will most of the time reduce the effectiveness of a neural network through lost flexibility in how it fits to the target function.
  
  Once you have $N$ output neurons in a layer leading to needing $N$ values for bias, then it is fairly straightforward to model this collection of bias values as a vector.
  
  Often you will see a neural network layer function written in this form or similar:
  
  $$\mathbf{y} = f(\mathbf{W}\mathbf{x} + \mathbf{b})$$
  
  Where $f()$ is the activation function (applied element-wise), $\mathbf{W}$ the weights matrix for the layer and $\mathbf{b}$ is the bias. When written in this form, it is easy to see that $\mathbf{y}$, $\mathbf{W}\mathbf{x}$ and $\mathbf{b}$ must all be vectors of the same size.
  
  This layer design has become so standard that it is possible to forget that other designs and implementations are possible for neural network parameters, and can sometimes be useful. Frameworks like TensorFlow also make it easier to take the standard approach, which is why you need a vector for bias on the example you are using. Whilst you are learning, and probably 99% of the time after that, it will be best to go with what the framework is doing here.
  
  > **Comments:**
  > **username_0**: Makes sense, thanks! Not sure why I thought otherwise, in hindsight. Maybe one of the original tutorials I watched had it the more inflexible way.  
  
  ---
  
  ### Answer by username_2 on 2020-01-20. Score: 2
  
  To emphasize (and this is not emphasized in [this answer](https://ai.stackexchange.com/a/17590/2444)), in the case of neural networks, the biases or, more precisely, the [connections (or weights) between biases and other neurons are also *learnable* parameters](https://stackoverflow.com/a/48243204/3924118), so the back-propagation algorithm calculates a gradient of the loss function that contains the partial derivatives with respect to the connections between the biases and other neurons too and, in the gradient descent step, these connections can also be updated.
  
  Each neuron usually has its own bias. For example, in Keras, this is the case, as [you can easily verify](https://stackoverflow.com/a/59828935/3924118). However, in principle, you could also have a layer with a single scalar bias that is shared across all neurons of that layer, but this would probably have a different effect. The role of the bias is discussed in several places on the web. For example, in [this Stack Overflow post](https://stackoverflow.com/q/2480650/3924118) or in [this Stats SE post](https://stats.stackexchange.com/q/185911/82135).
  
  
  ---
  
  