title: Why We do NOT backpropagate deltas from bias units in Neural Network
body: <p>I'm trying to implement a simple neutral network in Python. My question is when we do backpropagation, why we don't backpropagate deltas from bias units? In the tutorial below, it says so, but I don't get it. </p>
  
  <p>i also pasted my code below. my question is 1) is there any mistake in the code? 2) if I change "W0 = W0 - alpha<em>np.dot(X_bias.T,L1_delta[:,:-1])" to "W0 = W0 - alpha</em>np.dot(X_bias.T,L1_delta")", it will throw an error. "operands could not be broadcast together with shapes (3,4) (3,5)" So my question is why do i have to omit the last column in this matrix?</p>
  
  <p><a href="http://outlace.com/Beginner-Tutorial-Backpropagation/" rel="nofollow noreferrer">http://outlace.com/Beginner-Tutorial-Backpropagation/</a></p>
  
  <pre><code>import numpy as np
  
  def sigmoid(num):
      return 1/(1+np.exp(-num))
  
  def sigmoid_derivative(num):
      return num*(1-num)
  
  def add_bias(matrix):
      newcol=np.ones((matrix.shape[0], 1))
      return np.hstack([matrix, newcol])
  
  
  print("Initializing parameters")
  alpha = 0.09
  
  X = np.array([[1,1],
                [1,0],
                [1,1],
                [0,1]])
  
  y=np.array([[1],
              [0],
              [0],
              [1]])
  
  W0 = np.random.rand(3,4)
  W1 = np.random.rand(5,1)
  
  
  print("Start iterating")
  for i in range(60000):
      X_bias = add_bias(X)
      L1 = sigmoid(np.dot(X_bias,W0))
  
      L1_bias = add_bias(L1)
      L2 = sigmoid(np.dot(L1_bias,W1))
  
      L2_error = L2 - y
      L2_delta = L2_error*sigmoid_derivative(L2)
      L1_error = np.dot(L2_delta,W1.T)
      L1_delta = L1_error*sigmoid_derivative(L1_bias)
  
      W1 = W1 - alpha*np.dot(L1_bias.T,L2_delta)
      W0 = W0 - alpha*np.dot(X_bias.T,L1_delta[:,:-1)
  
  
  if i%1000==0:
      print("Error after {0} iterations is: {1}".format(str(i),str(np.mean(np.abs(L2_error)))))
  </code></pre>