Answers.Body: <p>There seems to be a mismatch between the weights you provide and your network diagram. Since <code>w[0]</code> (the yellow connections) is meant to transform <span class="math-container">$ x \in \mathbb{R}^2 $</span> into the layer 0 activations which are <span class="math-container">$ \mathbb{R}^2 $</span>, <code>w[0]</code> should be a matrix <span class="math-container">$ \in \mathbb{R}^{2 \times 2} $</span>, not a vector in <span class="math-container">$\mathbb{R}^2 $</span> as you have. Likewise, your <code>w[1]</code> (the red connections) should be a vector <span class="math-container">$ \in \mathbb{R^2} $</span> and not a scalar. Finally, if you are indeed scaling the output of layer 1 (the blue connection), then you'll need an additional scalar value. However, the blue connection confuses me a bit as usually the activated output is used directly in the loss function, not a scaled version of it. Unless the blue connection stands for the loss function.</p>
  <p>In short, I believe if you change the shapes of your weight matrices to actually represent your network diagram, your update equations will work. I'll go through the network below to make sure I illustrate my point.</p>
  <p><span class="math-container">$ x \in \mathbb{R}^{2} $</span>, an input example</p>
  <p><span class="math-container">$ W^0 \in \mathbb{R}^{2 \times 2} $</span>, the yellow connections</p>
  <p><span class="math-container">$ W^1 \in \mathbb{R}^2 $</span>, the red connections</p>
  <p><span class="math-container">$ z^0 = xW^0 \in \mathbb{R}^{2} $</span>, the weighted inputs to the layer 0 nodes. The dimensions of this should match the number of nodes at layer 0.</p>
  <p><span class="math-container">$ a^0 = \sigma(z^0) \in \mathbb{R}^{2} $</span>, the output of the layer 0 nodes. The dimensions of this should match the number of nodes at layer 0.</p>
  <p><span class="math-container">$ z^1 = a^0 W^1 \in \mathbb{R} $</span>, the weighted inputs to the layer 1 nodes. The dimensions of this should match the number of nodes at layer 1.</p>
  <p><span class="math-container">$ a^1 = \sigma(z^1) \in \mathbb{R} $</span>, the output of the layer 1 nodes and thus the output of the network. The dimensions of this should match the number of nodes at layer 1.</p>
  <p><strong>Weight Updates</strong></p>
  <p>As you say before your edit, <span class="math-container">$\delta^1$</span>, as the product of two scalars <span class="math-container">$\nabla_a C$</span> and <span class="math-container">$\sigma'(z^1)$</span>, is also a scalar. Since <span class="math-container">$a^0$</span> is a vector in <span class="math-container">$\mathbb{R}^2$</span>, then
  <span class="math-container">$\delta^1(a^0)^T$</span> is also a vector in <span class="math-container">$\mathbb{R}^2$</span>. This matches what we expect, as it should match the dimensions of <span class="math-container">$W^1$</span> to allow the element-wise subtraction in the weight update equation.</p>
  <p><strong>NB.</strong> It is not the case, as you say in your edit, that the shape of <span class="math-container">$\delta^l$</span> should match the shape of <span class="math-container">$W^l$</span>. It should instead match the number of nodes, and it is the shape of <span class="math-container">$\delta^l(a^{l-1})^T$</span> that should match the shape of <span class="math-container">$W^l$</span>. You had this right in your original post.</p>
  <p><strong>Bias Updates</strong></p>
  <p>This brings us to the bias updates.
  There should be one bias term per node in a given layer, so the shapes of your biases are correct (i.e. <span class="math-container">$\mathbb{R}^2$</span> for layer 0 and <span class="math-container">$\mathbb{R}$</span> for layer 1). Now, we saw above that the shape of <span class="math-container">$\delta^l$</span> also matches the number of nodes in layer <span class="math-container">$l$</span>, so again the element-wise subtraction in your original bias update equation works.</p>
  <p>I also tried using this book to learn backprop, but I had a hard time connecting the variables with the different parts of the network and the corresponding code. I finally understood the algorithm in depth only after deriving all the update equations by hand for a very small network (2 inputs, one output, no hidden layers) and working my way up to larger networks, making sure to keep track of the shapes of the inputs and outputs along the way. If you're having trouble with the update equations I highly recommend this.</p>
  <p>A final piece of advice that helped me: drop the <span class="math-container">$x$</span> and the summations over input examples from your formulations and just treat everything as matrices (e.g. a scalar becomes a matrix in <span class="math-container">$\mathbb{R}^{1 \times 1}$</span>, <span class="math-container">$X$</span> is a matrix in <span class="math-container">$\mathbb{R}^{N \times D}$</span>). First, this allows you to better interpret matrix orientations and debug issues such as a missing transpose operation. Second, this is (in my limited understanding) how backprop should actually be implemented in order to take advantage of optimized linalg libraries and GPUs, so it's perhaps a bit more relevant.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-12T11:32:46.917Z
Answers.Text: Thank you for the comment. I actually realised this mistake too and updated the weight variables to as you said. So for layer0, a 2x2 matrix is used to represent the weights of that layer and for layer1, a 2x1 matrix is used for the weights. This has solved the issue with updating the weights as far as I can tell (delta is a vector with the same shape as the weight vector now). The issue I have is with updating the biases now. If I understand correctly, each neuron should have only 1 bias term (layer0 has a 2x1 vector of biases, layer1 has a 1x1 vector of biases).
Answers.UserDisplayName: user1147964
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-12T11:33:13.170Z
Answers.Text: But if delta has the shape of the weights, then how are the bias terms updated? I'll update my question to ask this in more detail. Thank you for the response.
Answers.UserDisplayName: user1147964
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-12T15:03:20.073Z
Answers.Text: delta does not have the shape of the weights. I've edited my answer to clarify this and to include an explanation of bias updates.
Answers.UserDisplayName: primussucks
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-12T17:48:03.300Z
Answers.Text: THank you for the reply again. Sorry, so for the NN I've specified, $W = [\R^{2x2}, \R^2]$ and $B = [\R^2, \R^1]$. Then $\sum_x \delta^{x, \mathbf{L}} (a^{x, \mathbf{L-1}})^T = [\R^{2x2}, \R^2]$ and $\sum_x \delta^{x, \mathbf{L}} = [\R^2, \R^1]$
  
  Have I understood that correctly? Because you need as many update terms as you have weights and biases?
Answers.UserDisplayName: user1147964
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-12T18:03:31.840Z
Answers.Text: Yes, exactly. The weight and bias updates are element-wise subtractions, so the updates need to have the same shape as the thing they're updating. Again, I highly recommend deriving the update equations for your network by applying the chain rule so you can check the shapes at each step. You're very welcome and best of luck with your implementation. Also, please accept my answer if you're satisfied.
Answers.UserDisplayName: primussucks
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-12T00:04:10.137Z
Answers.LastActivityDate: 2020-08-12T15:00:40.680Z
Answers.LastEditDate: 2020-08-12T15:00:40.680Z
Answers.UserDisplayName: primussucks
Body: <p>I'm following the guide as outlined at this link: <a href="http://neuralnetworksanddeeplearning.com/chap2.html" rel="nofollow noreferrer">http://neuralnetworksanddeeplearning.com/chap2.html</a></p>
  <p>For the purposes of this question, I've written a basic network 2 hidden layers, one with 2 neurons and one with one neuron. For a very basic task, the network will learn how to compute an OR logic gate so the training data will be:</p>
  <pre><code>X = [[0, 0], [0, 1], [1, 0], [1, 1]]
  Y = [0, 1, 1, 1]
  </code></pre>
  <p>And the diagram:
  <a href="https://i.sstatic.net/E4aCO.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/E4aCO.png" alt="enter image description here" /></a></p>
  <p>For this example, the weights and biases are:</p>
  <pre><code>w = [[0.3, 0.4], [0.1]]
  b = [[1, 1], [1]]
  </code></pre>
  <p>The feedforward part was pretty easy to implement so I don't think I need to post that here. The tutorial I've been following summarises calculating the errors and the gradient descent algorithm with the following equations:</p>
  <p>For each training example <span class="math-container">$x$</span>, compute the output error <span class="math-container">$\delta^{x, L}$</span> where <span class="math-container">$L =$</span> Final layer (Layer 1 in this case). <span class="math-container">$\delta^{x, L} = \nabla_aC_x \circ \sigma'(z^{x, L})$</span> where <span class="math-container">$\nabla_aC_x$</span> is the differential of the cost function (basic MSE) with respect to the Layer 1 activation output, and <span class="math-container">$\sigma'(z^{x, L})$</span> is the derivative of the sigmoid function of the Layer 1 output i.e. <span class="math-container">$\sigma(z^{x, L})(1-\sigma(z^{x, L}))$</span>.</p>
  <p>That's all good so far and I can calculate that quite straightforwardly. Now for <span class="math-container">$l = L-1, L-2, ...$</span>, the error for each previous layer can be calculated as</p>
  <p><span class="math-container">$\delta^{x, l} = ((w^{l+1})^T \delta^{x, l+1}) \circ \sigma(z^{x, l})$</span></p>
  <p>Which again, is pretty straight forward to implement.</p>
  <p>Finally, to update the weights (and bias), the equations are for <span class="math-container">$l = L, L-1, ...$</span>:</p>
  <p><span class="math-container">$w^l \rightarrow w^l - \frac{\eta}{m}\sum_x\delta^{x,l}(a^{x, l-1})^T$</span></p>
  <p><span class="math-container">$b^l \rightarrow b^l - \frac{\eta}{m}\sum_x\delta^{x,l}$</span></p>
  <p>What I don't understand is how this works with vectors of different numbers of elements (I think the lack of vector notation here confuses me).</p>
  <p>For example, Layer 1 has one neuron, so <span class="math-container">$\delta^{x, 1}$</span> will be a scalar value since it only outputs one value. However, <span class="math-container">$a^{x, 0}$</span> is a vector with two elements since layer 0 has two neurons. Which means that <span class="math-container">$\delta^{x, l}(a^{x, l-1})^T$</span> will be a vector even if I sum over all training samples <span class="math-container">$x$</span>. What am I supposed to do here? Am I just supposed to sum the components of the vector as well?</p>
  <p>Hopefully my question makes sense; I feel I'm very close to implementing this entirely and I'm just stuck here.</p>
  <p>Thank you</p>
  <p>[edit] Okay, so I realised that I've been misrepresenting the weights of the neurons and have corrected for that.</p>
  <pre><code>weights = [np.random.randn(y, x) for x, y in zip(sizes[:-1], sizes[1:])]
  </code></pre>
  <p>Which has the output</p>
  <pre><code>[array([[0.27660583, 1.00106314],
     [0.34017727, 0.74990392]])
  array([[ 1.095244  , -0.22719165]])
  </code></pre>
  <p>Which means that layer0 has a weight matrix with shape 2x2 representing the 2 weights on neuron01 and the 2 weights on neuron02.</p>
  <p>My understanding then is that <span class="math-container">$\delta^{x,l}$</span> has the same shape as the weights array because each weight gets updated indepedently. That's also fine.</p>
  <p>But the bias term (according to the link I sourced) has 1 term for each neuron, which means layer 0 will has two bias terms (b00 and b01) and layer 1 has one bias term (b10).</p>
  <p>However, to calculate the update for the bias terms, you sum the deltas over x i.e <span class="math-container">$\sum_x \delta^{x, l}$</span>; if delta has the size of the weight matrix, then there are too many terms to update the bias terms. What have I missed here?</p>
  <p>Many thanks</p>
  
Tags: <neural-networks><python><gradient-descent>
Title: Implementing Gradient Descent Algorithm in Python, bit confused regarding equations
UserDisplayName: user1147964
ThreadText: # Implementing Gradient Descent Algorithm in Python, bit confused regarding equations
  
  (Asked by: username_0 on 2020-08-11)
  
  I'm following the guide as outlined at this link: http://neuralnetworksanddeeplearning.com/chap2.html
  For the purposes of this question, I've written a basic network 2 hidden layers, one with 2 neurons and one with one neuron. For a very basic task, the network will learn how to compute an OR logic gate so the training data will be:
  X = [[0, 0], [0, 1], [1, 0], [1, 1]]
  Y = [0, 1, 1, 1]
  
  And the diagram:
  
  For this example, the weights and biases are:
  w = [[0.3, 0.4], [0.1]]
  b = [[1, 1], [1]]
  
  The feedforward part was pretty easy to implement so I don't think I need to post that here. The tutorial I've been following summarises calculating the errors and the gradient descent algorithm with the following equations:
  For each training example $x$, compute the output error $\delta^{x, L}$ where $L =$ Final layer (Layer 1 in this case). $\delta^{x, L} = \nabla_aC_x \circ \sigma'(z^{x, L})$ where $\nabla_aC_x$ is the differential of the cost function (basic MSE) with respect to the Layer 1 activation output, and $\sigma'(z^{x, L})$ is the derivative of the sigmoid function of the Layer 1 output i.e. $\sigma(z^{x, L})(1-\sigma(z^{x, L}))$.
  That's all good so far and I can calculate that quite straightforwardly. Now for $l = L-1, L-2, ...$, the error for each previous layer can be calculated as
  $\delta^{x, l} = ((w^{l+1})^T \delta^{x, l+1}) \circ \sigma(z^{x, l})$
  Which again, is pretty straight forward to implement.
  Finally, to update the weights (and bias), the equations are for $l = L, L-1, ...$:
  $w^l \rightarrow w^l - \frac{\eta}{m}\sum_x\delta^{x,l}(a^{x, l-1})^T$
  $b^l \rightarrow b^l - \frac{\eta}{m}\sum_x\delta^{x,l}$
  What I don't understand is how this works with vectors of different numbers of elements (I think the lack of vector notation here confuses me).
  For example, Layer 1 has one neuron, so $\delta^{x, 1}$ will be a scalar value since it only outputs one value. However, $a^{x, 0}$ is a vector with two elements since layer 0 has two neurons. Which means that $\delta^{x, l}(a^{x, l-1})^T$ will be a vector even if I sum over all training samples $x$. What am I supposed to do here? Am I just supposed to sum the components of the vector as well?
  Hopefully my question makes sense; I feel I'm very close to implementing this entirely and I'm just stuck here.
  Thank you
  [edit] Okay, so I realised that I've been misrepresenting the weights of the neurons and have corrected for that.
  weights = [np.random.randn(y, x) for x, y in zip(sizes[:-1], sizes[1:])]
  
  Which has the output
  [array([[0.27660583, 1.00106314],
     [0.34017727, 0.74990392]])
  array([[ 1.095244  , -0.22719165]])
  
  Which means that layer0 has a weight matrix with shape 2x2 representing the 2 weights on neuron01 and the 2 weights on neuron02.
  My understanding then is that $\delta^{x,l}$ has the same shape as the weights array because each weight gets updated indepedently. That's also fine.
  But the bias term (according to the link I sourced) has 1 term for each neuron, which means layer 0 will has two bias terms (b00 and b01) and layer 1 has one bias term (b10).
  However, to calculate the update for the bias terms, you sum the deltas over x i.e $\sum_x \delta^{x, l}$; if delta has the size of the weight matrix, then there are too many terms to update the bias terms. What have I missed here?
  Many thanks
  
  **Tags:** neural-networks, python, gradient-descent
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-08-12. Score: 0 | Accepted answer
  
  There seems to be a mismatch between the weights you provide and your network diagram. Since `w[0]` (the yellow connections) is meant to transform $ x \in \mathbb{R}^2 $ into the layer 0 activations which are $ \mathbb{R}^2 $, `w[0]` should be a matrix $ \in \mathbb{R}^{2 \times 2} $, not a vector in $\mathbb{R}^2 $ as you have. Likewise, your `w[1]` (the red connections) should be a vector $ \in \mathbb{R^2} $ and not a scalar. Finally, if you are indeed scaling the output of layer 1 (the blue connection), then you'll need an additional scalar value. However, the blue connection confuses me a bit as usually the activated output is used directly in the loss function, not a scaled version of it. Unless the blue connection stands for the loss function.
  
  In short, I believe if you change the shapes of your weight matrices to actually represent your network diagram, your update equations will work. I'll go through the network below to make sure I illustrate my point.
  
  $ x \in \mathbb{R}^{2} $, an input example
  
  $ W^0 \in \mathbb{R}^{2 \times 2} $, the yellow connections
  
  $ W^1 \in \mathbb{R}^2 $, the red connections
  
  $ z^0 = xW^0 \in \mathbb{R}^{2} $, the weighted inputs to the layer 0 nodes. The dimensions of this should match the number of nodes at layer 0.
  
  $ a^0 = \sigma(z^0) \in \mathbb{R}^{2} $, the output of the layer 0 nodes. The dimensions of this should match the number of nodes at layer 0.
  
  $ z^1 = a^0 W^1 \in \mathbb{R} $, the weighted inputs to the layer 1 nodes. The dimensions of this should match the number of nodes at layer 1.
  
  $ a^1 = \sigma(z^1) \in \mathbb{R} $, the output of the layer 1 nodes and thus the output of the network. The dimensions of this should match the number of nodes at layer 1.
  
  **Weight Updates**
  
  As you say before your edit, $\delta^1$, as the product of two scalars $\nabla\_a C$ and $\sigma'(z^1)$, is also a scalar. Since $a^0$ is a vector in $\mathbb{R}^2$, then
  $\delta^1(a^0)^T$ is also a vector in $\mathbb{R}^2$. This matches what we expect, as it should match the dimensions of $W^1$ to allow the element-wise subtraction in the weight update equation.
  
  **NB.** It is not the case, as you say in your edit, that the shape of $\delta^l$ should match the shape of $W^l$. It should instead match the number of nodes, and it is the shape of $\delta^l(a^{l-1})^T$ that should match the shape of $W^l$. You had this right in your original post.
  
  **Bias Updates**
  
  This brings us to the bias updates.
  There should be one bias term per node in a given layer, so the shapes of your biases are correct (i.e. $\mathbb{R}^2$ for layer 0 and $\mathbb{R}$ for layer 1). Now, we saw above that the shape of $\delta^l$ also matches the number of nodes in layer $l$, so again the element-wise subtraction in your original bias update equation works.
  
  I also tried using this book to learn backprop, but I had a hard time connecting the variables with the different parts of the network and the corresponding code. I finally understood the algorithm in depth only after deriving all the update equations by hand for a very small network (2 inputs, one output, no hidden layers) and working my way up to larger networks, making sure to keep track of the shapes of the inputs and outputs along the way. If you're having trouble with the update equations I highly recommend this.
  
  A final piece of advice that helped me: drop the $x$ and the summations over input examples from your formulations and just treat everything as matrices (e.g. a scalar becomes a matrix in $\mathbb{R}^{1 \times 1}$, $X$ is a matrix in $\mathbb{R}^{N \times D}$). First, this allows you to better interpret matrix orientations and debug issues such as a missing transpose operation. Second, this is (in my limited understanding) how backprop should actually be implemented in order to take advantage of optimized linalg libraries and GPUs, so it's perhaps a bit more relevant.
  
  > **Comments:**
  > **username_0**: Thank you for the comment. I actually realised this mistake too and updated the weight variables to as you said. So for layer0, a 2x2 matrix is used to represent the weights of that layer and for layer1, a 2x1 matrix is used for the weights. This has solved the issue with updating the weights as far as I can tell (delta is a vector with the same shape as the weight vector now). The issue I have is with updating the biases now. If I understand correctly, each neuron should have only 1 bias term (layer0 has a 2x1 vector of biases, layer1 has a 1x1 vector of biases).  
  > **username_0**: But if delta has the shape of the weights, then how are the bias terms updated? I'll update my question to ask this in more detail. Thank you for the response.  
  > **username_1**: delta does not have the shape of the weights. I've edited my answer to clarify this and to include an explanation of bias updates.  
  > **username_0**: THank you for the reply again. Sorry, so for the NN I've specified, $W = [\R^{2x2}, \R^2]$ and $B = [\R^2, \R^1]$. Then $\sum_x \delta^{x, \mathbf{L}} (a^{x, \mathbf{L-1}})^T = [\R^{2x2}, \R^2]$ and $\sum_x \delta^{x, \mathbf{L}} = [\R^2, \R^1]$  Have I understood that correctly? Because you need as many update terms as you have weights and biases?  
  > **username_1**: Yes, exactly. The weight and bias updates are element-wise subtractions, so the updates need to have the same shape as the thing they're updating. Again, I highly recommend deriving the update equations for your network by applying the chain rule so you can check the shapes at each step. You're very welcome and best of luck with your implementation. Also, please accept my answer if you're satisfied.  
  
  ---
  
  