Answers.Body: <h1>Short answers</h1>
  <blockquote>
  <p>Is back-propagation applied immediately after getting the output for each input or after getting the output for all inputs in a batch?</p>
  </blockquote>
  <p>You can perform back-propagation using (or after) only one training <em>input</em> (also known as <em>data point</em>, <em>example</em>, <em>sample</em> or <em>observation</em>) or multiple ones (a <em>batch</em>). However, the loss function to train the neural network is <em>slightly</em> different in both cases.</p>
  <blockquote>
  <p>Is back-propagation applied <span class="math-container">$n$</span> times till the neural network gives a satisfactory result for a single data point before going to work on the next data point?</p>
  </blockquote>
  <p>If we use only one example, we usually do not wait until the neural network gives satisfactory results for a single input-label pair <span class="math-container">$(x_i, y_i)$</span>, but we keep feeding it with several input-label pairs, one after the other (and each time updating the parameters of the neural network using back-propagation), without <em>usually</em> caring whether the neural network already produces a satisfactory output for an input-label pair before feeding it with the next (although you could also do that).</p>
  <h1>Long answer (to the 1st question)</h1>
  <p>In case you want to know more about the first question, keep reading!</p>
  <h2>What is back-propagation?</h2>
  <p>Back-propagation is the (automatic) process of <strong>differentiating</strong> the <strong>loss function</strong>, which we use to train the neural network, <span class="math-container">$\mathcal{L}$</span>, with respect to all of the parameters (or weights) of the same neural network. If you collect the <span class="math-container">$N$</span> parameters of the neural network in a vector</p>
  <p><span class="math-container">$$\boldsymbol{\theta} = 
  \begin{bmatrix}
  \theta_1\\
  \vdots \\
  \theta_N
  \end{bmatrix},
  $$</span></p>
  <p>then the derivative of the loss function <span class="math-container">$\mathcal{L}$</span> with respect to <span class="math-container">$\boldsymbol{\theta}$</span> is called the <a href="https://en.wikipedia.org/wiki/Gradient" rel="noreferrer"><strong>gradient</strong></a>, which is a vector that contains the <strong>partial derivatives</strong> of the loss function with respect to each single scalar parameter of the network, <span class="math-container">$\theta_i$</span>, for <span class="math-container">$i=1, \dots, N$</span>, that is, the gradient looks something like this</p>
  <p><span class="math-container">$$
  \nabla \mathcal{L} =
  \begin{bmatrix}
  \frac{\partial \mathcal{L}}{ \partial \theta_1}\\
  \vdots \\
  \frac{\partial \mathcal{L}}{ \partial \theta_N}
  \end{bmatrix},
  $$</span></p>
  <p>where the symbol <span class="math-container">$\nabla$</span> denotes the gradient of the function <span class="math-container">$\mathcal{L}$</span>.</p>
  <h2>Loss functions</h2>
  <p>The specific loss function <span class="math-container">$\mathcal{L}$</span> that is used to train the neural network depends on the problem that we need to solve. For example, if we need to solve a <strong>regression problem</strong> (i.e. predict a real number), then the <strong>mean squared error (MSE)</strong> can be used. If we need to solve a <strong>classification problem</strong> (i.e. predict a class), the <strong>cross-entropy (CE)</strong> (aka <em>negative log-likelihood</em>) may be used instead.</p>
  <h2>Example</h2>
  <p>Let us assume that we need to solve a regression problem, so we choose the mean squared error (MSE) as the loss function. For simplicity, let's also assume that the neural network, denoted by <span class="math-container">$f_{\boldsymbol{\theta}}$</span>, contains only one output neuron and contains no biases.</p>
  <h3>Stochastic gradient descent</h3>
  <p>Given an input-label pair <span class="math-container">$(x_i, y_i) \in D$</span> (where <span class="math-container">$D$</span> is a dataset of input-label pairs), the <strong>squared error</strong> function (not the <em>mean</em> squared error yet!) is defined as</p>
  <p><span class="math-container">$$\mathcal{L}_i(\boldsymbol{\theta}) = \frac{1}{2} (f_{\boldsymbol{\theta}}(x_i) - y_i)^2,$$</span></p>
  <p>where <span class="math-container">$f_{\boldsymbol{\theta}}(x_i) = \hat{y}_i$</span> is the output of the neural network for the the data point <span class="math-container">$x_i$</span> (which depends on the specific values of <span class="math-container">$\boldsymbol{\theta}$</span>) and <span class="math-container">$y_i$</span> is the corresponding ground-truth <strong>label</strong>.</p>
  <p>As the name suggests, <span class="math-container">$\mathcal{L}_i$</span> (where the subscript <span class="math-container">$_i$</span> is only used to refer to the specific input-label pair <span class="math-container">$(x_i, y_i)$</span>) measures the squared <strong>error</strong> (i.e. some notion of distance) between the current prediction (or output) of the neural network, <span class="math-container">$\hat{y}_i$</span>, and the <em>expected</em> output for the given input <span class="math-container">$x_i$</span>, i.e. <span class="math-container">$y_i$</span>.</p>
  <p>We can differentiate <span class="math-container">$\mathcal{L}_i(\boldsymbol{\theta})$</span> with respect to the parameters of the neural network, <span class="math-container">$\boldsymbol{\theta}$</span>. However, given that the details of back-propagation can easily become tedious, I will not describe them here. You can find more details <a href="http://neuralnetworksanddeeplearning.com/chap2.html" rel="noreferrer">here</a>.</p>
  <p>So, let me assume that we have a computer program that is able to compute <span class="math-container">$\nabla \mathcal{L}_i$</span>. At that point, we can perform one step of the <strong>gradient descent</strong> algorithm</p>
  <p><span class="math-container">$$
  \boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \gamma \nabla \mathcal{L}_i, \label{1}\tag{1}
  $$</span></p>
  <p>where <span class="math-container">$\gamma$</span> is the <strong>learning rate</strong> and <span class="math-container">$\leftarrow$</span> is the <em>assignment</em> operator. Note that <span class="math-container">$\boldsymbol{\theta}$</span> and <span class="math-container">$\nabla \mathcal{L}_i$</span> have the same dimensions, <span class="math-container">$N$</span>.</p>
  <p>So, I have just shown you that you can update the parameters of the neural network using only one input-label pair, <span class="math-container">$(x_i, y_i)$</span>. This way of performing GD with only one input-label pair is known as <strong>stochastic gradient descent (SGD)</strong>.</p>
  <h3>Batch (or mini-batch) gradient descent</h3>
  <p>In practice, for several reasons (including learning instability and inefficiency), it is rarely the case that you update the parameters using only one input-label pair <span class="math-container">$(x_i, y_i)$</span>. Instead, you use multiple input-label pairs, which are collected in a so-called <strong>batch</strong></p>
  <p><span class="math-container">$$B = \{(x_1, y_i), \dots, (x_M, y_M) \},$$</span></p>
  <p>where <span class="math-container">$M$</span> is the size of <strong>batch</strong> <span class="math-container">$B$</span>, which is also known as <strong>mini-batch</strong> when <span class="math-container">$M$</span> is smaller than the total number of input-label pairs in the training dataset, i.e. when <span class="math-container">$|B| = M &lt; |D|$</span>. If you use mini-batches, typical values for <span class="math-container">$M$</span> are <span class="math-container">$32$</span>, <span class="math-container">$64$</span>, <span class="math-container">$128$</span>, and <span class="math-container">$256$</span> (yes, powers of 2: can you guess why?). See <a href="https://ai.stackexchange.com/q/8560/2444">this question for other details</a>.</p>
  <p>In this case, the loss function <span class="math-container">$\mathcal{L}_M(\boldsymbol{\theta})$</span> is defined as as the <strong>mean</strong> (or <em>average</em>) of the squared errors for single input-label pairs, <span class="math-container">$\mathcal{L}_i(\boldsymbol{\theta})$</span>, i.e.</p>
  <p><span class="math-container">\begin{align}
  \mathcal{L}_M(\boldsymbol{\theta}) 
  &amp;= \frac{1}{M} \sum_{i=1}^M \mathcal{L}_i(\boldsymbol{\theta}) \\
  &amp;= \frac{1}{M} \sum_{i=1}^M \frac{1}{2} (f_{\boldsymbol{\theta}}(x_i)-y_i)^2 \\
  &amp;= \frac{1}{M} \frac{1}{2} \sum_{i=1}^M (f_{\boldsymbol{\theta}}(x_i)-y_i)^2.
  \end{align}</span>
  The normalisation factor, <span class="math-container">$\frac{1}{M}$</span>, can be thought of as <em>averaging out</em> the losses of all input-label pairs. Note also that we can take the <span class="math-container">$\frac{1}{2}$</span> out of the summation because it is a constant with respect to the variable of the summation, <span class="math-container">$i$</span>.</p>
  <p>In this case, let me also assume that we are able to compute (using back-propagation) the gradient of <span class="math-container">$\mathcal{L}_M$</span>, so that we can perform a gradient descent (GD) update (using a batch of examples)</p>
  <p><span class="math-container">$$
  \boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \gamma \nabla \mathcal{L}_M \label{2} \tag{2}
  $$</span>
  The only thing that changes with respect to the GD update in equation \ref{1} is the loss function, which is now <span class="math-container">$\mathcal{L}_M$</span> rather than <span class="math-container">$\mathcal{L}_i$</span>.</p>
  <p>This is known as <strong>mini-batch</strong> (or <strong>batch</strong>) <strong>gradient descent</strong>. In the case <span class="math-container">$M = |D|$</span>, this is simply known as <strong>gradient descent</strong>, which is a term that can also be used to refer to any of its variants (including SGD or mini-batch GD).</p>
  <h3>Further reading</h3>
  <p>You may also be interested in <a href="https://ai.stackexchange.com/a/20380/2444">this answer</a>, which provides more details about mini-batch gradient descent.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-05T15:07:23.873Z
Answers.Text: Thanks!.  In the case of a batch, can I assume that the weight adjust factor for a given parameter w1 is an average weight adjust factor for all items in the batch?
Answers.UserDisplayName: Maanu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-05T15:16:15.570Z
Answers.Text: @Maanu I would think of the update of a weight using a batch as an update that contains more info (than using only just one training example). This is also why training using a batches (rather than single examples) is usually more stable, that is, the NN tends to approximate the desidered function more rapidly. However, note that training using big batches can also require a lot of memory. So, there is a trade-off between memory requirements and stability of the training process.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-22T15:44:18.847Z
Answers.Text: I have joined this community now, just because I wanted to thank you for this clear answer, after weeks of searching
Answers.UserDisplayName: Osama Mohammed
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-30T18:06:47.007Z
Answers.Text: @nbro, for the mini-batch case, the losses across the different points are averaged ('normalized') but does backpropagation not produce a gradient vector for all the weights based upon each pair (x_i,y_i)? The losses are averaged over the pairs and if each pair has a distinct gradient vector of the parameters how are those different gradients used to update the weights in that single step of eq2?
Answers.UserDisplayName: Vass
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-30T19:20:18.203Z
Answers.Text: @Vass Maybe [this](https://ai.stackexchange.com/questions/20377/what-exactly-is-averaged-when-doing-batch-gradient-descent?noredirect=1&lq=1) can be useful.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-05T14:18:28.067Z
Answers.LastActivityDate: 2020-11-29T23:56:30.773Z
Answers.LastEditDate: 2020-11-29T23:56:30.773Z
Answers.UserDisplayName: nbro
Body: <p>I am new to deep learning and trying to understand the concept of back-propagation. I have a doubt about when the back-propagation is applied.  Assume that I have a training data set of 1000 images for handwritten letters,</p>
  <ol>
  <li><p>Is back-propagation applied immediately after getting the output for each input or after getting the output for all inputs in a batch?</p>
  </li>
  <li><p>Is back-propagation applied <span class="math-container">$n$</span> times till the neural network gives a satisfactory result for a single data point before going to work on the next data point?</p>
  </li>
  </ol>
  
Tags: <neural-networks><backpropagation><gradient-descent><stochastic-gradient-descent><mini-batch-gradient-descent>
Title: Is back-propagation applied for each data point or for a batch of data points?
UserDisplayName: Maanu
ThreadText: # Is back-propagation applied for each data point or for a batch of data points?
  
  (Asked by: username_0 on 2019-04-05)
  
  I am new to deep learning and trying to understand the concept of back-propagation. I have a doubt about when the back-propagation is applied. Assume that I have a training data set of 1000 images for handwritten letters,
  
  1. Is back-propagation applied immediately after getting the output for each input or after getting the output for all inputs in a batch?
  2. Is back-propagation applied $n$ times till the neural network gives a satisfactory result for a single data point before going to work on the next data point?
  
  **Tags:** neural-networks, backpropagation, gradient-descent, stochastic-gradient-descent, mini-batch-gradient-descent
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-04-05. Score: 14 | Accepted answer
  
  # Short answers
  
  > Is back-propagation applied immediately after getting the output for each input or after getting the output for all inputs in a batch?
  
  You can perform back-propagation using (or after) only one training *input* (also known as *data point*, *example*, *sample* or *observation*) or multiple ones (a *batch*). However, the loss function to train the neural network is *slightly* different in both cases.
  
  > Is back-propagation applied $n$ times till the neural network gives a satisfactory result for a single data point before going to work on the next data point?
  
  If we use only one example, we usually do not wait until the neural network gives satisfactory results for a single input-label pair $(x\_i, y\_i)$, but we keep feeding it with several input-label pairs, one after the other (and each time updating the parameters of the neural network using back-propagation), without *usually* caring whether the neural network already produces a satisfactory output for an input-label pair before feeding it with the next (although you could also do that).
  
  # Long answer (to the 1st question)
  
  In case you want to know more about the first question, keep reading!
  
  ## What is back-propagation?
  
  Back-propagation is the (automatic) process of **differentiating** the **loss function**, which we use to train the neural network, $\mathcal{L}$, with respect to all of the parameters (or weights) of the same neural network. If you collect the $N$ parameters of the neural network in a vector
  
  $$\boldsymbol{\theta} =
  \begin{bmatrix}
  \theta\_1\\
  \vdots \\
  \theta\_N
  \end{bmatrix},
  $$
  
  then the derivative of the loss function $\mathcal{L}$ with respect to $\boldsymbol{\theta}$ is called the [**gradient**](https://en.wikipedia.org/wiki/Gradient), which is a vector that contains the **partial derivatives** of the loss function with respect to each single scalar parameter of the network, $\theta\_i$, for $i=1, \dots, N$, that is, the gradient looks something like this
  
  $$
  \nabla \mathcal{L} =
  \begin{bmatrix}
  \frac{\partial \mathcal{L}}{ \partial \theta\_1}\\
  \vdots \\
  \frac{\partial \mathcal{L}}{ \partial \theta\_N}
  \end{bmatrix},
  $$
  
  where the symbol $\nabla$ denotes the gradient of the function $\mathcal{L}$.
  
  ## Loss functions
  
  The specific loss function $\mathcal{L}$ that is used to train the neural network depends on the problem that we need to solve. For example, if we need to solve a **regression problem** (i.e. predict a real number), then the **mean squared error (MSE)** can be used. If we need to solve a **classification problem** (i.e. predict a class), the **cross-entropy (CE)** (aka *negative log-likelihood*) may be used instead.
  
  ## Example
  
  Let us assume that we need to solve a regression problem, so we choose the mean squared error (MSE) as the loss function. For simplicity, let's also assume that the neural network, denoted by $f\_{\boldsymbol{\theta}}$, contains only one output neuron and contains no biases.
  
  ### Stochastic gradient descent
  
  Given an input-label pair $(x\_i, y\_i) \in D$ (where $D$ is a dataset of input-label pairs), the **squared error** function (not the *mean* squared error yet!) is defined as
  
  $$\mathcal{L}\_i(\boldsymbol{\theta}) = \frac{1}{2} (f\_{\boldsymbol{\theta}}(x\_i) - y\_i)^2,$$
  
  where $f\_{\boldsymbol{\theta}}(x\_i) = \hat{y}\_i$ is the output of the neural network for the the data point $x\_i$ (which depends on the specific values of $\boldsymbol{\theta}$) and $y\_i$ is the corresponding ground-truth **label**.
  
  As the name suggests, $\mathcal{L}\_i$ (where the subscript $\_i$ is only used to refer to the specific input-label pair $(x\_i, y\_i)$) measures the squared **error** (i.e. some notion of distance) between the current prediction (or output) of the neural network, $\hat{y}\_i$, and the *expected* output for the given input $x\_i$, i.e. $y\_i$.
  
  We can differentiate $\mathcal{L}\_i(\boldsymbol{\theta})$ with respect to the parameters of the neural network, $\boldsymbol{\theta}$. However, given that the details of back-propagation can easily become tedious, I will not describe them here. You can find more details [here](http://neuralnetworksanddeeplearning.com/chap2.html).
  
  So, let me assume that we have a computer program that is able to compute $\nabla \mathcal{L}\_i$. At that point, we can perform one step of the **gradient descent** algorithm
  
  $$
  \boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \gamma \nabla \mathcal{L}\_i, \label{1}\tag{1}
  $$
  
  where $\gamma$ is the **learning rate** and $\leftarrow$ is the *assignment* operator. Note that $\boldsymbol{\theta}$ and $\nabla \mathcal{L}\_i$ have the same dimensions, $N$.
  
  So, I have just shown you that you can update the parameters of the neural network using only one input-label pair, $(x\_i, y\_i)$. This way of performing GD with only one input-label pair is known as **stochastic gradient descent (SGD)**.
  
  ### Batch (or mini-batch) gradient descent
  
  In practice, for several reasons (including learning instability and inefficiency), it is rarely the case that you update the parameters using only one input-label pair $(x\_i, y\_i)$. Instead, you use multiple input-label pairs, which are collected in a so-called **batch**
  
  $$B = \{(x\_1, y\_i), \dots, (x\_M, y\_M) \},$$
  
  where $M$ is the size of **batch** $B$, which is also known as **mini-batch** when $M$ is smaller than the total number of input-label pairs in the training dataset, i.e. when $|B| = M < |D|$. If you use mini-batches, typical values for $M$ are $32$, $64$, $128$, and $256$ (yes, powers of 2: can you guess why?). See [this question for other details](https://ai.stackexchange.com/q/8560/2444).
  
  In this case, the loss function $\mathcal{L}\_M(\boldsymbol{\theta})$ is defined as as the **mean** (or *average*) of the squared errors for single input-label pairs, $\mathcal{L}\_i(\boldsymbol{\theta})$, i.e.
  
  \begin{align}
  \mathcal{L}\_M(\boldsymbol{\theta})
  &= \frac{1}{M} \sum\_{i=1}^M \mathcal{L}\_i(\boldsymbol{\theta}) \\
  &= \frac{1}{M} \sum\_{i=1}^M \frac{1}{2} (f\_{\boldsymbol{\theta}}(x\_i)-y\_i)^2 \\
  &= \frac{1}{M} \frac{1}{2} \sum\_{i=1}^M (f\_{\boldsymbol{\theta}}(x\_i)-y\_i)^2.
  \end{align}
  The normalisation factor, $\frac{1}{M}$, can be thought of as *averaging out* the losses of all input-label pairs. Note also that we can take the $\frac{1}{2}$ out of the summation because it is a constant with respect to the variable of the summation, $i$.
  
  In this case, let me also assume that we are able to compute (using back-propagation) the gradient of $\mathcal{L}\_M$, so that we can perform a gradient descent (GD) update (using a batch of examples)
  
  $$
  \boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \gamma \nabla \mathcal{L}\_M \label{2} \tag{2}
  $$
  The only thing that changes with respect to the GD update in equation \ref{1} is the loss function, which is now $\mathcal{L}\_M$ rather than $\mathcal{L}\_i$.
  
  This is known as **mini-batch** (or **batch**) **gradient descent**. In the case $M = |D|$, this is simply known as **gradient descent**, which is a term that can also be used to refer to any of its variants (including SGD or mini-batch GD).
  
  ### Further reading
  
  You may also be interested in [this answer](https://ai.stackexchange.com/a/20380/2444), which provides more details about mini-batch gradient descent.
  
  > **Comments:**
  > **username_0**: Thanks!.  In the case of a batch, can I assume that the weight adjust factor for a given parameter w1 is an average weight adjust factor for all items in the batch?  
  > **username_1**: @username_0 I would think of the update of a weight using a batch as an update that contains more info (than using only just one training example). This is also why training using a batches (rather than single examples) is usually more stable, that is, the NN tends to approximate the desidered function more rapidly. However, note that training using big batches can also require a lot of memory. So, there is a trade-off between memory requirements and stability of the training process.  
  > **username_2**: I have joined this community now, just because I wanted to thank you for this clear answer, after weeks of searching  
  > **username_3**: @username_1, for the mini-batch case, the losses across the different points are averaged ('normalized') but does backpropagation not produce a gradient vector for all the weights based upon each pair (x_i,y_i)? The losses are averaged over the pairs and if each pair has a distinct gradient vector of the parameters how are those different gradients used to update the weights in that single step of eq2?  
  > **username_1**: @username_3 Maybe [this](https://ai.stackexchange.com/questions/20377/what-exactly-is-averaged-when-doing-batch-gradient-descent?noredirect=1&lq=1) can be useful.  
  
  ---
  
  