Answers.Body: <p>The objective in parametric ML is to find parameter values that minimize the overall cost function <span class="math-container">$J(w)$</span> across the entire training dataset, whether in terms of the framework of empirical risk minimization (ERM) or information-theoretic cross-entropy (CE). Intuitively minimizing the average cost over <em>all</em> examples leads to a more <em>generalizable</em> model as it helps the model generalize well to unseen data rather than being influenced too heavily by the noise or fluctuations in individual examples.</p>
  <p>Having said that, in the common stochastic gradient descent (SGD) training method, if your minibatch size is 1 then effectively during each batch you're updating the parameters w.r.t the loss of the single current sample, and this can be proved to be convergent under certain conditions albeit at a lower rate as referenced <a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent" rel="nofollow noreferrer">here</a>:</p>
  <blockquote>
  <p>The sum-minimization problem also arises for empirical risk minimization. There,
  <span class="math-container">${Q_{i}(w)}$</span> is the value of the loss function at <span class="math-container">${i}$</span>-th example, and <span class="math-container">${Q(w)}$</span> is the empirical risk... However, in other cases, evaluating the sum-gradient may require expensive evaluations of the gradients from all summand functions. When the training set is enormous and no simple formulas exist, evaluating the sums of gradients becomes very expensive, because evaluating the gradient requires evaluating all the summand functions' gradients. To economize on the computational cost at every iteration, stochastic gradient descent samples a subset of summand functions at every step.</p>
  </blockquote>
  <blockquote>
  <p>The convergence of stochastic gradient descent has been analyzed using the theories of convex minimization and of stochastic approximation. Briefly, when the learning rates <span class="math-container">${\eta }$</span> decrease with an appropriate rate, and subject to relatively mild assumptions, stochastic gradient descent converges almost surely to a global minimum when the objective function is convex or pseudoconvex, and otherwise converges almost surely to a local minimum. This is in fact a consequence of the Robbins–Siegmund theorem.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-09T06:36:46.493Z
Answers.LastActivityDate: 2024-03-09T06:36:46.493Z
Answers.UserDisplayName: cinch
Answers.Body: <p>Because you want to do ERM, which aims at minimizing:
  <span class="math-container">$$
  \mathbb{E}_{x,y \sim p}[L(h(x))]
  $$</span></p>
  <p>Given this, if we apply a monte carlo approach to estimate that expected value, you end up with the mean of <span class="math-container">$N$</span> samples... the higher it is <span class="math-container">$N$</span>, the better is the approximation (thus the lower the variance)</p>
  <p>You can for sure update at each iteration wrt the loss of the single current sample, however you will struggle to converge</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-07T15:11:12.467Z
Answers.LastActivityDate: 2024-03-07T15:11:12.467Z
Answers.UserDisplayName: Alberto
Body: <p>Assuming that the cost function <span class="math-container">$J$</span> is the average of the loss function <span class="math-container">$\mathcal{L}$</span> over all training examples <span class="math-container">$m$</span></p>
  <p><span class="math-container">$J(w) = \frac{1}{m} \displaystyle\sum_{i=0}^{m} \mathcal{L}$</span></p>
  <p>why do we update the parameters with respect to the cost function</p>
  <p><span class="math-container">$w := w - \alpha \frac{dJ}{dw}$</span></p>
  <p>instead of updating the parameters at each iteration with respect to its loss?</p>
  <pre><code>for i in m:
      w = w - learning_rate*(dL/dw)
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-07T14:50:01.343Z
Comments.Text: Can you provide the source where this is done? Do you know what is the derivative of a summation?
Comments.UserDisplayName: nbro
Tags: <deep-learning><backpropagation>
Title: Why do we update parameters with respect to cost instead of loss?
UserDisplayName: Aditya Tomar
ThreadText: # Why do we update parameters with respect to cost instead of loss?
  
  (Asked by: username_0 on 2024-03-07)
  
  Assuming that the cost function $J$ is the average of the loss function $\mathcal{L}$ over all training examples $m$
  $J(w) = \frac{1}{m} \displaystyle\sum_{i=0}^{m} \mathcal{L}$
  why do we update the parameters with respect to the cost function
  $w := w - \alpha \frac{dJ}{dw}$
  instead of updating the parameters at each iteration with respect to its loss?
  for i in m:
      w = w - learning_rate*(dL/dw)
  
  **Tags:** deep-learning, backpropagation
  > **Comments:**
  > **username_1**: Can you provide the source where this is done? Do you know what is the derivative of a summation?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-03-09. Score: 2 | Accepted answer
  
  The objective in parametric ML is to find parameter values that minimize the overall cost function $J(w)$ across the entire training dataset, whether in terms of the framework of empirical risk minimization (ERM) or information-theoretic cross-entropy (CE). Intuitively minimizing the average cost over *all* examples leads to a more *generalizable* model as it helps the model generalize well to unseen data rather than being influenced too heavily by the noise or fluctuations in individual examples.
  
  Having said that, in the common stochastic gradient descent (SGD) training method, if your minibatch size is 1 then effectively during each batch you're updating the parameters w.r.t the loss of the single current sample, and this can be proved to be convergent under certain conditions albeit at a lower rate as referenced [here](https://en.wikipedia.org/wiki/Stochastic_gradient_descent):
  
  > The sum-minimization problem also arises for empirical risk minimization. There,
  > ${Q\_{i}(w)}$ is the value of the loss function at ${i}$-th example, and ${Q(w)}$ is the empirical risk... However, in other cases, evaluating the sum-gradient may require expensive evaluations of the gradients from all summand functions. When the training set is enormous and no simple formulas exist, evaluating the sums of gradients becomes very expensive, because evaluating the gradient requires evaluating all the summand functions' gradients. To economize on the computational cost at every iteration, stochastic gradient descent samples a subset of summand functions at every step.
  
  > The convergence of stochastic gradient descent has been analyzed using the theories of convex minimization and of stochastic approximation. Briefly, when the learning rates ${\eta }$ decrease with an appropriate rate, and subject to relatively mild assumptions, stochastic gradient descent converges almost surely to a global minimum when the objective function is convex or pseudoconvex, and otherwise converges almost surely to a local minimum. This is in fact a consequence of the Robbins–Siegmund theorem.
  
  
  ---
  
  ### Answer by username_3 on 2024-03-07. Score: 2
  
  Because you want to do ERM, which aims at minimizing:
  $$
  \mathbb{E}\_{x,y \sim p}[L(h(x))]
  $$
  
  Given this, if we apply a monte carlo approach to estimate that expected value, you end up with the mean of $N$ samples... the higher it is $N$, the better is the approximation (thus the lower the variance)
  
  You can for sure update at each iteration wrt the loss of the single current sample, however you will struggle to converge
  
  
  ---
  
  