Answers.Body: <p>The first variation is named "<span class="math-container">$E_{total}$</span>". It contains a sum which is not very well-specified (has no index, no limits). <strong>Rewriting it using the notation of the second variation</strong> would lead to:</p>
  
  <p><span class="math-container">$$E_{total} = \sum_{i = 1}^m \frac{1}{2} \left( y^{(i)} - h_{\theta}(x^{(i)}) \right)^2,$$</span></p>
  
  <p>where:</p>
  
  <ul>
  <li><span class="math-container">$x^{(i)}$</span> denotes the <span class="math-container">$i$</span>th training example</li>
  <li><span class="math-container">$h_{\theta}(x^{(i)})$</span> denotes the model's output for that instance/example</li>
  <li><span class="math-container">$y^{(i)}$</span> denotes the ground truth / target / label for that instance</li>
  <li><span class="math-container">$m$</span> denotes the number of training examples</li>
  </ul>
  
  <p>Because the term inside the large brackets is squared, the sign doesn't matter, so we can rewrite it (switch around the subtracted terms) to:</p>
  
  <p><span class="math-container">$$E_{total} = \sum_{i = 1}^m \frac{1}{2} \left( h_{\theta}(x^{(i)}) - y^{(i)} \right)^2.$$</span></p>
  
  <hr>
  
  <p>Now it already looks quite a lot like your second variation. </p>
  
  <p>The second variation does still have a <span class="math-container">$\frac{1}{m}$</span> terms outside the sum. That is because your <strong>second variation computes the mean squared error over all the training examples</strong>, rather than the <strong>total error computed by the first variation</strong>. </p>
  
  <p>Either error can be used for training. I'd personally lean towards using the <strong>mean error</strong> rather than the <strong>total error</strong>, mainly because the <strong>scale of the mean error is independent of the batch size <span class="math-container">$m$</span></strong>, whereas the <strong>scale of the total error is proportional to the batch size</strong> used for training. Either option is valid, but <strong>they'll likely require different hyperparameter values (especially for the learning rate), due to the difference in scale</strong>.</p>
  
  <hr>
  
  <p>With that <span class="math-container">$\frac{1}{m}$</span> term explained, the only remaining difference is the <span class="math-container">$\frac{1}{2}$</span> term inside the sum (can also be pulled out of the sum), which is present in the first variation but not in the second. The reason for including that term is given <a href="https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/" rel="nofollow noreferrer">in the page you linked to for the first variation</a>:</p>
  
  <blockquote>
    <p>The <span class="math-container">$\frac{1}{2}$</span> is included so that exponent is cancelled when we differentiate later on. The result is eventually multiplied by a learning rate anyway so it doesn’t matter that we introduce a constant here.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-22T20:34:07.790Z
Answers.Text: **question**: Notice in variation 1 he just uses a sigma symbol without specifying what to sum, can he do that? Don't he have to denote the start and end value? e.g. "start: i=0" and "end: m" as in the second variation.
Answers.UserDisplayName: Sebastian Nielsen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-23T08:47:53.530Z
Answers.Text: @SebastianNielsen I don't think it's entirely correct to do it like that mathematically. It's not uncommon though, when it is relatively "obvious" what we're summing over (e.g. summing over all training examples), it can be convenient to simply leave the indices out. In my opinion, nothing should be considered "obvious" though in what seems to be very much an introductory-level tutorial thing.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-22T18:55:34.987Z
Answers.LastActivityDate: 2018-09-22T18:55:34.987Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>Depending on the source, I find people using different variations of the &quot;squared error function&quot;. How come that be?</p>
  <p><a href="https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/" rel="nofollow noreferrer">Here</a>, it is defined as</p>
  <p><span class="math-container">$$
  E_{\text {total }}=\sum \frac{1}{2}(\text {target}-\text {output})^{2}
  $$</span></p>
  <p>OTOH, <a href="http://mccormickml.com/2014/03/04/gradient-descent-derivation/" rel="nofollow noreferrer">here</a>, it's defined as</p>
  <p><span class="math-container">$$
  \frac{1}{m} \sum_{i=1}^{m}\left(h_{\theta}\left(x^{(i)}\right)-y^{(i)}\right)^{2}
  $$</span></p>
  <p>Notice that it is being divided by 1 over <span class="math-container">$m$</span> as opposed to variation 1, where we multiply by <span class="math-container">$1/2$</span>.</p>
  <p>The stuff inside the <span class="math-container">$()^2$</span> is simply notation, I get that, but dividing by <span class="math-container">$1/m$</span> and <span class="math-container">$1/2$</span> will clearly get a different result. Which version is the &quot;correct&quot; one, or is there no such thing as a correct or &quot;official&quot; squared error function?</p>
  
Tags: <neural-networks><objective-functions><definitions><mean-squared-error>
Title: Why is the "square error function" sometimes defined with the constant 1/2 and sometimes with the constant 1/m?
UserDisplayName: Sebastian Nielsen
ThreadText: # Why is the "square error function" sometimes defined with the constant 1/2 and sometimes with the constant 1/m?
  
  (Asked by: username_0 on 2018-09-22)
  
  Depending on the source, I find people using different variations of the "squared error function". How come that be?
  
  [Here](https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/), it is defined as
  
  $$
  E\_{\text {total }}=\sum \frac{1}{2}(\text {target}-\text {output})^{2}
  $$
  
  OTOH, [here](http://mccormickml.com/2014/03/04/gradient-descent-derivation/), it's defined as
  
  $$
  \frac{1}{m} \sum\_{i=1}^{m}\left(h\_{\theta}\left(x^{(i)}\right)-y^{(i)}\right)^{2}
  $$
  
  Notice that it is being divided by 1 over $m$ as opposed to variation 1, where we multiply by $1/2$.
  
  The stuff inside the $()^2$ is simply notation, I get that, but dividing by $1/m$ and $1/2$ will clearly get a different result. Which version is the "correct" one, or is there no such thing as a correct or "official" squared error function?
  
  **Tags:** neural-networks, objective-functions, definitions, mean-squared-error
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-09-22. Score: 3 | Accepted answer
  
  The first variation is named "$E\_{total}$". It contains a sum which is not very well-specified (has no index, no limits). **Rewriting it using the notation of the second variation** would lead to:
  
  $$E\_{total} = \sum\_{i = 1}^m \frac{1}{2} \left( y^{(i)} - h\_{\theta}(x^{(i)}) \right)^2,$$
  
  where:
  
  * $x^{(i)}$ denotes the $i$th training example
  * $h\_{\theta}(x^{(i)})$ denotes the model's output for that instance/example
  * $y^{(i)}$ denotes the ground truth / target / label for that instance
  * $m$ denotes the number of training examples
  
  Because the term inside the large brackets is squared, the sign doesn't matter, so we can rewrite it (switch around the subtracted terms) to:
  
  $$E\_{total} = \sum\_{i = 1}^m \frac{1}{2} \left( h\_{\theta}(x^{(i)}) - y^{(i)} \right)^2.$$
  
  ---
  
  Now it already looks quite a lot like your second variation.
  
  The second variation does still have a $\frac{1}{m}$ terms outside the sum. That is because your **second variation computes the mean squared error over all the training examples**, rather than the **total error computed by the first variation**.
  
  Either error can be used for training. I'd personally lean towards using the **mean error** rather than the **total error**, mainly because the **scale of the mean error is independent of the batch size $m$**, whereas the **scale of the total error is proportional to the batch size** used for training. Either option is valid, but **they'll likely require different hyperparameter values (especially for the learning rate), due to the difference in scale**.
  
  ---
  
  With that $\frac{1}{m}$ term explained, the only remaining difference is the $\frac{1}{2}$ term inside the sum (can also be pulled out of the sum), which is present in the first variation but not in the second. The reason for including that term is given [in the page you linked to for the first variation](https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/):
  
  > The $\frac{1}{2}$ is included so that exponent is cancelled when we differentiate later on. The result is eventually multiplied by a learning rate anyway so it doesn’t matter that we introduce a constant here.
  
  > **Comments:**
  > **username_0**: **question**: Notice in variation 1 he just uses a sigma symbol without specifying what to sum, can he do that? Don't he have to denote the start and end value? e.g. "start: i=0" and "end: m" as in the second variation.  
  > **username_1**: @username_0 I don't think it's entirely correct to do it like that mathematically. It's not uncommon though, when it is relatively "obvious" what we're summing over (e.g. summing over all training examples), it can be convenient to simply leave the indices out. In my opinion, nothing should be considered "obvious" though in what seems to be very much an introductory-level tutorial thing.  
  
  ---
  
  