Answers.Body: <p>In deep learning, the <strong>error function</strong> is sometimes known as <strong>loss function</strong> or <strong>cost function</strong> (but I do not exclude the possibility that these terms/expressions have also been used to refer to different although related functions, so you should take into account your context).</p>
  <p>In <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.332.356&amp;rep=rep1&amp;type=pdf" rel="nofollow noreferrer">statistical learning theory</a>, the function that you want to minimize is known as <strong>expected risk</strong></p>
  <p><span class="math-container">\begin{align}
  R(\theta)
  &amp;=
  \int L(y, f_{\theta}(x)) d P(x, y) \\
  &amp;=
  \mathbb{E}_{P(x, y)} \left[ L(y, f_{\theta}(x)) \right],
  \end{align}</span>
  where <span class="math-container">$\theta$</span> are the parameters of your model <span class="math-container">$f$</span>, and the function that you actually minimize is the <strong>empirical risk</strong> given the dataset <span class="math-container">$D = \{ (x_i, y_i) \}_{i=1}^n$</span>,</p>
  <p><span class="math-container">$$
  R_{\mathrm{emp}}(\theta)=\frac{1}{n} \sum_{i=1}^{n}L(y_i, f_{\theta}(x_i)),
  $$</span>
  which is a <em>generalization</em> of the commonly used cost functions, such as <em>mean squared error (MSE)</em> or <em>binary cross-entropy</em>. For example, in the case of MSE, <span class="math-container">$L = \left(y_{i}-f_\theta(x_i)\right)^{2}$</span>, which can be called the <em>loss function</em> (though this terminology may not be standard).</p>
  <p>You optimize the empirical risk, a <a href="https://ai.stackexchange.com/a/23020/2444">proxy</a> for the expected risk, because the expected risk is <em>incomputable</em> (given that <span class="math-container">$P(x, y)$</span> is generally <em>unknown</em>), so, in this sense, the expected risk is unknown.</p>
  <p>If you use the term <em>error function</em> to refer to the expected risk, then, yes, the error function is typically unknown, but <em>error function</em> is typically used to refer to an instance of the empirical risk, so, in that case, it is known and computable.</p>
  <p>Note that I purposely used the term <em>loss function above</em> to refer to <span class="math-container">$L$</span> and the term <em>cost function</em> to refer to the <em>empirical risks</em>, such as the MSE (i.e., in this case, I did not use <em>loss function</em> and <em>cost function</em> as synonyms), which shows that terminology is not always used consistently, so take into account your context.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-22T17:32:44.817Z
Answers.Text: I am not sure if this answers your question, given that I don't fully understand your doubts. If not, please, let me know.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-22T18:29:53.477Z
Answers.Text: I got a little confused when I looked up all the explanations on how the error is considered.
   E.g. when the error function is mentioned sth like E=(y'-y)^2 is meant. and a parabola is shown. Sometimes however (maybe in another context) I see a sort of scatter plot that just shows how big an error is (for every entry maybe in chronological order as the batch is passed through the net). I am just asking what is what.
Answers.UserDisplayName: MScott
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-22T19:06:19.000Z
Answers.Text: @MScott As I said, sometimes it depends on the context. $(y' - y)^2$ could be called the **error function** and the sum of all errors could be called the **empirical risk** or often in the context of deep learning also the **error function** (yes, error function again!).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-22T17:26:39.680Z
Answers.LastActivityDate: 2020-08-23T13:30:53.610Z
Answers.LastEditDate: 2020-08-23T13:30:53.610Z
Answers.UserDisplayName: nbro
Body: <p>What is the error function? Is it the same as the cost function?</p>
  <p>Is the error function known or unknown?</p>
  <p>When I get the outcome of a neural net I compare it with the target value. The difference between both is called the error. When I get mutiple error values e.g. when I pass a batch through the NN I will get as many error value as the size of my batch. Is the error function the plot of the points? If yes, to me the error function would be unknown. I would only know some point on the graph of the error function.</p>
  
Tags: <neural-networks><terminology><objective-functions>
Title: Is the error function known or unknown?
UserDisplayName: MScott
ThreadText: # Is the error function known or unknown?
  
  (Asked by: username_0 on 2020-08-22)
  
  What is the error function? Is it the same as the cost function?
  
  Is the error function known or unknown?
  
  When I get the outcome of a neural net I compare it with the target value. The difference between both is called the error. When I get mutiple error values e.g. when I pass a batch through the NN I will get as many error value as the size of my batch. Is the error function the plot of the points? If yes, to me the error function would be unknown. I would only know some point on the graph of the error function.
  
  **Tags:** neural-networks, terminology, objective-functions
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-08-22. Score: 2
  
  In deep learning, the **error function** is sometimes known as **loss function** or **cost function** (but I do not exclude the possibility that these terms/expressions have also been used to refer to different although related functions, so you should take into account your context).
  
  In [statistical learning theory](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.332.356&rep=rep1&type=pdf), the function that you want to minimize is known as **expected risk**
  
  \begin{align}
  R(\theta)
  &=
  \int L(y, f\_{\theta}(x)) d P(x, y) \\
  &=
  \mathbb{E}\_{P(x, y)} \left[ L(y, f\_{\theta}(x)) \right],
  \end{align}
  where $\theta$ are the parameters of your model $f$, and the function that you actually minimize is the **empirical risk** given the dataset $D = \{ (x\_i, y\_i) \}\_{i=1}^n$,
  
  $$
  R\_{\mathrm{emp}}(\theta)=\frac{1}{n} \sum\_{i=1}^{n}L(y\_i, f\_{\theta}(x\_i)),
  $$
  which is a *generalization* of the commonly used cost functions, such as *mean squared error (MSE)* or *binary cross-entropy*. For example, in the case of MSE, $L = \left(y\_{i}-f\_\theta(x\_i)\right)^{2}$, which can be called the *loss function* (though this terminology may not be standard).
  
  You optimize the empirical risk, a [proxy](https://ai.stackexchange.com/a/23020/2444) for the expected risk, because the expected risk is *incomputable* (given that $P(x, y)$ is generally *unknown*), so, in this sense, the expected risk is unknown.
  
  If you use the term *error function* to refer to the expected risk, then, yes, the error function is typically unknown, but *error function* is typically used to refer to an instance of the empirical risk, so, in that case, it is known and computable.
  
  Note that I purposely used the term *loss function above* to refer to $L$ and the term *cost function* to refer to the *empirical risks*, such as the MSE (i.e., in this case, I did not use *loss function* and *cost function* as synonyms), which shows that terminology is not always used consistently, so take into account your context.
  
  > **Comments:**
  > **username_1**: I am not sure if this answers your question, given that I don't fully understand your doubts. If not, please, let me know.  
  > **username_0**: I got a little confused when I looked up all the explanations on how the error is considered.  E.g. when the error function is mentioned sth like E=(y'-y)^2 is meant. and a parabola is shown. Sometimes however (maybe in another context) I see a sort of scatter plot that just shows how big an error is (for every entry maybe in chronological order as the batch is passed through the net). I am just asking what is what.  
  > **username_1**: @username_0 As I said, sometimes it depends on the context. $(y' - y)^2$ could be called the **error function** and the sum of all errors could be called the **empirical risk** or often in the context of deep learning also the **error function** (yes, error function again!).  
  
  ---
  
  