Answers.Body: <p>The loss is</p>
  <p><span class="math-container">$$\mathcal{L}=\sum_{i=1}^{N} \ell\left(y_{i}, f\left(\mathbf{x}_{i}\right)\right) \equiv \sum_{i=1}^{N} \exp \left(-y_{i} f\left(\mathbf{x}_{i}\right)\right),$$</span></p>
  <p>which can also be written as follows</p>
  <p><span class="math-container">$$\mathcal{L} = \sum_{i=1}^{N} e^{-y_{i} f\left(\mathbf{x}_{i}\right)} \tag{1}\label{1}$$</span></p>
  <p>The important thing to note here is the <span class="math-container">$-$</span> in the exponent, which allows us to write \ref{1} as follows (see <a href="https://web.archive.org/web/20210507020014/https://www.mesacc.edu/%7Escotz47781/mat120/notes/exponents/review/review.html" rel="nofollow noreferrer">this</a>)</p>
  <p><span class="math-container">$$\mathcal{L} = \sum_{i=1}^{N} \frac{1}{e^{y_{i} f\left(\mathbf{x}_{i}\right)}} \tag{2}\label{2}$$</span></p>
  <p>So, the loss becomes smaller the higher <span class="math-container">$y_{i}$</span> and <span class="math-container">$f\left(\mathbf{x}_{i}\right)$</span> are. If, for example, <span class="math-container">$y_{i}$</span> is negative and <span class="math-container">$f\left(\mathbf{x}_{i}\right)$</span> positive (or vice-versa), then \ref{2} would be higher. If both are positive or negative, then the loss will be smaller, as we will be summing fractions of the form <span class="math-container">$\frac{1}{e^k} &lt; 1$</span> for some positive constant <span class="math-container">$k$</span>: the higher the <span class="math-container">$k$</span>, the smaller the loss.</p>
  <p>So, if you use this loss, it seems that you want that</p>
  <ol>
  <li><span class="math-container">$y_{i}$</span> and <span class="math-container">$f\left(\mathbf{x}_{i}\right)$</span> have the same sign (either both negative or both positive)</li>
  <li><span class="math-container">$f\left(\mathbf{x}_{i}\right)$</span> is as high as possible (note that you cannot change <span class="math-container">$y_{i}$</span>), which could even be much higher (in terms of magnitude) than <span class="math-container">$y_{i}$</span> (not sure why they would want this)</li>
  </ol>
  <p>I don't know why they chose this loss function because I didn't read the paper (yet), but it seems to me that this is how you should interpret this loss function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-25T14:11:47.947Z
Answers.LastActivityDate: 2021-12-10T09:32:19.137Z
Answers.LastEditDate: 2021-12-10T09:32:19.137Z
Answers.UserDisplayName: nbro
Body: <p>I am reading the paper <a href="https://openaccess.thecvf.com/content_iccv_2015/papers/Son_Tracking-by-Segmentation_With_Online_ICCV_2015_paper.pdf" rel="nofollow noreferrer">Tracking-by-Segmentation With Online Gradient Boosting Decision Tree</a>. In Section 2.1, the paper says</p>
  <blockquote>
  <p>Given training examples, <span class="math-container">$\left\{\left(\mathbf{x}_{i}, y_{i}\right) \mid \mathbf{x}_{i} \in \mathbb{R}^{n}\right.$</span> and <span class="math-container">$y_{i} \in$</span> <span class="math-container">$\mathbb{R}\}_{i=1: N}, f(\cdot)$</span> is constructed in a greedy manner by selecting parameter <span class="math-container">$\theta_{j}$</span> and weight <span class="math-container">$\alpha_{j}$</span> of a weak learner iteratively to minimize an augmented loss function given by
  <span class="math-container">$$
  \mathcal{L}=\sum_{i=1}^{N} \ell\left(y_{i}, f\left(\mathbf{x}_{i}\right)\right) \equiv \sum_{i=1}^{N} \exp \left(-y_{i} f\left(\mathbf{x}_{i}\right)\right)
  $$</span>
  where an exponential loss function is adopted <span class="math-container">${ }^{1}$</span>. The greedy optimization procedure is summarized in Algorithm 1.</p>
  </blockquote>
  <p>I cannot understand the exponential loss function. In my opinion, the loss function should get the smallest value when <span class="math-container">$y_i=f(x_i)$</span>. But the loss function in the image obtains a smaller value if <span class="math-container">$(-y_i f(x_i))$</span> becomes smaller.</p>
  
Tags: <machine-learning><papers><objective-functions><decision-trees><gradient-boosting>
Title: Why is the exponential loss used in this case?
UserDisplayName: Zhang Liao
ThreadText: # Why is the exponential loss used in this case?
  
  (Asked by: username_0 on 2021-07-25)
  
  I am reading the paper [Tracking-by-Segmentation With Online Gradient Boosting Decision Tree](https://openaccess.thecvf.com/content_iccv_2015/papers/Son_Tracking-by-Segmentation_With_Online_ICCV_2015_paper.pdf). In Section 2.1, the paper says
  
  > Given training examples, $\left\{\left(\mathbf{x}\_{i}, y\_{i}\right) \mid \mathbf{x}\_{i} \in \mathbb{R}^{n}\right.$ and $y\_{i} \in$ $\mathbb{R}\}\_{i=1: N}, f(\cdot)$ is constructed in a greedy manner by selecting parameter $\theta\_{j}$ and weight $\alpha\_{j}$ of a weak learner iteratively to minimize an augmented loss function given by
  > $$
  > \mathcal{L}=\sum\_{i=1}^{N} \ell\left(y\_{i}, f\left(\mathbf{x}\_{i}\right)\right) \equiv \sum\_{i=1}^{N} \exp \left(-y\_{i} f\left(\mathbf{x}\_{i}\right)\right)
  > $$
  > where an exponential loss function is adopted ${ }^{1}$. The greedy optimization procedure is summarized in Algorithm 1.
  
  I cannot understand the exponential loss function. In my opinion, the loss function should get the smallest value when $y\_i=f(x\_i)$. But the loss function in the image obtains a smaller value if $(-y\_i f(x\_i))$ becomes smaller.
  
  **Tags:** machine-learning, papers, objective-functions, decision-trees, gradient-boosting
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-07-25. Score: 3 | Accepted answer
  
  The loss is
  
  $$\mathcal{L}=\sum\_{i=1}^{N} \ell\left(y\_{i}, f\left(\mathbf{x}\_{i}\right)\right) \equiv \sum\_{i=1}^{N} \exp \left(-y\_{i} f\left(\mathbf{x}\_{i}\right)\right),$$
  
  which can also be written as follows
  
  $$\mathcal{L} = \sum\_{i=1}^{N} e^{-y\_{i} f\left(\mathbf{x}\_{i}\right)} \tag{1}\label{1}$$
  
  The important thing to note here is the $-$ in the exponent, which allows us to write \ref{1} as follows (see [this](https://web.archive.org/web/20210507020014/https://www.mesacc.edu/%7Escotz47781/mat120/notes/exponents/review/review.html))
  
  $$\mathcal{L} = \sum\_{i=1}^{N} \frac{1}{e^{y\_{i} f\left(\mathbf{x}\_{i}\right)}} \tag{2}\label{2}$$
  
  So, the loss becomes smaller the higher $y\_{i}$ and $f\left(\mathbf{x}\_{i}\right)$ are. If, for example, $y\_{i}$ is negative and $f\left(\mathbf{x}\_{i}\right)$ positive (or vice-versa), then \ref{2} would be higher. If both are positive or negative, then the loss will be smaller, as we will be summing fractions of the form $\frac{1}{e^k} < 1$ for some positive constant $k$: the higher the $k$, the smaller the loss.
  
  So, if you use this loss, it seems that you want that
  
  1. $y\_{i}$ and $f\left(\mathbf{x}\_{i}\right)$ have the same sign (either both negative or both positive)
  2. $f\left(\mathbf{x}\_{i}\right)$ is as high as possible (note that you cannot change $y\_{i}$), which could even be much higher (in terms of magnitude) than $y\_{i}$ (not sure why they would want this)
  
  I don't know why they chose this loss function because I didn't read the paper (yet), but it seems to me that this is how you should interpret this loss function.
  
  
  ---
  
  