Answers.Body: <p>Yes, <span class="math-container">$E$</span> is the cross-entropy function and a direct generalization of the binary case.</p>
  <p>For the binary case, probability to belong to the class <span class="math-container">$1$</span> is given by a sigmoid function <span class="math-container">$\sigma(x)$</span> of the output <span class="math-container">$x$</span>, and the probability to belong to the class <span class="math-container">$0$</span> is <span class="math-container">$1 - \sigma(x)$</span>.</p>
  <p>Therefore the binary crossentropy will give:
  <span class="math-container">$$
  -\sum_i (l_i \log \sigma(x) + (1 - l_i)  \log (1 - \sigma(x))
  $$</span>
  Where the sum is over all samples in the dataset. Because <span class="math-container">$l_i$</span> is a binary variable one of these two terms would be zero. The gradient descent forces the model to predict true label with more confidence.</p>
  <p>For the multiclass case, now the output is a <span class="math-container">$K$</span>-vector and the softmax function forces its elements to sum to one:
  <span class="math-container">$$
  \sum_{i = 1}^{K} p_i = 0
  $$</span>
  The true label is one-hot encoded vector, with <span class="math-container">$1$</span> on the position of the true label, and <span class="math-container">$0$</span> elsewhere. The generalization of the binary case is:
  <span class="math-container">$$
  -\sum_i \sum_{j = 1}^{K}(l_{ij} \log \sigma_j(x) + (1 - l_{ij})  \log (1 - \sigma_j(x))
  $$</span>
  In this case, there will be <span class="math-container">$K$</span> non-vanishing contributions.The minization of <span class="math-container">$E$</span> forces the classifier to predict the true label more confidently, and all other (to make <span class="math-container">$\log(1 - \sigma_j(x))$</span>) as small  as possible.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-08T07:02:44.380Z
Answers.LastActivityDate: 2021-01-08T07:02:44.380Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Body: <p>In <a href="https://arxiv.org/pdf/1505.04597.pdf" rel="nofollow noreferrer">the original U-Net paper</a>, it is written</p>
  <blockquote>
  <p>The energy function is computed by a pixel-wise soft-max over the final
  feature map combined with the cross entropy loss function.</p>
  <p>...</p>
  <p><span class="math-container">$$
  E=\sum_{\mathbf{x} \in \Omega} w(\mathbf{x}) \log \left(p_{\ell(\mathbf{x})}(\mathbf{x})\right) \tag{1}\label{1}
  $$</span></p>
  </blockquote>
  <p>where <span class="math-container">$w(\mathbf{x})$</span> is a weight map (I'm not interested in that part right now), and <span class="math-container">$p_{k}(\mathbf{x})$</span> is</p>
  <p><span class="math-container">$$
  p_{k}(\mathbf{x})=\exp \left(a_{k}(\mathbf{x})\right) /\left(\sum_{k^{\prime}=1}^{K} \exp \left(a_{k^{\prime}}(\mathbf{x})\right)\right)
  $$</span></p>
  <p>The pixel-wise softmax with <span class="math-container">$a_{k}(\mathbf{x})$</span> being the activation in feature channel <span class="math-container">$k$</span> at pixel position <span class="math-container">$\mathbf{x}$</span> and <span class="math-container">$K$</span> the number of classes. Then <span class="math-container">$\ell(\mathbf{x})$</span> from <span class="math-container">$p_{\ell(\mathbf{x})}$</span> is the true label of each pixel, i.e. if the pixel at position <span class="math-container">$\mathbf{x}$</span> is part of class <span class="math-container">$1$</span>, then <span class="math-container">$p_{\ell(\mathbf{x})}$</span> is equal to <span class="math-container">$p_1(\mathbf{x})$</span>.</p>
  <p>As far as is understand <span class="math-container">$-E$</span> should be the cross-entropy function. Right? I've already done the math for the binary case (ignoring <span class="math-container">$w(\mathbf{x})$</span>) and it seemed to be equal.</p>
  
Tags: <objective-functions><papers><image-segmentation><u-net><cross-entropy>
Title: Have I understood the loss function from the original U-Net paper correctly?
UserDisplayName: Bert Gayus
ThreadText: # Have I understood the loss function from the original U-Net paper correctly?
  
  (Asked by: username_0 on 2021-01-07)
  
  In [the original U-Net paper](https://arxiv.org/pdf/1505.04597.pdf), it is written
  
  > The energy function is computed by a pixel-wise soft-max over the final
  > feature map combined with the cross entropy loss function.
  >
  > ...
  >
  > $$
  > E=\sum\_{\mathbf{x} \in \Omega} w(\mathbf{x}) \log \left(p\_{\ell(\mathbf{x})}(\mathbf{x})\right) \tag{1}\label{1}
  > $$
  
  where $w(\mathbf{x})$ is a weight map (I'm not interested in that part right now), and $p\_{k}(\mathbf{x})$ is
  
  $$
  p\_{k}(\mathbf{x})=\exp \left(a\_{k}(\mathbf{x})\right) /\left(\sum\_{k^{\prime}=1}^{K} \exp \left(a\_{k^{\prime}}(\mathbf{x})\right)\right)
  $$
  
  The pixel-wise softmax with $a\_{k}(\mathbf{x})$ being the activation in feature channel $k$ at pixel position $\mathbf{x}$ and $K$ the number of classes. Then $\ell(\mathbf{x})$ from $p\_{\ell(\mathbf{x})}$ is the true label of each pixel, i.e. if the pixel at position $\mathbf{x}$ is part of class $1$, then $p\_{\ell(\mathbf{x})}$ is equal to $p\_1(\mathbf{x})$.
  
  As far as is understand $-E$ should be the cross-entropy function. Right? I've already done the math for the binary case (ignoring $w(\mathbf{x})$) and it seemed to be equal.
  
  **Tags:** objective-functions, papers, image-segmentation, u-net, cross-entropy
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-01-08. Score: 1 | Accepted answer
  
  Yes, $E$ is the cross-entropy function and a direct generalization of the binary case.
  
  For the binary case, probability to belong to the class $1$ is given by a sigmoid function $\sigma(x)$ of the output $x$, and the probability to belong to the class $0$ is $1 - \sigma(x)$.
  
  Therefore the binary crossentropy will give:
  $$
  -\sum\_i (l\_i \log \sigma(x) + (1 - l\_i) \log (1 - \sigma(x))
  $$
  Where the sum is over all samples in the dataset. Because $l\_i$ is a binary variable one of these two terms would be zero. The gradient descent forces the model to predict true label with more confidence.
  
  For the multiclass case, now the output is a $K$-vector and the softmax function forces its elements to sum to one:
  $$
  \sum\_{i = 1}^{K} p\_i = 0
  $$
  The true label is one-hot encoded vector, with $1$ on the position of the true label, and $0$ elsewhere. The generalization of the binary case is:
  $$
  -\sum\_i \sum\_{j = 1}^{K}(l\_{ij} \log \sigma\_j(x) + (1 - l\_{ij}) \log (1 - \sigma\_j(x))
  $$
  In this case, there will be $K$ non-vanishing contributions.The minization of $E$ forces the classifier to predict the true label more confidently, and all other (to make $\log(1 - \sigma\_j(x))$) as small as possible.
  
  
  ---
  
  