Answers.Body: <p><strong>Classic Question</strong></p>
  
  <p>Not only can the reliability or accuracy needs be asymmetrically distributed between category boundaries, but the asymmetry might not be describable in terms of a first degree polynomial translation of the error surface.</p>
  
  <p>The question data set are images belonging to class <span class="math-container">$c \in \Big( A: \{A_1, \, ..., \, A_n\} \land  B: \{B_1, \, ..., \, B_m\} \Big)$</span>.  The convolutional network must be trained with a sample of labeled images to categorize with a reliability of <span class="math-container">$1 - \delta$</span> and accuracy of <span class="math-container">$1 - \epsilon$</span>, based on the PAC (probably approximately correct) framework.</p>
  
  <p>How can the loss function reflect the conceptualization of optimal training result that the reliability of categorization between <span class="math-container">$A$</span> and <span class="math-container">$B$</span> be of greater value than categorization between <span class="math-container">$A_i$</span> and <span class="math-container">$A_j$</span> where <span class="math-container">$i \ne j$</span>, and similarly with B.</p>
  
  <p>Criteria X would state that the value of differentiation <span class="math-container">$D$</span> is not sub-category index dependent for differentiation between super-categories.  That is, <span class="math-container">$\forall \; (i, j) \; \text{where} \; i \ne j$</span>, <span class="math-container">$D(A_i, B_i) = D(A_i, B_j)$</span>.</p>
  
  <p>Criteria Y, the categorization WITHIN a super-category is not unlike the typical symmetric case <span class="math-container">$\forall \; (i, j, k) \; \text{where} \; i \ne j \land i \ne k \land j \ne k$</span>, <span class="math-container">$D(A_i, A_j) = D(A_i, B_k)$</span>.)</p>
  
  <ol>
  <li>(a) Are there any loss functions that take this requirement into account?</li>
  </ol>
  
  <p>Let <span class="math-container">$1_p$</span> be the bit value that is 1 if and only if the label matches the su<strong>p</strong>er-category and let <span class="math-container">$1_b$</span> be the bit value that is 1 if and only if the label matches the super-category and the su<strong>b</strong>-category.  Because of Criteria X, a conical and normalized loss function could then be defined as thus.</p>
  
  <p><span class="math-container">$\epsilon = \dfrac {(1 - 1_p) + \alpha (1 - 1_b)} {1 + \alpha}$</span></p>
  
  <p>The proportional value of sub-category reliability in relation to super-category reliability <span class="math-container">$\alpha &lt; 1.0$</span>.</p>
  
  <p>Because of criteria Y, there is no meaning to concavity in this context.  Applying mean squaring to the loss simply changes the curvature and meaning of <span class="math-container">$\alpha$</span> in relation to the loss as can be seen here.</p>
  
  <p><span class="math-container">$\epsilon = \dfrac {(1 - 1_p)^2 + \alpha^2 (1 - 1_b)^2} {1 + \alpha^2} = \dfrac {(1 - 1_p) + \tau (1 - 1_b)} {1 + \tau}\text{, where }\tau = \alpha^2$</span></p>
  
  <ol>
  <li>(b) Could a weighted cross entropy work in this case?</li>
  </ol>
  
  <p>Cross entropy between the two levels of categorization would be baseless unless there were correlative effects between super classification feature and sub classification feature inherent in the target concept of the learning.</p>
  
  <ol start="2">
  <li>How this loss would change if the classes were unbalanced?</li>
  </ol>
  
  <p>Although there may be schemes for modifying the loss function to compensate for skewed distribution of categories represented in the sample, this is primarily a data input problem best solved by improving data collection or choosing training algorithms, designs, architectures, treating input, and properly controlling hyper-parameters.  The loss function represents the inverse of what is considered optimal, not how one achieves it. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-11T14:59:52.490Z
Answers.Text: can this function be minimized with stochastic gradient descent?
Answers.UserDisplayName: firion
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-11T13:33:48.357Z
Answers.LastActivityDate: 2018-10-11T13:33:48.357Z
Answers.UserDisplayName: Douglas Daseeco
Body: <p>I have a dataset of images belonging to <span class="math-container">$N$</span> classes, <span class="math-container">$A_1, A_2...A_n,B_1,B_2...B_m$</span> and I want to train a CNN to classify them. The classes can be considered as subclasses of two broader classes <span class="math-container">$A$</span> and <span class="math-container">$B$</span>, therefore the confusion between <span class="math-container">$A_i$</span> and <span class="math-container">$A_j$</span> is much less problematic than the confusion between <span class="math-container">$A_i$</span> and <span class="math-container">$B_j$</span>. Therefore I want the CNN to be trained in such a way that the difference between <span class="math-container">$A_i$</span> and <span class="math-container">$B_j$</span> is considered as more relevant. </p>
  
  <p>1) Are there any loss functions that take this requirement into account? Could a weighted cross entropy work in this case?</p>
  
  <p>2) How this loss would change if the classes were unbalanced?</p>
  
Tags: <convolutional-neural-networks><classification><objective-functions>
Title: How to define a loss function for a classifier where the confusion between some classes is more important than the confusion between others?
UserDisplayName: firion
ThreadText: # How to define a loss function for a classifier where the confusion between some classes is more important than the confusion between others?
  
  (Asked by: username_0 on 2018-10-10)
  
  I have a dataset of images belonging to $N$ classes, $A\_1, A\_2...A\_n,B\_1,B\_2...B\_m$ and I want to train a CNN to classify them. The classes can be considered as subclasses of two broader classes $A$ and $B$, therefore the confusion between $A\_i$ and $A\_j$ is much less problematic than the confusion between $A\_i$ and $B\_j$. Therefore I want the CNN to be trained in such a way that the difference between $A\_i$ and $B\_j$ is considered as more relevant.
  
  1) Are there any loss functions that take this requirement into account? Could a weighted cross entropy work in this case?
  
  2) How this loss would change if the classes were unbalanced?
  
  **Tags:** convolutional-neural-networks, classification, objective-functions
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-10-11. Score: 2
  
  **Classic Question**
  
  Not only can the reliability or accuracy needs be asymmetrically distributed between category boundaries, but the asymmetry might not be describable in terms of a first degree polynomial translation of the error surface.
  
  The question data set are images belonging to class $c \in \Big( A: \{A\_1, \, ..., \, A\_n\} \land B: \{B\_1, \, ..., \, B\_m\} \Big)$. The convolutional network must be trained with a sample of labeled images to categorize with a reliability of $1 - \delta$ and accuracy of $1 - \epsilon$, based on the PAC (probably approximately correct) framework.
  
  How can the loss function reflect the conceptualization of optimal training result that the reliability of categorization between $A$ and $B$ be of greater value than categorization between $A\_i$ and $A\_j$ where $i \ne j$, and similarly with B.
  
  Criteria X would state that the value of differentiation $D$ is not sub-category index dependent for differentiation between super-categories. That is, $\forall \; (i, j) \; \text{where} \; i \ne j$, $D(A\_i, B\_i) = D(A\_i, B\_j)$.
  
  Criteria Y, the categorization WITHIN a super-category is not unlike the typical symmetric case $\forall \; (i, j, k) \; \text{where} \; i \ne j \land i \ne k \land j \ne k$, $D(A\_i, A\_j) = D(A\_i, B\_k)$.)
  
  1. (a) Are there any loss functions that take this requirement into account?
  
  Let $1\_p$ be the bit value that is 1 if and only if the label matches the su**p**er-category and let $1\_b$ be the bit value that is 1 if and only if the label matches the super-category and the su**b**-category. Because of Criteria X, a conical and normalized loss function could then be defined as thus.
  
  $\epsilon = \dfrac {(1 - 1\_p) + \alpha (1 - 1\_b)} {1 + \alpha}$
  
  The proportional value of sub-category reliability in relation to super-category reliability $\alpha < 1.0$.
  
  Because of criteria Y, there is no meaning to concavity in this context. Applying mean squaring to the loss simply changes the curvature and meaning of $\alpha$ in relation to the loss as can be seen here.
  
  $\epsilon = \dfrac {(1 - 1\_p)^2 + \alpha^2 (1 - 1\_b)^2} {1 + \alpha^2} = \dfrac {(1 - 1\_p) + \tau (1 - 1\_b)} {1 + \tau}\text{, where }\tau = \alpha^2$
  
  1. (b) Could a weighted cross entropy work in this case?
  
  Cross entropy between the two levels of categorization would be baseless unless there were correlative effects between super classification feature and sub classification feature inherent in the target concept of the learning.
  
  2. How this loss would change if the classes were unbalanced?
  
  Although there may be schemes for modifying the loss function to compensate for skewed distribution of categories represented in the sample, this is primarily a data input problem best solved by improving data collection or choosing training algorithms, designs, architectures, treating input, and properly controlling hyper-parameters. The loss function represents the inverse of what is considered optimal, not how one achieves it.
  
  > **Comments:**
  > **username_0**: can this function be minimized with stochastic gradient descent?  
  
  ---
  
  