Answers.Body: <p>In my opinion, the problem you pose is best described as an <a href="https://ieeexplore.ieee.org/document/7161338" rel="nofollow noreferrer">ordinal classification</a> problem, rather than a <a href="https://link.springer.com/article/10.1007/s10618-010-0175-9" rel="nofollow noreferrer">hierarchical classification</a> problem.  There are a number of approaches (besides ordinal loss functions) to address this problem discussed in the linked review article.</p>
  <p><strong>What loss functions should you use?</strong>
  There are a number of order-aware loss functions that have been described, such as mean absolute deviation (MAD), mean squared error (MSE), and ordinal crossentropy loss.  Which one works best for your dataset must be empirically determined.</p>
  <p>Sparse categorical crossentropy loss will not by itself take order into consideration.  However, as I mentioned, a version of crossentropy loss has been described, which does take order into account (see <a href="https://www.sciencedirect.com/science/article/abs/pii/S1361841521002516" rel="nofollow noreferrer">article</a>).</p>
  <p>Don't forget to re-code your target variable such so that ordinal methods can be applied (1, 2, 3, ...).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-21T02:23:29.470Z
Answers.LastActivityDate: 2022-10-21T02:23:29.470Z
Answers.UserDisplayName: Snehal Patel
Body: <p>I am working on a classification problem into progressive classes. In other words, there is some hierarchy of categories in such a way, that A &lt; B &lt; C, e.g. low, medium, high, very high. What loss function and activation function for the output layer should I use to take advantage of the class hierarchy, so that true A and predicted C is penalized more than true A and predicted B?</p>
  
  <p>My ideas are:</p>
  
  <p>1) To assign some value to each category, use one output unit with the sigmoid activation and RMS loss function. Then to assign each class to an interval, e.g. 0-033 - class A, 0.33-0.66 class B 0.66-1 - class C. It seem to do the trick, but can favor the extreme categories over the middle ones.</p>
  
  <p>2) Use K softmax output units, integer labels instead of one-hot encoded and the sparse categorical crossentropy loss function. In this case I am not sure how exactly sparse categorical crossentropy works and if it really takes into account the hierarchy.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-14T22:17:43.867Z
Comments.Text: Welcome to SE:AI!
Comments.UserDisplayName: DukeZhou
Tags: <neural-networks><deep-learning><classification><objective-functions><activation-functions>
Title: Single label classification into hierarchical categories using a neural network
UserDisplayName: Karol Borkowski
ThreadText: # Single label classification into hierarchical categories using a neural network
  
  (Asked by: username_0 on 2020-04-14)
  
  I am working on a classification problem into progressive classes. In other words, there is some hierarchy of categories in such a way, that A < B < C, e.g. low, medium, high, very high. What loss function and activation function for the output layer should I use to take advantage of the class hierarchy, so that true A and predicted C is penalized more than true A and predicted B?
  
  My ideas are:
  
  1) To assign some value to each category, use one output unit with the sigmoid activation and RMS loss function. Then to assign each class to an interval, e.g. 0-033 - class A, 0.33-0.66 class B 0.66-1 - class C. It seem to do the trick, but can favor the extreme categories over the middle ones.
  
  2) Use K softmax output units, integer labels instead of one-hot encoded and the sparse categorical crossentropy loss function. In this case I am not sure how exactly sparse categorical crossentropy works and if it really takes into account the hierarchy.
  
  **Tags:** neural-networks, deep-learning, classification, objective-functions, activation-functions
  
  > **Comments:**
  > **username_1**: Welcome to SE:AI!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-10-21. Score: 1 | Accepted answer
  
  In my opinion, the problem you pose is best described as an [ordinal classification](https://ieeexplore.ieee.org/document/7161338) problem, rather than a [hierarchical classification](https://link.springer.com/article/10.1007/s10618-010-0175-9) problem. There are a number of approaches (besides ordinal loss functions) to address this problem discussed in the linked review article.
  
  **What loss functions should you use?**
  There are a number of order-aware loss functions that have been described, such as mean absolute deviation (MAD), mean squared error (MSE), and ordinal crossentropy loss. Which one works best for your dataset must be empirically determined.
  
  Sparse categorical crossentropy loss will not by itself take order into consideration. However, as I mentioned, a version of crossentropy loss has been described, which does take order into account (see [article](https://www.sciencedirect.com/science/article/abs/pii/S1361841521002516)).
  
  Don't forget to re-code your target variable such so that ordinal methods can be applied (1, 2, 3, ...).
  
  
  ---
  
  