Answers.Body: <p>In general the only way to deal with this is by quantifying these labeling mistakes in the output of the model, since the model will learn for them. And in many cases these are not really mistakes, but they encode the ambiguity of the task, particularly in interpreting facial emotions.</p>
  <p>For this you need to use a model that can estimate Aleatoric uncertainty (this is data uncertainty). For classification you can use the cross-entropy loss combined with the sampling softmax function, which replaces a standard Dense layer with one that computes a Gaussian distribution for the softmax logits (so it predicts mean logit and variance logits).</p>
  <p>This method is described in the paper &quot;What uncertainties do we need in bayesian deep learning for computer vision?&quot; (<a href="https://proceedings.neurips.cc/paper/2017/file/2650d6089a6d640c5e85b2b88265dc2b-Paper.pdf" rel="nofollow noreferrer">link</a>), where the authors can also disentangle aleatoric from epistemic uncertainty.</p>
  <p>I have run experiments exactly on the facial emotion recognition task (using the FER+ dataset), which you can find in my paper &quot;A Deeper Look into Aleatoric and Epistemic Uncertainty Disentanglement&quot; (<a href="https://openaccess.thecvf.com/content/CVPR2022W/LXCV/papers/Valdenegro-Toro_A_Deeper_Look_Into_Aleatoric_and_Epistemic_Uncertainty_Disentanglement_CVPRW_2022_paper.pdf" rel="nofollow noreferrer">link</a>). In this paper I show multiple uncertainty quantification methods and their ability to produce disentangled probabilities. If you only need aleatoric uncertainty, then only the sampling softmax is needed.</p>
  <p>I even have an implementation of this layer in my <a href="https://github.com/mvaldenegro/keras-uncertainty/blob/2def7abc77b9e986021574efb43769b23e2242d7/keras_uncertainty/layers/stochastic_layers.py#L5" rel="nofollow noreferrer">keras-uncertainty</a> library.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-26T20:30:43.053Z
Answers.Text: Quantifying uncertainty is useful, but I am not sure how it directly answers the OP's question. Could you perhaps add a paragraph about how the OP can use the uncertainty measurement to cope with incorrect ground truths in the data set.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-26T21:19:45.510Z
Answers.Text: @NeilSlater yes, just did
Answers.UserDisplayName: Dr. Snoopy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-26T16:30:48.603Z
Answers.LastActivityDate: 2022-06-26T20:54:13.767Z
Answers.LastEditDate: 2022-06-26T20:54:13.767Z
Answers.UserDisplayName: Dr. Snoopy
Body: <p>I am working on the Facial Expression Recognition Task. One of the most challenging tasks that I faced was human error in labeling the datasets (ex: let's say FER2013).</p>
  <p>Are there anyways to Handle incorrect labeling of datasets in the classification tasks? Will using clustering methods and treating it as an unsupervised Learning problem (just a thought not have a clear idea) solve this kind of issue? or are any other methods available without treating it as an unsupervised task?</p>
  
Tags: <neural-networks><deep-learning><facial-recognition><clustering><data-science>
Title: How to tackle the human error made in labeling datasets for classification tasks like facial expression recognition?
UserDisplayName: Dilip C M Dept of MCA
ThreadText: # How to tackle the human error made in labeling datasets for classification tasks like facial expression recognition?
  
  (Asked by: username_0 on 2022-06-26)
  
  I am working on the Facial Expression Recognition Task. One of the most challenging tasks that I faced was human error in labeling the datasets (ex: let's say FER2013).
  
  Are there anyways to Handle incorrect labeling of datasets in the classification tasks? Will using clustering methods and treating it as an unsupervised Learning problem (just a thought not have a clear idea) solve this kind of issue? or are any other methods available without treating it as an unsupervised task?
  
  **Tags:** neural-networks, deep-learning, facial-recognition, clustering, data-science
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-06-26. Score: 2 | Accepted answer
  
  In general the only way to deal with this is by quantifying these labeling mistakes in the output of the model, since the model will learn for them. And in many cases these are not really mistakes, but they encode the ambiguity of the task, particularly in interpreting facial emotions.
  
  For this you need to use a model that can estimate Aleatoric uncertainty (this is data uncertainty). For classification you can use the cross-entropy loss combined with the sampling softmax function, which replaces a standard Dense layer with one that computes a Gaussian distribution for the softmax logits (so it predicts mean logit and variance logits).
  
  This method is described in the paper "What uncertainties do we need in bayesian deep learning for computer vision?" ([link](https://proceedings.neurips.cc/paper/2017/file/2650d6089a6d640c5e85b2b88265dc2b-Paper.pdf)), where the authors can also disentangle aleatoric from epistemic uncertainty.
  
  I have run experiments exactly on the facial emotion recognition task (using the FER+ dataset), which you can find in my paper "A Deeper Look into Aleatoric and Epistemic Uncertainty Disentanglement" ([link](https://openaccess.thecvf.com/content/CVPR2022W/LXCV/papers/Valdenegro-Toro_A_Deeper_Look_Into_Aleatoric_and_Epistemic_Uncertainty_Disentanglement_CVPRW_2022_paper.pdf)). In this paper I show multiple uncertainty quantification methods and their ability to produce disentangled probabilities. If you only need aleatoric uncertainty, then only the sampling softmax is needed.
  
  I even have an implementation of this layer in my [keras-uncertainty](https://github.com/mvaldenegro/keras-uncertainty/blob/2def7abc77b9e986021574efb43769b23e2242d7/keras_uncertainty/layers/stochastic_layers.py#L5) library.
  
  > **Comments:**
  > **username_2**: Quantifying uncertainty is useful, but I am not sure how it directly answers the OP's question. Could you perhaps add a paragraph about how the OP can use the uncertainty measurement to cope with incorrect ground truths in the data set.  
  > **username_1**: @username_2 yes, just did  
  
  ---
  
  