Answers.Body: <p><a href="https://stats.stackexchange.com/questions/260505/machine-learning-should-i-use-a-categorical-cross-entropy-or-binary-cross-entro">https://stats.stackexchange.com/questions/260505/machine-learning-should-i-use-a-categorical-cross-entropy-or-binary-cross-entro</a>
  Is relevant.</p>
  
  <p>based on my reading when you have a NN and do Binary crossentropy on what you might call 'Linked category data' the accuracy can tend to be better than in a Categorical crossentropy model. The binary aspect implies the categories can undergo multiple splits before deciding on the exact category when the data is categorically splittable like this in a tree like hierarchy the accuracy can tend to be better.</p>
  
  <p>Think of how difficult it would be to memorize the name of every type of clothing in someones wardrobe if each piece had its own special name. Vs if they had structurally relevant names like Upper/lower for category number one is it warn on your top half or bottom half. followed by inner or outer. It an inner or outer layer of clothing. learning such binary name/feature categories enables a more accurate model. If it was data unrelated in this way it would most likely not be as accurate. The binary model can take advantage learning such features while a muti categorical model I think assumes independence and tries to learn best the features of each group and gives out a prediction of how sure it falls in each category. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-09T23:33:15.093Z
Answers.Text: But, in that post, they say that binary cross entropy is only for 2 classes, meanwhile I have more in my classification problems
Answers.UserDisplayName: joão correia
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-09T23:55:24.803Z
Answers.Text: I think you should imagine the output layer of a binary output is it true or not for each class vs the training done on a 3> softmax where it chose the highest value and back propagates based on being wrong or right. if you use binary crossentropy on a multi class output layer the loss is applied in a binary fashion, benefiting tree like classification data. Maybe reading these articles will help you. https://medium.com/data-science-bootcamp/understand-cross-entropy-loss-in-minutes-9fb263caee9a https://gombru.github.io/2018/05/23/cross_entropy_loss/
Answers.UserDisplayName: Michael Hearn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-09T22:35:38.370Z
Answers.LastActivityDate: 2019-11-09T22:35:38.370Z
Answers.UserDisplayName: Michael Hearn
Body: <p>I was just doing a simple NN example with the fashion MNIST dataset, where I was getting 97% accuracy, when I noticed that I was using Binary cross-entropy instead of categorical cross-entropy by accident. When I switched to categorical cross-entropy, the accuracy dropped to 90%. I then got curious and tried to use binary cross-entropy instead of categorical cross-entropy in my other projects and in all of them the accuracy increased.</p>
  
  <p>Now, I know that binary cross-entropy can be used in a multi-class, multi-label classification problem, but why is working better than categorical cross-entropy in a multiclass single label problem?</p>
  
Tags: <neural-networks><classification><objective-functions><cross-entropy>
Title: Why does the binary cross-entropy work better than categorical cross-entropy in a multi-class single label problem?
UserDisplayName: joão correia
ThreadText: # Why does the binary cross-entropy work better than categorical cross-entropy in a multi-class single label problem?
  
  (Asked by: username_0 on 2019-11-09)
  
  I was just doing a simple NN example with the fashion MNIST dataset, where I was getting 97% accuracy, when I noticed that I was using Binary cross-entropy instead of categorical cross-entropy by accident. When I switched to categorical cross-entropy, the accuracy dropped to 90%. I then got curious and tried to use binary cross-entropy instead of categorical cross-entropy in my other projects and in all of them the accuracy increased.
  
  Now, I know that binary cross-entropy can be used in a multi-class, multi-label classification problem, but why is working better than categorical cross-entropy in a multiclass single label problem?
  
  **Tags:** neural-networks, classification, objective-functions, cross-entropy
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-11-09. Score: 1
  
  <https://stats.stackexchange.com/questions/260505/machine-learning-should-i-use-a-categorical-cross-entropy-or-binary-cross-entro>
  Is relevant.
  
  based on my reading when you have a NN and do Binary crossentropy on what you might call 'Linked category data' the accuracy can tend to be better than in a Categorical crossentropy model. The binary aspect implies the categories can undergo multiple splits before deciding on the exact category when the data is categorically splittable like this in a tree like hierarchy the accuracy can tend to be better.
  
  Think of how difficult it would be to memorize the name of every type of clothing in someones wardrobe if each piece had its own special name. Vs if they had structurally relevant names like Upper/lower for category number one is it warn on your top half or bottom half. followed by inner or outer. It an inner or outer layer of clothing. learning such binary name/feature categories enables a more accurate model. If it was data unrelated in this way it would most likely not be as accurate. The binary model can take advantage learning such features while a muti categorical model I think assumes independence and tries to learn best the features of each group and gives out a prediction of how sure it falls in each category.
  
  > **Comments:**
  > **username_0**: But, in that post, they say that binary cross entropy is only for 2 classes, meanwhile I have more in my classification problems  
  > **username_1**: I think you should imagine the output layer of a binary output is it true or not for each class vs the training done on a 3> softmax where it chose the highest value and back propagates based on being wrong or right. if you use binary crossentropy on a multi class output layer the loss is applied in a binary fashion, benefiting tree like classification data. Maybe reading these articles will help you. https://medium.com/data-science-bootcamp/understand-cross-entropy-loss-in-minutes-9fb263caee9a https://gombru.github.io/2018/05/23/cross_entropy_loss/  
  
  ---
  
  