Answers.Body: <p>One problem that I see is that you can no longer use the cross-entropy loss function for training, or at least I am not sure how you could do it. This cost function has many advantages, one of them being that you can interpret the activation of the output neurons  as the probability of the input being that category (they all sum to one). You can read about some other advantages here: <a href="https://kobejean.github.io/machine-learning/2019/07/22/cross-entropy-loss/" rel="nofollow noreferrer">https://kobejean.github.io/machine-learning/2019/07/22/cross-entropy-loss/</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-05T00:33:53.730Z
Answers.LastActivityDate: 2023-02-05T00:33:53.730Z
Answers.UserDisplayName: Wolphram jonny
Answers.Body: <p>I believe it is because when using the binary output, it makes the neurons <em>dependent</em> on each other, but technically it should not be, because it complicates two problems:</p>
  <ul>
  <li>The first problem is <em>how would you calculate the loss function for that?</em> In your case <code>0011</code> translates to <code>3</code>, which means that the third one and fourth one depend on each other, so it is &quot;both neuron 3 and 4 must be 1 at the same time&quot;. As far as I am aware, there is no existing loss function for output-dependent binary outputs. The reason lies in the second problem below.</li>
  <li>The second problem is that <em>it simply complicates thing that can be done in a multi-class manner</em>. Even in your case, the simplest implementation is to have 16 neurons, each corresponds to a combination of four binary numbers. For example, if neuron 0 is activated, it is <code>0000</code>, and if it is neuron 1, it is <code>0001</code>, and so on. This implementation can use the standard cross-entropy loss, which we know it currently works very well.</li>
  </ul>
  <p>The ImageNet dataset has 1000 classes, but the <a href="https://paperswithcode.com/sota/image-classification-on-imagenet" rel="nofollow noreferrer">state-of-the-art</a> is already 91% over 100k validation images. Since the 1000 neuron outputs work very well, I don't think they even bother experimenting with 10 neurons (for <span class="math-container">$2^{10}$</span> case).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-05T03:33:06.927Z
Answers.LastActivityDate: 2023-02-05T03:33:06.927Z
Answers.UserDisplayName: Minh-Long Luu
Answers.Body: <p>Of course you can but I'd not recommend doing this way.</p>
  <p>First - it is not part of ML because it is straight logic and should not be learned, so I don't think backpropagation or other algorithms continue performing correctly that way. That way I believe your model could be fitted to avoid middle numbers and respond with only lowest or biggest. And that is the question to you if you really need that.</p>
  <p>Second - I suppose it is better to keep the model not for super specific task but for common tasks and be adaptive for cases if you want to gain more details (statistics for example).</p>
  <p>The model with output of 10 neurons will allow you to know how sure it is in its answer - that way you may want to behave the other way or not do action at all.</p>
  <p>The other reason is in clearness - better to keep parts of the system splitted, that way in case of problems you'll find the root of the problem faster.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-04T21:43:15.420Z
Answers.LastActivityDate: 2023-02-04T21:43:15.420Z
Answers.UserDisplayName: barbariania
Body: <p>Hello world of ANN usually uses MNIST hand written digit data.  For classes there are <code>10</code>, therefore it takes <code>10 neurons</code> in the output layer, each class is <code>0 to 9</code> handwritten digit images.</p>
  <p>If in the end there is only one active neuron in the output layer, why does not use only <code>4 neurons</code> in the output layer where each neuron represents a binary digit, so that <code>16 classes</code> will be more than enough (10 classes).</p>
  <p>For example, if the neuron values ​​after the activation function in the output layer are successive like this</p>
  <pre><code> 0.1 0.2 0.8 0.9
  </code></pre>
  <p>Then it can be rounded to:</p>
  <pre><code> 0 0 1 1
  </code></pre>
  <p>Or instead of being rounded up manually, Why does not use the binary step activation function on the hidden layer before the output layer.</p>
  <p>So the prediction result is <code>3</code> because <code>0011</code> if converted to decimal is <code>3</code>.</p>
  <p>By using 4 neurons, less computational load should be used than using 10 neurons for each class.</p>
  <p>So can I use only 4 neurons only in output layer to classify 10 handwritten digit (10 classes).</p>
  <p>Below picture is just sweet picture that represent 10 neurons for every class in output layer:
  <a href="https://i.sstatic.net/QrTFj.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/QrTFj.jpg" alt="enter image description here" /></a></p>
  
Tags: <neural-networks><mnist><handwritten-characters>
Title: Can I use 4 neurons for output layer to classify hand written digit?
UserDisplayName: Muhammad Ikhwan Perwira
ThreadText: # Can I use 4 neurons for output layer to classify hand written digit?
  
  (Asked by: username_0 on 2023-02-04)
  
  Hello world of ANN usually uses MNIST hand written digit data. For classes there are `10`, therefore it takes `10 neurons` in the output layer, each class is `0 to 9` handwritten digit images.
  
  If in the end there is only one active neuron in the output layer, why does not use only `4 neurons` in the output layer where each neuron represents a binary digit, so that `16 classes` will be more than enough (10 classes).
  
  For example, if the neuron values ​​after the activation function in the output layer are successive like this
  
  ```
   0.1 0.2 0.8 0.9
  
  ```
  
  Then it can be rounded to:
  
  ```
   0 0 1 1
  
  ```
  
  Or instead of being rounded up manually, Why does not use the binary step activation function on the hidden layer before the output layer.
  
  So the prediction result is `3` because `0011` if converted to decimal is `3`.
  
  By using 4 neurons, less computational load should be used than using 10 neurons for each class.
  
  So can I use only 4 neurons only in output layer to classify 10 handwritten digit (10 classes).
  
  Below picture is just sweet picture that represent 10 neurons for every class in output layer:
  [![enter image description here](https://i.sstatic.net/QrTFj.jpg)](https://i.sstatic.net/QrTFj.jpg)
  
  **Tags:** neural-networks, mnist, handwritten-characters
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2023-02-05. Score: 2
  
  One problem that I see is that you can no longer use the cross-entropy loss function for training, or at least I am not sure how you could do it. This cost function has many advantages, one of them being that you can interpret the activation of the output neurons as the probability of the input being that category (they all sum to one). You can read about some other advantages here: <https://kobejean.github.io/machine-learning/2019/07/22/cross-entropy-loss/>
  
  
  ---
  
  ### Answer by username_2 on 2023-02-05. Score: 1
  
  I believe it is because when using the binary output, it makes the neurons *dependent* on each other, but technically it should not be, because it complicates two problems:
  
  * The first problem is *how would you calculate the loss function for that?* In your case `0011` translates to `3`, which means that the third one and fourth one depend on each other, so it is "both neuron 3 and 4 must be 1 at the same time". As far as I am aware, there is no existing loss function for output-dependent binary outputs. The reason lies in the second problem below.
  * The second problem is that *it simply complicates thing that can be done in a multi-class manner*. Even in your case, the simplest implementation is to have 16 neurons, each corresponds to a combination of four binary numbers. For example, if neuron 0 is activated, it is `0000`, and if it is neuron 1, it is `0001`, and so on. This implementation can use the standard cross-entropy loss, which we know it currently works very well.
  
  The ImageNet dataset has 1000 classes, but the [state-of-the-art](https://paperswithcode.com/sota/image-classification-on-imagenet) is already 91% over 100k validation images. Since the 1000 neuron outputs work very well, I don't think they even bother experimenting with 10 neurons (for $2^{10}$ case).
  
  
  ---
  
  ### Answer by username_3 on 2023-02-04. Score: 1
  
  Of course you can but I'd not recommend doing this way.
  
  First - it is not part of ML because it is straight logic and should not be learned, so I don't think backpropagation or other algorithms continue performing correctly that way. That way I believe your model could be fitted to avoid middle numbers and respond with only lowest or biggest. And that is the question to you if you really need that.
  
  Second - I suppose it is better to keep the model not for super specific task but for common tasks and be adaptive for cases if you want to gain more details (statistics for example).
  
  The model with output of 10 neurons will allow you to know how sure it is in its answer - that way you may want to behave the other way or not do action at all.
  
  The other reason is in clearness - better to keep parts of the system splitted, that way in case of problems you'll find the root of the problem faster.
  
  
  ---
  
  