Answers.Body: <p>Advice from Neil, yes, output targeting class labels is still classification.</p>
  
  <p>Output range is in contiguous range:</p>
  
  <ul>
  <li>This is regression</li>
  <li>For example: Linear activation, it has full numeric range of outputs.</li>
  </ul>
  
  <p>Output targeting class labels is classification:</p>
  
  <ul>
  <li>Single output neuron with sigmoid-like functions. This will classify to 2 classes, although Y data can be normalised to classify more classes.</li>
  <li>Multiple output neurons (probabilities of classes) with sigmoid-like functions (<code>softmax</code> is mainly used). This will classify to 2 or more classes.</li>
  </ul>
  
  <p>Multiple combinations of loss functions and optimisers can make the single-neuron output layer work, with different configs for them. Note that learning rates of different optimisers are different, some take 1e-1, some need 1e-3 for good training.</p>
  
  <p>For example, this combination should work:</p>
  
  <pre><code>loss_fn   = tf.losses.LogCosh();
  optimizer = tf.optimizers.RMSprop(1e-3);
  </code></pre>
  
  <p>From my trying out, these other combinations also work for single output neuron with my data (Adam, Adamax, Nadam, RMSprop work when learning_rate=1e-3 instead of 1e-1):</p>
  
  <pre><code>                         Adadelta  Adagrad  Adam  Adamax  Ftrl  Nadam  RMSprop  SGD
  BinaryCrossentropy       Yes       Yes      --    --      --    --     --       Yes
  CategoricalCrossentropy  Yes       --       --    --      --    --     --       --
  CategoricalHinge         --        --       --    --      --    --     --       --
  CosineSimilarity         --        --       --    --      --    --     --       --
  Hinge                    Yes       Yes      --    --      --    --     --       Yes
  Huber                    Yes       Yes      --    --      --    --     --       Yes
  LogCosh                  Yes       Yes      --    --      --    --     --       Yes
  Poisson                  Yes       Yes      --    --      --    --     --       Yes
  SquaredHinge             Yes       Yes      --    --      --    --     --       Yes
  
  KLD: lambda a,b: KLD(a,b)
  MAE,MAPE,MSE,MSLE: lambda a,b: Mxxx(a,b)
  The above lambdas are direct functions, not classes like SGD, Adadelta, etc.
  SparseCategorialCrossentropy: Seems not working with single output neuron.
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-06T06:34:00.437Z
Answers.LastActivityDate: 2019-10-06T08:39:33.427Z
Answers.LastEditDate: 2019-10-06T08:39:33.427Z
Answers.UserDisplayName: Dan D.
Body: <p>Here's a simple image classifier implemented in TensorFlow Keras (right click to open in new tab): <a href="https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/quickstart/advanced.ipynb" rel="nofollow noreferrer">https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/quickstart/advanced.ipynb</a></p>
  
  <p>I altered it a bit to fit with my <strong>2-class dataset</strong>. And the output layer is:</p>
  
  <pre><code>Dense(2, activation=tf.nn.softmax);
  </code></pre>
  
  <p>The loss function and optimiser are still the same as in the example in the link above.</p>
  
  <pre><code>loss_fn   = tf.losses.SparseCategoricalCrossentropy();
  optimizer = tf.optimizers.Adam();
  </code></pre>
  
  <p>I wish to turn it into a classifier with single output neuron as I have only 2 classes in dataset, and sigmoid does the 2 classes good. Tried some combinations of output activation functions + loss functions + optimisers, but the network doesn't work any more (ie. it doesn't converge).</p>
  
  <p>For example, this doesn't work:</p>
  
  <pre><code>//output layer
  Dense(1, activation=tf.sigmoid);
  
  //loss and optim
  loss_fn   = tf.losses.mse;
  optimizer = tf.optimizers.Adagrad(1e-1);
  </code></pre>
  
  <p>Which combination of <strong>output activation + loss + optimiser</strong> should work for the single-output-neuron model? <strong>And generically, which loss functions and optimisers should pair well?</strong></p>
  
Tags: <classification><tensorflow><keras><objective-functions><regression>
Title: TF Keras: How to turn this probability-based classifier into single-output-neuron label-based classifier
UserDisplayName: Dan D.
ThreadText: # TF Keras: How to turn this probability-based classifier into single-output-neuron label-based classifier
  
  (Asked by: username_0 on 2019-10-04)
  
  Here's a simple image classifier implemented in TensorFlow Keras (right click to open in new tab): <https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/quickstart/advanced.ipynb>
  
  I altered it a bit to fit with my **2-class dataset**. And the output layer is:
  
  ```
  Dense(2, activation=tf.nn.softmax);
  
  ```
  
  The loss function and optimiser are still the same as in the example in the link above.
  
  ```
  loss_fn   = tf.losses.SparseCategoricalCrossentropy();
  optimizer = tf.optimizers.Adam();
  
  ```
  
  I wish to turn it into a classifier with single output neuron as I have only 2 classes in dataset, and sigmoid does the 2 classes good. Tried some combinations of output activation functions + loss functions + optimisers, but the network doesn't work any more (ie. it doesn't converge).
  
  For example, this doesn't work:
  
  ```
  //output layer
  Dense(1, activation=tf.sigmoid);
  
  //loss and optim
  loss_fn   = tf.losses.mse;
  optimizer = tf.optimizers.Adagrad(1e-1);
  
  ```
  
  Which combination of **output activation + loss + optimiser** should work for the single-output-neuron model? **And generically, which loss functions and optimisers should pair well?**
  
  **Tags:** classification, tensorflow, keras, objective-functions, regression
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-10-06. Score: 0 | Accepted answer
  
  Advice from Neil, yes, output targeting class labels is still classification.
  
  Output range is in contiguous range:
  
  * This is regression
  * For example: Linear activation, it has full numeric range of outputs.
  
  Output targeting class labels is classification:
  
  * Single output neuron with sigmoid-like functions. This will classify to 2 classes, although Y data can be normalised to classify more classes.
  * Multiple output neurons (probabilities of classes) with sigmoid-like functions (`softmax` is mainly used). This will classify to 2 or more classes.
  
  Multiple combinations of loss functions and optimisers can make the single-neuron output layer work, with different configs for them. Note that learning rates of different optimisers are different, some take 1e-1, some need 1e-3 for good training.
  
  For example, this combination should work:
  
  ```
  loss_fn   = tf.losses.LogCosh();
  optimizer = tf.optimizers.RMSprop(1e-3);
  
  ```
  
  From my trying out, these other combinations also work for single output neuron with my data (Adam, Adamax, Nadam, RMSprop work when learning\_rate=1e-3 instead of 1e-1):
  
  ```
                           Adadelta  Adagrad  Adam  Adamax  Ftrl  Nadam  RMSprop  SGD
  BinaryCrossentropy       Yes       Yes      --    --      --    --     --       Yes
  CategoricalCrossentropy  Yes       --       --    --      --    --     --       --
  CategoricalHinge         --        --       --    --      --    --     --       --
  CosineSimilarity         --        --       --    --      --    --     --       --
  Hinge                    Yes       Yes      --    --      --    --     --       Yes
  Huber                    Yes       Yes      --    --      --    --     --       Yes
  LogCosh                  Yes       Yes      --    --      --    --     --       Yes
  Poisson                  Yes       Yes      --    --      --    --     --       Yes
  SquaredHinge             Yes       Yes      --    --      --    --     --       Yes
  
  KLD: lambda a,b: KLD(a,b)
  MAE,MAPE,MSE,MSLE: lambda a,b: Mxxx(a,b)
  The above lambdas are direct functions, not classes like SGD, Adadelta, etc.
  SparseCategorialCrossentropy: Seems not working with single output neuron.
  
  ```
  
  
  ---
  
  