Answers.Body: <p>The most likely explanation is that the neural network model proposed has a much higher <em>model capacity</em> than the logistic regression model.</p>
  <p>In fact, the neural network used in the code has 246,343 trainable parameters; the full Iris data set has only 150 samples and only four features – so the model is <strong>much</strong> more complex than the training data. A neural network model with fewer neurons or layers will likely generalise much better. A logistic regression model is much simpler and has far fewer parameters, so it is in some sense forced to &quot;learn&quot;
  better.</p>
  <p>The neural network may simply overfit to the training data as the capacity of the model is large enough; an analogous idea is trying to fit a line to data, then trying to fit a degree 100 polynomial. While the polynomial may fit the training
  data better, it is less likely to generalise well.</p>
  <p>The hyperparameters used in the training of the network might also not be optimal; I haven't experimented, but neural network training can be a little sensitive to the choices made during training.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-08T22:33:56.357Z
Answers.Text: Thanks for your reply. I did try fewer neurons and layers but none of them got better performance.
Answers.UserDisplayName: JJJohn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-08T14:28:44.253Z
Answers.LastActivityDate: 2021-08-08T14:28:44.253Z
Answers.UserDisplayName: htl
Answers.Body: <p>Yes, it absolutely makes sense as the Iris dataset is linearly separable (in the sense that linear decision boundaries are near optimal).  This can just about be seen in the scatter plots:</p>
  <p><a href="https://i.sstatic.net/su21f.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/su21f.png" alt="enter image description here" /></a></p>
  <p>Particularly look at petal width versus petal length, two linear decision boundaries for that pair of variables gives a very low error rate already.  This shouldn't be surprising as the dataset was introduced in a paper by Fisher describing his linear discriminant method (about 1936?).</p>
  <p>As @htl (+1) rightly points out, a large neural network has way too much capacity for this problem and will likely overfit.  However you will probably find that a small neural network fares no better than a linear classifier for this problem.</p>
  <p>There is a lot of excitement about deep neural networks over the last few years, but they are no better than most existing classifiers for these sorts of benchmark dataset.  They are best (IMHO) for problems with very large datasets, or where convolutional layers are useful.</p>
  <p>The <a href="http://www.faqs.org/faqs/ai-faq/neural-nets/part1/preamble.html" rel="nofollow noreferrer">comp.ai.neural-nets</a> (an old Usenet news group that we used before the WWW - I trained my first neural net back in 1990) FAQ is well worth a read for anyone wanting to experiment with neural networks.  It contains a lot of folk wisdom that is good to know.  From <a href="http://www.faqs.org/faqs/ai-faq/neural-nets/part3/" rel="nofollow noreferrer">part 3</a>:</p>
  <blockquote>
  <h1>Subject: How many hidden layers should I use?</h1>
  <p>You may not need any hidden layers at all. Linear and generalized
  linear models are useful in a wide variety of applications (McCullagh
  and Nelder 1989). And even if the function you want to learn is mildly
  nonlinear, you may get better generalization with a simple linear
  model than with a complicated nonlinear model if there is too little
  data or too much noise to estimate the nonlinearities accurately.</p>
  </blockquote>
  <p>This is excellent advice.</p>
  <p>BTW I this case, you might want to try and diagnose the problem by training your neural network on a two-dimensional version of the problem, so that you can plot the decision boundary of the model and see what it is doing.  Sepal Width -v- Sepal Length might be a good choice.  Experimenting with low dimensional problems that can be directly visualised is often very informative.  I Often use Brian Ripley's synthetic dataset in my work because you can easily generate as much data as you could want.</p>
  <p><a href="https://i.sstatic.net/aPdqx.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/aPdqx.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-12T08:13:21.690Z
Answers.LastActivityDate: 2021-08-12T08:34:17.263Z
Answers.LastEditDate: 2021-08-12T08:34:17.263Z
Answers.UserDisplayName: Dikran Marsupial
Body: <p>Per <a href="https://codereview.stackexchange.com/q/265767/">a review post</a>, a simple Logistic Regression model on the <a href="https://archive.ics.uci.edu/ml/datasets/iris" rel="nofollow noreferrer">Iris data set</a> gets about 97% test accuracy on iris dataset whereas a <a href="https://codereview.stackexchange.com/q/265771/">neural network</a> gets just 94%. The neural network model used in Keras is</p>
  <pre><code>model = tf.keras.Sequential([
      tf.keras.layers.Dense(500, input_dim=4, activation='relu'),
      tf.keras.layers.Dense(256, activation='relu'),
      tf.keras.layers.Dense(256, activation='relu'),
      tf.keras.layers.Dense(128, activation='relu'),
      tf.keras.layers.Dropout(0.2),
      tf.keras.layers.Dense(128, activation='relu'),
      tf.keras.layers.Dense(3, activation='softmax')
  ])
  model.compile(loss='categorical_crossentropy',
                optimizer='adam',
                metrics=['accuracy'])
  </code></pre>
  <p>The model is fit for 30 epochs using a batch size of 20.</p>
  <p>Note that I <strong>did try</strong> fewer neurons and layers but none of them got better performance.</p>
  <p>Does this make sense? Can any other neural network get a higher test accuracy than a logistic regression model?</p>
  
Tags: <neural-networks><logistic-regression><iris-dataset>
Title: Does it make sense for a logistic regression model to perform better than a neural network on the Iris data set?
UserDisplayName: JJJohn
ThreadText: # Does it make sense for a logistic regression model to perform better than a neural network on the Iris data set?
  
  (Asked by: username_0 on 2021-08-07)
  
  Per a review post, a simple Logistic Regression model on the Iris data set gets about 97% test accuracy on iris dataset whereas a neural network gets just 94%. The neural network model used in Keras is
  model = tf.keras.Sequential([
      tf.keras.layers.Dense(500, input_dim=4, activation='relu'),
      tf.keras.layers.Dense(256, activation='relu'),
      tf.keras.layers.Dense(256, activation='relu'),
      tf.keras.layers.Dense(128, activation='relu'),
      tf.keras.layers.Dropout(0.2),
      tf.keras.layers.Dense(128, activation='relu'),
      tf.keras.layers.Dense(3, activation='softmax')
  ])
  model.compile(loss='categorical_crossentropy',
                optimizer='adam',
                metrics=['accuracy'])
  
  The model is fit for 30 epochs using a batch size of 20.
  Note that I did try fewer neurons and layers but none of them got better performance.
  Does this make sense? Can any other neural network get a higher test accuracy than a logistic regression model?
  
  **Tags:** neural-networks, logistic-regression, iris-dataset
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-08-08. Score: 4
  
  The most likely explanation is that the neural network model proposed has a much higher *model capacity* than the logistic regression model.
  
  In fact, the neural network used in the code has 246,343 trainable parameters; the full Iris data set has only 150 samples and only four features – so the model is **much** more complex than the training data. A neural network model with fewer neurons or layers will likely generalise much better. A logistic regression model is much simpler and has far fewer parameters, so it is in some sense forced to "learn"
  better.
  
  The neural network may simply overfit to the training data as the capacity of the model is large enough; an analogous idea is trying to fit a line to data, then trying to fit a degree 100 polynomial. While the polynomial may fit the training
  data better, it is less likely to generalise well.
  
  The hyperparameters used in the training of the network might also not be optimal; I haven't experimented, but neural network training can be a little sensitive to the choices made during training.
  
  > **Comments:**
  > **username_0**: Thanks for your reply. I did try fewer neurons and layers but none of them got better performance.  
  
  ---
  
  ### Answer by username_2 on 2021-08-12. Score: 1
  
  Yes, it absolutely makes sense as the Iris dataset is linearly separable (in the sense that linear decision boundaries are near optimal). This can just about be seen in the scatter plots:
  
  [![enter image description here](https://i.sstatic.net/su21f.png)](https://i.sstatic.net/su21f.png)
  
  Particularly look at petal width versus petal length, two linear decision boundaries for that pair of variables gives a very low error rate already. This shouldn't be surprising as the dataset was introduced in a paper by Fisher describing his linear discriminant method (about 1936?).
  
  As @htl (+1) rightly points out, a large neural network has way too much capacity for this problem and will likely overfit. However you will probably find that a small neural network fares no better than a linear classifier for this problem.
  
  There is a lot of excitement about deep neural networks over the last few years, but they are no better than most existing classifiers for these sorts of benchmark dataset. They are best (IMHO) for problems with very large datasets, or where convolutional layers are useful.
  
  The [comp.ai.neural-nets](http://www.faqs.org/faqs/ai-faq/neural-nets/part1/preamble.html) (an old Usenet news group that we used before the WWW - I trained my first neural net back in 1990) FAQ is well worth a read for anyone wanting to experiment with neural networks. It contains a lot of folk wisdom that is good to know. From [part 3](http://www.faqs.org/faqs/ai-faq/neural-nets/part3/):
  
  > # Subject: How many hidden layers should I use?
  >
  > You may not need any hidden layers at all. Linear and generalized
  > linear models are useful in a wide variety of applications (McCullagh
  > and Nelder 1989). And even if the function you want to learn is mildly
  > nonlinear, you may get better generalization with a simple linear
  > model than with a complicated nonlinear model if there is too little
  > data or too much noise to estimate the nonlinearities accurately.
  
  This is excellent advice.
  
  BTW I this case, you might want to try and diagnose the problem by training your neural network on a two-dimensional version of the problem, so that you can plot the decision boundary of the model and see what it is doing. Sepal Width -v- Sepal Length might be a good choice. Experimenting with low dimensional problems that can be directly visualised is often very informative. I Often use Brian Ripley's synthetic dataset in my work because you can easily generate as much data as you could want.
  
  [![enter image description here](https://i.sstatic.net/aPdqx.png)](https://i.sstatic.net/aPdqx.png)
  
  
  ---
  
  