Answers.Body: <p>I think that the issue is still in the way you generate the fake data.</p>
  <p>Try to look at your generated features as follows:</p>
  <pre><code>import seaborn as sns
  
  sns.jointplot(data=mock_data, x='x', y='y', hue='target')
  # or
  sns.jointplot(data=mock_data, x='z', y='w', hue='target')
  </code></pre>
  <p>The second plot yields:</p>
  <p><a href="https://i.sstatic.net/sQgKt.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/sQgKt.png" alt="enter image description here" /></a></p>
  <p>that basically means that if your model learns the rule <code>(w &gt; 1.1) and (z &gt; 0.6)</code> it can achieve <strong>perfect accuracy</strong> whatever test data you use. This should imply that the problem is too simple, and so overfitting is very unlikely.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-14T22:40:13.160Z
Answers.LastActivityDate: 2023-07-14T22:40:13.160Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I am working on a very basic binary classification problem. For each set of four float numbers <span class="math-container">$(x,y,z,w)$</span>, I want to check if they fall or not into one category.</p>
  <p>I have written a model with 3 dense layers (ReLU activation function) and an output layer (with sigmoid activation function). The model doesn't overfit, so I tried increasing the hyper-parameters, but still it doesn't overfit. I thought that reaching overfit was easy, as long as you increase the number of nodes. Isn't true?</p>
  <p>Initially, I thought the problem was with the data, so I have decided to generate a mock dataset, but still the model doesn't overfit. In the code below, the function <code>generate_pattern()</code> generate a valid pattern that I want to label with the integer 1. I populate a Pandas dataframe with this function, and I add some noise by inserting random generated patterns.</p>
  <p>Why is the model not overfitting? What is the best model's architecture for this kind of problems?</p>
  <p><a href="https://i.sstatic.net/HTWAB.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/HTWAB.png" alt="enter image description here" /></a></p>
  <pre><code>import pandas as pd
  import numpy as np
  from tensorflow.keras.models import Sequential
  from tensorflow.keras.layers import Dense
  from sklearn.model_selection import train_test_split
  
  def generate_pattern():
      [x,y,z,w] = np.random.rand(4)/50
      return [0.35+x,0.45+y,0.7+z,1.32+w]
  
  mock_data = pd.DataFrame(columns=['x','y','z','w','target'])
  i=0
  while i &lt; 10000:
      if np.random.randint(2) == 0:
          mock_data.loc[i] = generate_pattern() +[1]
          i+=1
      else:
          if np.random.randint(2) == 0:
              if np.random.randint(2) == 1:
                  mock_data.loc[i] = list(np.random.rand(4)) + [0]
                  i+=1
  
  df_input = mock_data[['x','y','z','w']]
  df_output = mock_data[['target']]
  X = df_input.values
  Y = df_output.astype(int).values
  X_train, X_test, y_train, y_test = train_test_split(X, Y[:,0], test_size=0.33, random_state=52)
  X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=0.33, random_state=52)
  
  
  # Create the neural network model
  model = Sequential()
  model.add(Dense(5, activation='relu', input_dim=4))
  model.add(Dense(16, activation='relu'))
  model.add(Dense(32, activation='relu'))
  model.add(Dense(1, activation='sigmoid'))
  
  # Compile the model
  model.compile(loss=['binary_crossentropy'], optimizer='rmsprop', metrics=['accuracy','mean_squared_error','binary_crossentropy'])
  
  # Train the model
  history = model.fit(X_train, y_train, epochs=150, batch_size=64, validation_data=(X_valid,y_valid))
  </code></pre>
  
Tags: <neural-networks><tensorflow><python><hyper-parameters><architecture>
Title: Patterns binary classification - model doesn't overfit
UserDisplayName: apt45
ThreadText: # Patterns binary classification - model doesn't overfit
  
  (Asked by: username_0 on 2023-07-13)
  
  I am working on a very basic binary classification problem. For each set of four float numbers $(x,y,z,w)$, I want to check if they fall or not into one category.
  
  I have written a model with 3 dense layers (ReLU activation function) and an output layer (with sigmoid activation function). The model doesn't overfit, so I tried increasing the hyper-parameters, but still it doesn't overfit. I thought that reaching overfit was easy, as long as you increase the number of nodes. Isn't true?
  
  Initially, I thought the problem was with the data, so I have decided to generate a mock dataset, but still the model doesn't overfit. In the code below, the function `generate_pattern()` generate a valid pattern that I want to label with the integer 1. I populate a Pandas dataframe with this function, and I add some noise by inserting random generated patterns.
  
  Why is the model not overfitting? What is the best model's architecture for this kind of problems?
  
  [![enter image description here](https://i.sstatic.net/HTWAB.png)](https://i.sstatic.net/HTWAB.png)
  
  ```
  import pandas as pd
  import numpy as np
  from tensorflow.keras.models import Sequential
  from tensorflow.keras.layers import Dense
  from sklearn.model_selection import train_test_split
  
  def generate_pattern():
      [x,y,z,w] = np.random.rand(4)/50
      return [0.35+x,0.45+y,0.7+z,1.32+w]
  
  mock_data = pd.DataFrame(columns=['x','y','z','w','target'])
  i=0
  while i < 10000:
      if np.random.randint(2) == 0:
          mock_data.loc[i] = generate_pattern() +[1]
          i+=1
      else:
          if np.random.randint(2) == 0:
              if np.random.randint(2) == 1:
                  mock_data.loc[i] = list(np.random.rand(4)) + [0]
                  i+=1
  
  df_input = mock_data[['x','y','z','w']]
  df_output = mock_data[['target']]
  X = df_input.values
  Y = df_output.astype(int).values
  X_train, X_test, y_train, y_test = train_test_split(X, Y[:,0], test_size=0.33, random_state=52)
  X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=0.33, random_state=52)
  
  # Create the neural network model
  model = Sequential()
  model.add(Dense(5, activation='relu', input_dim=4))
  model.add(Dense(16, activation='relu'))
  model.add(Dense(32, activation='relu'))
  model.add(Dense(1, activation='sigmoid'))
  
  # Compile the model
  model.compile(loss=['binary_crossentropy'], optimizer='rmsprop', metrics=['accuracy','mean_squared_error','binary_crossentropy'])
  
  # Train the model
  history = model.fit(X_train, y_train, epochs=150, batch_size=64, validation_data=(X_valid,y_valid))
  
  ```
  
  **Tags:** neural-networks, tensorflow, python, hyper-parameters, architecture
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-14. Score: 1 | Accepted answer
  
  I think that the issue is still in the way you generate the fake data.
  
  Try to look at your generated features as follows:
  
  ```
  import seaborn as sns
  
  sns.jointplot(data=mock_data, x='x', y='y', hue='target')
  # or
  sns.jointplot(data=mock_data, x='z', y='w', hue='target')
  
  ```
  
  The second plot yields:
  
  [![enter image description here](https://i.sstatic.net/sQgKt.png)](https://i.sstatic.net/sQgKt.png)
  
  that basically means that if your model learns the rule `(w > 1.1) and (z > 0.6)` it can achieve **perfect accuracy** whatever test data you use. This should imply that the problem is too simple, and so overfitting is very unlikely.
  
  
  ---
  
  