Answers.Body: <p>I think that the issue is still in the way you generate the fake data.</p>
  <p>Try to look at your generated features as follows:</p>
  <pre><code>import seaborn as sns
  
  sns.jointplot(data=mock_data, x='x', y='y', hue='target')
  # or
  sns.jointplot(data=mock_data, x='z', y='w', hue='target')
  </code></pre>
  <p>The second plot yields:</p>
  <p><a href="https://i.sstatic.net/sQgKt.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/sQgKt.png" alt="enter image description here" /></a></p>
  <p>that basically means that if your model learns the rule <code>(w &gt; 1.1) and (z &gt; 0.6)</code> it can achieve <strong>perfect accuracy</strong> whatever test data you use. This should imply that the problem is too simple, and so overfitting is very unlikely.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-14T22:40:13.160Z
Answers.LastActivityDate: 2023-07-14T22:40:13.160Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I am working on a very basic binary classification problem. For each set of four float numbers <span class="math-container">$(x,y,z,w)$</span>, I want to check if they fall or not into one category.</p>
  <p>I have written a model with 3 dense layers (ReLU activation function) and an output layer (with sigmoid activation function). The model doesn't overfit, so I tried increasing the hyper-parameters, but still it doesn't overfit. I thought that reaching overfit was easy, as long as you increase the number of nodes. Isn't true?</p>
  <p>Initially, I thought the problem was with the data, so I have decided to generate a mock dataset, but still the model doesn't overfit. In the code below, the function <code>generate_pattern()</code> generate a valid pattern that I want to label with the integer 1. I populate a Pandas dataframe with this function, and I add some noise by inserting random generated patterns.</p>
  <p>Why is the model not overfitting? What is the best model's architecture for this kind of problems?</p>
  <p><a href="https://i.sstatic.net/HTWAB.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/HTWAB.png" alt="enter image description here" /></a></p>
  <pre><code>import pandas as pd
  import numpy as np
  from tensorflow.keras.models import Sequential
  from tensorflow.keras.layers import Dense
  from sklearn.model_selection import train_test_split
  
  def generate_pattern():
      [x,y,z,w] = np.random.rand(4)/50
      return [0.35+x,0.45+y,0.7+z,1.32+w]
  
  mock_data = pd.DataFrame(columns=['x','y','z','w','target'])
  i=0
  while i &lt; 10000:
      if np.random.randint(2) == 0:
          mock_data.loc[i] = generate_pattern() +[1]
          i+=1
      else:
          if np.random.randint(2) == 0:
              if np.random.randint(2) == 1:
                  mock_data.loc[i] = list(np.random.rand(4)) + [0]
                  i+=1
  
  df_input = mock_data[['x','y','z','w']]
  df_output = mock_data[['target']]
  X = df_input.values
  Y = df_output.astype(int).values
  X_train, X_test, y_train, y_test = train_test_split(X, Y[:,0], test_size=0.33, random_state=52)
  X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=0.33, random_state=52)
  
  
  # Create the neural network model
  model = Sequential()
  model.add(Dense(5, activation='relu', input_dim=4))
  model.add(Dense(16, activation='relu'))
  model.add(Dense(32, activation='relu'))
  model.add(Dense(1, activation='sigmoid'))
  
  # Compile the model
  model.compile(loss=['binary_crossentropy'], optimizer='rmsprop', metrics=['accuracy','mean_squared_error','binary_crossentropy'])
  
  # Train the model
  history = model.fit(X_train, y_train, epochs=150, batch_size=64, validation_data=(X_valid,y_valid))
  </code></pre>
  
Tags: <neural-networks><tensorflow><python><hyper-parameters><architecture>
Title: Patterns binary classification - model doesn't overfit
UserDisplayName: apt45
ThreadText: # Patterns binary classification - model doesn't overfit
  
  (Asked by: username_0 on 2023-07-13)
  
  I am working on a very basic binary classification problem. For each set of four float numbers $(x,y,z,w)$, I want to check if they fall or not into one category.
  I have written a model with 3 dense layers (ReLU activation function) and an output layer (with sigmoid activation function). The model doesn't overfit, so I tried increasing the hyper-parameters, but still it doesn't overfit. I thought that reaching overfit was easy, as long as you increase the number of nodes. Isn't true?
  Initially, I thought the problem was with the data, so I have decided to generate a mock dataset, but still the model doesn't overfit. In the code below, the function generate_pattern() generate a valid pattern that I want to label with the integer 1. I populate a Pandas dataframe with this function, and I add some noise by inserting random generated patterns.
  Why is the model not overfitting? What is the best model's architecture for this kind of problems?
  
  import pandas as pd
  import numpy as np
  from tensorflow.keras.models import Sequential
  from tensorflow.keras.layers import Dense
  from sklearn.model_selection import train_test_split
  
  def generate_pattern():
      [x,y,z,w] = np.random.rand(4)/50
      return [0.35+x,0.45+y,0.7+z,1.32+w]
  
  mock_data = pd.DataFrame(columns=['x','y','z','w','target'])
  i=0
  while i < 10000:
      if np.random.randint(2) == 0:
          mock_data.loc[i] = generate_pattern() +[1]
          i+=1
      else:
          if np.random.randint(2) == 0:
              if np.random.randint(2) == 1:
                  mock_data.loc[i] = list(np.random.rand(4)) + [0]
                  i+=1
  
  df_input = mock_data[['x','y','z','w']]
  df_output = mock_data[['target']]
  X = df_input.values
  Y = df_output.astype(int).values
  X_train, X_test, y_train, y_test = train_test_split(X, Y[:,0], test_size=0.33, random_state=52)
  X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=0.33, random_state=52)
  
  
  # Create the neural network model
  model = Sequential()
  model.add(Dense(5, activation='relu', input_dim=4))
  model.add(Dense(16, activation='relu'))
  model.add(Dense(32, activation='relu'))
  model.add(Dense(1, activation='sigmoid'))
  
  # Compile the model
  model.compile(loss=['binary_crossentropy'], optimizer='rmsprop', metrics=['accuracy','mean_squared_error','binary_crossentropy'])
  
  # Train the model
  history = model.fit(X_train, y_train, epochs=150, batch_size=64, validation_data=(X_valid,y_valid))
  
  **Tags:** neural-networks, tensorflow, python, hyper-parameters, architecture
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-14. Score: 1 | Accepted answer
  
  I think that the issue is still in the way you generate the fake data.
  Try to look at your generated features as follows:
  import seaborn as sns
  
  sns.jointplot(data=mock_data, x='x', y='y', hue='target')
  # or
  sns.jointplot(data=mock_data, x='z', y='w', hue='target')
  
  The second plot yields:
  
  that basically means that if your model learns the rule (w > 1.1) and (z > 0.6) it can achieve perfect accuracy whatever test data you use. This should imply that the problem is too simple, and so overfitting is very unlikely.
  
  
  ---
  
  