Answers.Body: <p>You probably need to do better hyper parameter search. Here are a set of steps you should generally follow when trying to squeeze out as much performance as possible with neural networks:</p>
  <ol>
  <li>Get a very small subset of your data</li>
  <li>Initialize a model that is large enough to overfit a dataset</li>
  <li>While |subset| != |set|
  <ol>
  <li>Overfit the model on the data subset</li>
  <li>Add more data</li>
  </ol>
  </li>
  <li>Add regularization to the model that overfit the entire dataset</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-01T13:46:56.357Z
Answers.LastActivityDate: 2024-07-01T13:46:56.357Z
Answers.UserDisplayName: Jayson Meribe
Body: <p>I am fairly new to TensorFlow and ML in general and am currently working on a regression neural network while learning about different parts and concepts of it. My goal is to try &amp; achieve a model accuracy with a predicted mean squared error (val_mse) in the range of 0.25 - 0.5. However, my current model has an val_mse of around 1.2, which is significantly higher than what I aim for.</p>
  <p>Here is some information about my setup and data:</p>
  <ul>
  <li>The value I am predicting ranges from +/- 6 in existing traning data rows.</li>
  <li>I have a total of 270,000 rows of a custom dataset in my dataframe. All data columns are floats or integers.</li>
  <li>Each data row has 23 features (I have already removed highly-correlated ones)</li>
  </ul>
  <p>My question is: does the training graph indicate possible over-fitting or is the data with current features just not good enough to improve val_mse futher. Maybe a very obvious thing I am already doing wrong in my NN model?</p>
  <p>I would really appreciate any tips.</p>
  <p>Tensorflow code:</p>
  <pre><code>X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2, random_state=25)
  
  scaler = MinMaxScaler()
  X_train_scaled = scaler.fit_transform(X_train)
  X_test_scaled = scaler.transform(X_test)
  
  model = Sequential()
  model.add(Dense(64, input_dim=X_train.shape[1], activation='relu'))
  
  model.add(Dense(128, activation='relu', kernel_regularizer=regularizers.l2(0.001)))
  model.add(BatchNormalization())
  
  model.add(Dense(256, activation='relu'))
  model.add(BatchNormalization())
  model.add(Dropout(0.2))
  
  model.add(Dense(128, activation='relu', kernel_regularizer=regularizers.l2(0.001)))
  model.add(BatchNormalization())
  model.add(Dropout(0.3))
  
  model.add(Dense(64, activation='relu'))
  model.add(BatchNormalization())
  
  model.add(Dense(1, activation='linear'))
  
  checkpoint_path = &quot;./cp.ckpt&quot;
  early_stopping = EarlyStopping(monitor='val_loss', patience=30, restore_best_weights=True)
  save_ckpt = tensorflow.keras.callbacks.ModelCheckpoint(
              filepath=checkpoint_path,
              save_weights_only=True,
              verbose=1)
  
  print(model.summary())
  model.compile(optimizer=Adam(learning_rate=0.0001), loss='huber', metrics=['mse'])
  history = model.fit(X_train_scaled, y_train, epochs=150, batch_size=128, validation_data=(X_test_scaled, y_test),
                      callbacks=[early_stopping, save_ckpt])
  </code></pre>
  <p>Training graph after 150 epochs
  <a href="https://i.sstatic.net/82hm5ZxT.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/82hm5ZxT.jpg" alt="graph after 150 epochs" /></a></p>
  
Tags: <training><tensorflow><regression>
Title: Regression model training improvement
UserDisplayName: tomazj
ThreadText: # Regression model training improvement
  
  (Asked by: username_0 on 2024-07-01)
  
  I am fairly new to TensorFlow and ML in general and am currently working on a regression neural network while learning about different parts and concepts of it. My goal is to try & achieve a model accuracy with a predicted mean squared error (val\_mse) in the range of 0.25 - 0.5. However, my current model has an val\_mse of around 1.2, which is significantly higher than what I aim for.
  
  Here is some information about my setup and data:
  
  * The value I am predicting ranges from +/- 6 in existing traning data rows.
  * I have a total of 270,000 rows of a custom dataset in my dataframe. All data columns are floats or integers.
  * Each data row has 23 features (I have already removed highly-correlated ones)
  
  My question is: does the training graph indicate possible over-fitting or is the data with current features just not good enough to improve val\_mse futher. Maybe a very obvious thing I am already doing wrong in my NN model?
  
  I would really appreciate any tips.
  
  Tensorflow code:
  
  ```
  X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2, random_state=25)
  
  scaler = MinMaxScaler()
  X_train_scaled = scaler.fit_transform(X_train)
  X_test_scaled = scaler.transform(X_test)
  
  model = Sequential()
  model.add(Dense(64, input_dim=X_train.shape[1], activation='relu'))
  
  model.add(Dense(128, activation='relu', kernel_regularizer=regularizers.l2(0.001)))
  model.add(BatchNormalization())
  
  model.add(Dense(256, activation='relu'))
  model.add(BatchNormalization())
  model.add(Dropout(0.2))
  
  model.add(Dense(128, activation='relu', kernel_regularizer=regularizers.l2(0.001)))
  model.add(BatchNormalization())
  model.add(Dropout(0.3))
  
  model.add(Dense(64, activation='relu'))
  model.add(BatchNormalization())
  
  model.add(Dense(1, activation='linear'))
  
  checkpoint_path = "./cp.ckpt"
  early_stopping = EarlyStopping(monitor='val_loss', patience=30, restore_best_weights=True)
  save_ckpt = tensorflow.keras.callbacks.ModelCheckpoint(
              filepath=checkpoint_path,
              save_weights_only=True,
              verbose=1)
  
  print(model.summary())
  model.compile(optimizer=Adam(learning_rate=0.0001), loss='huber', metrics=['mse'])
  history = model.fit(X_train_scaled, y_train, epochs=150, batch_size=128, validation_data=(X_test_scaled, y_test),
                      callbacks=[early_stopping, save_ckpt])
  
  ```
  
  Training graph after 150 epochs
  [![graph after 150 epochs](https://i.sstatic.net/82hm5ZxT.jpg)](https://i.sstatic.net/82hm5ZxT.jpg)
  
  **Tags:** training, tensorflow, regression
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-07-01. Score: 1
  
  You probably need to do better hyper parameter search. Here are a set of steps you should generally follow when trying to squeeze out as much performance as possible with neural networks:
  
  1. Get a very small subset of your data
  2. Initialize a model that is large enough to overfit a dataset
  3. While |subset| != |set|
     1. Overfit the model on the data subset
     2. Add more data
  4. Add regularization to the model that overfit the entire dataset
  
  
  ---
  
  