<p>You probably need to do better hyper parameter search. Here are a set of steps you should generally follow when trying to squeeze out as much performance as possible with neural networks:</p>
<ol>
<li>Get a very small subset of your data</li>
<li>Initialize a model that is large enough to overfit a dataset</li>
<li>While |subset| != |set|
<ol>
<li>Overfit the model on the data subset</li>
<li>Add more data</li>
</ol>
</li>
<li>Add regularization to the model that overfit the entire dataset</li>
</ol>

CC BY-SA 4.0
2024-07-01T13:46:56.357Z
2024-07-01T13:46:56.357Z
Jayson Meribe
<p>I am fairly new to TensorFlow and ML in general and am currently working on a regression neural network while learning about different parts and concepts of it. My goal is to try &amp; achieve a model accuracy with a predicted mean squared error (val_mse) in the range of 0.25 - 0.5. However, my current model has an val_mse of around 1.2, which is significantly higher than what I aim for.</p>
<p>Here is some information about my setup and data:</p>
<ul>
<li>The value I am predicting ranges from +/- 6 in existing traning data rows.</li>
<li>I have a total of 270,000 rows of a custom dataset in my dataframe. All data columns are floats or integers.</li>
<li>Each data row has 23 features (I have already removed highly-correlated ones)</li>
</ul>
<p>My question is: does the training graph indicate possible over-fitting or is the data with current features just not good enough to improve val_mse futher. Maybe a very obvious thing I am already doing wrong in my NN model?</p>
<p>I would really appreciate any tips.</p>
<p>Tensorflow code:</p>
<pre><code>X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2, random_state=25)

scaler = MinMaxScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

model = Sequential()
model.add(Dense(64, input_dim=X_train.shape[1], activation='relu'))

model.add(Dense(128, activation='relu', kernel_regularizer=regularizers.l2(0.001)))
model.add(BatchNormalization())

model.add(Dense(256, activation='relu'))
model.add(BatchNormalization())
model.add(Dropout(0.2))

model.add(Dense(128, activation='relu', kernel_regularizer=regularizers.l2(0.001)))
model.add(BatchNormalization())
model.add(Dropout(0.3))

model.add(Dense(64, activation='relu'))
model.add(BatchNormalization())

model.add(Dense(1, activation='linear'))

checkpoint_path = &quot;./cp.ckpt&quot;
early_stopping = EarlyStopping(monitor='val_loss', patience=30, restore_best_weights=True)
save_ckpt = tensorflow.keras.callbacks.ModelCheckpoint(
            filepath=checkpoint_path,
            save_weights_only=True,
            verbose=1)

print(model.summary())
model.compile(optimizer=Adam(learning_rate=0.0001), loss='huber', metrics=['mse'])
history = model.fit(X_train_scaled, y_train, epochs=150, batch_size=128, validation_data=(X_test_scaled, y_test),
                    callbacks=[early_stopping, save_ckpt])
</code></pre>
<p>Training graph after 150 epochs
<a href="https://i.sstatic.net/82hm5ZxT.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/82hm5ZxT.jpg" alt="graph after 150 epochs" /></a></p>

<training><tensorflow><regression>
Regression model training improvement
tomazj
# Regression model training improvement

(Asked by: username_0 on 2024-07-01)

I am fairly new to TensorFlow and ML in general and am currently working on a regression neural network while learning about different parts and concepts of it. My goal is to try & achieve a model accuracy with a predicted mean squared error (val_mse) in the range of 0.25 - 0.5. However, my current model has an val_mse of around 1.2, which is significantly higher than what I aim for.
Here is some information about my setup and data:

The value I am predicting ranges from +/- 6 in existing traning data rows.
I have a total of 270,000 rows of a custom dataset in my dataframe. All data columns are floats or integers.
Each data row has 23 features (I have already removed highly-correlated ones)

My question is: does the training graph indicate possible over-fitting or is the data with current features just not good enough to improve val_mse futher. Maybe a very obvious thing I am already doing wrong in my NN model?
I would really appreciate any tips.
Tensorflow code:
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2, random_state=25)

scaler = MinMaxScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

model = Sequential()
model.add(Dense(64, input_dim=X_train.shape[1], activation='relu'))

model.add(Dense(128, activation='relu', kernel_regularizer=regularizers.l2(0.001)))
model.add(BatchNormalization())

model.add(Dense(256, activation='relu'))
model.add(BatchNormalization())
model.add(Dropout(0.2))

model.add(Dense(128, activation='relu', kernel_regularizer=regularizers.l2(0.001)))
model.add(BatchNormalization())
model.add(Dropout(0.3))

model.add(Dense(64, activation='relu'))
model.add(BatchNormalization())

model.add(Dense(1, activation='linear'))

checkpoint_path = "./cp.ckpt"
early_stopping = EarlyStopping(monitor='val_loss', patience=30, restore_best_weights=True)
save_ckpt = tensorflow.keras.callbacks.ModelCheckpoint(
            filepath=checkpoint_path,
            save_weights_only=True,
            verbose=1)

print(model.summary())
model.compile(optimizer=Adam(learning_rate=0.0001), loss='huber', metrics=['mse'])
history = model.fit(X_train_scaled, y_train, epochs=150, batch_size=128, validation_data=(X_test_scaled, y_test),
                    callbacks=[early_stopping, save_ckpt])

Training graph after 150 epochs

**Tags:** training, tensorflow, regression


---

## 1 answers

### Answer by username_1 on 2024-07-01. Score: 1

You probably need to do better hyper parameter search. Here are a set of steps you should generally follow when trying to squeeze out as much performance as possible with neural networks:

1. Get a very small subset of your data
2. Initialize a model that is large enough to overfit a dataset
3. While |subset| != |set|
   1. Overfit the model on the data subset
   2. Add more data
4. Add regularization to the model that overfit the entire dataset


---

