<p>I feel like I'm doing a pretty straightforward sequence of tasks and must be making a simple mistake - I simply build a sequential model, tune it, build a clone of the optimal model by extracting the hyperparameters from the tuner, and re-train this &quot;cloned&quot; model. However, the behavior seems completely unreplicable. By the last epoch of re-training the cloned model, the validation loss has completely diverged from the supposed best val loss from the tuner, being anywhere from 1-3x higher.</p>
<p>For the record, even though in this particular example I am doing an LSTM model, in the past I also had this issue with a convolutional neural network. This is why my hunch is I must be making a mistake in my logic, so I really appreciate any insights! A snippet of code that I hope fully illustrates my approach is below. The raw data is pretty standard text data you might find from any introductory NLP exercise, i.e. tweet data.</p>
<pre><code>def build_model(hp):
    model = Sequential()
    model.add(Embedding(input_dim=len(tokenizer.word_index) + 1, output_dim=128, input_length=max_length))

    # LSTM Layer 1
    model.add(Bidirectional(LSTM(units=hp.Int(&quot;lstm_units1&quot;, min_value=5, max_value=10, step=1), 
                                 dropout=hp.Float(&quot;dropout1&quot;, min_value=0.15, max_value=0.6, step=0.025), 
                                 recurrent_dropout=hp.Float(&quot;recurrent_dropout1&quot;, min_value=0.15, max_value=0.6, step=0.025),
                                 kernel_regularizer=regularizers.L1L2(l1=0.0001, l2=0.001),
                                 bias_regularizer=regularizers.L2(0.0001),
                                 return_sequences=True)))

    # LSTM Layer 2
    model.add(Bidirectional(LSTM(units=hp.Int(&quot;lstm_units2&quot;, min_value=5, max_value=10, step=1), 
                                 dropout=hp.Float(&quot;dropout2&quot;, min_value=0.15, max_value=0.6, step=0.025), 
                                 recurrent_dropout=hp.Float(&quot;recurrent_dropout2&quot;, min_value=0.15, max_value=0.6, step=0.025),
                                 kernel_regularizer=regularizers.L1L2(l1=0.0001, l2=0.001),
                                 bias_regularizer=regularizers.L2(0.0001))))

    # Neural Network/Classifier Layer
    model.add(Dense(units=hp.Int(&quot;dense_units1&quot;, min_value=64, max_value=256, step=16), kernel_regularizer=regularizers.L2(0.0005), activation='relu'))
    model.add(Dense(units=hp.Int(&quot;dense_units2&quot;, min_value=24, max_value=48, step=4), kernel_regularizer=regularizers.L2(0.0005), activation='relu'))
    model.add(Dense(units=hp.Int(&quot;dense_units3&quot;, min_value=8, max_value=16, step=2), kernel_regularizer=regularizers.L2(0.00025), activation='relu'))
    model.add(Dense(units=1, activation='sigmoid'))

    model.compile(optimizer=Adam(learning_rate=hp.Choice('learning_rate', [0.001, 0.0005, 0.0001, 0.00005, 0.00001])), loss='binary_crossentropy', metrics = ['accuracy'])
    return model


tuner = kt.RandomSearch(
    build_model,
    overwrite=True,
    objective='val_loss',
    max_trials=50)


keras.utils.set_random_seed(1)
tuner.search(X_train, y_train, epochs=15, batch_size=32, validation_data=(X_val, y_val))


best_model_clone = tuner.hypermodel.build(tuner.oracle.get_best_trials(1)[0].hyperparameters)


best_model_clone.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])


tuned_log = best_model_clone.fit(X_train, y_train, epochs=15, batch_size=32, validation_data=(X_val, y_val))
<span class="math-container">```</span>
</code></pre>

CC BY-SA 4.0
2024-07-29T08:56:21.667Z
I don't know about your specific logic or step, but there's usually some randomness in training models. In particular, you may be sampling data from a dataset in order to create the mini-batches. If you're sampling this randomly, then you may get different behaviours. There might be other points of randomness in your procedure.
nbro
CC BY-SA 4.0
2024-07-29T20:19:43.440Z
sure, @nbro, but I was hoping that the call to keras.utils.set_random_seed(1) would have accounted for that, but perhaps I'm not understanding all the sources of randomness here.
Archetupon
<convolutional-neural-networks><long-short-term-memory><keras><hyperparameter-optimization>
Why can't I replicate the validation loss from a Keras tuner (LSTM)
Archetupon
# Why can't I replicate the validation loss from a Keras tuner (LSTM)

(Asked by: username_0 on 2024-07-26)

I feel like I'm doing a pretty straightforward sequence of tasks and must be making a simple mistake - I simply build a sequential model, tune it, build a clone of the optimal model by extracting the hyperparameters from the tuner, and re-train this "cloned" model. However, the behavior seems completely unreplicable. By the last epoch of re-training the cloned model, the validation loss has completely diverged from the supposed best val loss from the tuner, being anywhere from 1-3x higher.
For the record, even though in this particular example I am doing an LSTM model, in the past I also had this issue with a convolutional neural network. This is why my hunch is I must be making a mistake in my logic, so I really appreciate any insights! A snippet of code that I hope fully illustrates my approach is below. The raw data is pretty standard text data you might find from any introductory NLP exercise, i.e. tweet data.
def build_model(hp):
    model = Sequential()
    model.add(Embedding(input_dim=len(tokenizer.word_index) + 1, output_dim=128, input_length=max_length))

    # LSTM Layer 1
    model.add(Bidirectional(LSTM(units=hp.Int("lstm_units1", min_value=5, max_value=10, step=1), 
                                 dropout=hp.Float("dropout1", min_value=0.15, max_value=0.6, step=0.025), 
                                 recurrent_dropout=hp.Float("recurrent_dropout1", min_value=0.15, max_value=0.6, step=0.025),
                                 kernel_regularizer=regularizers.L1L2(l1=0.0001, l2=0.001),
                                 bias_regularizer=regularizers.L2(0.0001),
                                 return_sequences=True)))

    # LSTM Layer 2
    model.add(Bidirectional(LSTM(units=hp.Int("lstm_units2", min_value=5, max_value=10, step=1), 
                                 dropout=hp.Float("dropout2", min_value=0.15, max_value=0.6, step=0.025), 
                                 recurrent_dropout=hp.Float("recurrent_dropout2", min_value=0.15, max_value=0.6, step=0.025),
                                 kernel_regularizer=regularizers.L1L2(l1=0.0001, l2=0.001),
                                 bias_regularizer=regularizers.L2(0.0001))))

    # Neural Network/Classifier Layer
    model.add(Dense(units=hp.Int("dense_units1", min_value=64, max_value=256, step=16), kernel_regularizer=regularizers.L2(0.0005), activation='relu'))
    model.add(Dense(units=hp.Int("dense_units2", min_value=24, max_value=48, step=4), kernel_regularizer=regularizers.L2(0.0005), activation='relu'))
    model.add(Dense(units=hp.Int("dense_units3", min_value=8, max_value=16, step=2), kernel_regularizer=regularizers.L2(0.00025), activation='relu'))
    model.add(Dense(units=1, activation='sigmoid'))

    model.compile(optimizer=Adam(learning_rate=hp.Choice('learning_rate', [0.001, 0.0005, 0.0001, 0.00005, 0.00001])), loss='binary_crossentropy', metrics = ['accuracy'])
    return model


tuner = kt.RandomSearch(
    build_model,
    overwrite=True,
    objective='val_loss',
    max_trials=50)


keras.utils.set_random_seed(1)
tuner.search(X_train, y_train, epochs=15, batch_size=32, validation_data=(X_val, y_val))


best_model_clone = tuner.hypermodel.build(tuner.oracle.get_best_trials(1)[0].hyperparameters)


best_model_clone.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])


tuned_log = best_model_clone.fit(X_train, y_train, epochs=15, batch_size=32, validation_data=(X_val, y_val))
```

**Tags:** convolutional-neural-networks, long-short-term-memory, keras, hyperparameter-optimization

> **Comments:**
> **username_1**: I don't know about your specific logic or step, but there's usually some randomness in training models. In particular, you may be sampling data from a dataset in order to create the mini-batches. If you're sampling this randomly, then you may get different behaviours. There might be other points of randomness in your procedure.  
> **username_0**: sure, @username_1, but I was hoping that the call to keras.utils.set_random_seed(1) would have accounted for that, but perhaps I'm not understanding all the sources of randomness here.  

---

## 0 answers

