Body: <p>I feel like I'm doing a pretty straightforward sequence of tasks and must be making a simple mistake - I simply build a sequential model, tune it, build a clone of the optimal model by extracting the hyperparameters from the tuner, and re-train this &quot;cloned&quot; model. However, the behavior seems completely unreplicable. By the last epoch of re-training the cloned model, the validation loss has completely diverged from the supposed best val loss from the tuner, being anywhere from 1-3x higher.</p>
  <p>For the record, even though in this particular example I am doing an LSTM model, in the past I also had this issue with a convolutional neural network. This is why my hunch is I must be making a mistake in my logic, so I really appreciate any insights! A snippet of code that I hope fully illustrates my approach is below. The raw data is pretty standard text data you might find from any introductory NLP exercise, i.e. tweet data.</p>
  <pre><code>def build_model(hp):
      model = Sequential()
      model.add(Embedding(input_dim=len(tokenizer.word_index) + 1, output_dim=128, input_length=max_length))
  
      # LSTM Layer 1
      model.add(Bidirectional(LSTM(units=hp.Int(&quot;lstm_units1&quot;, min_value=5, max_value=10, step=1), 
                                   dropout=hp.Float(&quot;dropout1&quot;, min_value=0.15, max_value=0.6, step=0.025), 
                                   recurrent_dropout=hp.Float(&quot;recurrent_dropout1&quot;, min_value=0.15, max_value=0.6, step=0.025),
                                   kernel_regularizer=regularizers.L1L2(l1=0.0001, l2=0.001),
                                   bias_regularizer=regularizers.L2(0.0001),
                                   return_sequences=True)))
  
      # LSTM Layer 2
      model.add(Bidirectional(LSTM(units=hp.Int(&quot;lstm_units2&quot;, min_value=5, max_value=10, step=1), 
                                   dropout=hp.Float(&quot;dropout2&quot;, min_value=0.15, max_value=0.6, step=0.025), 
                                   recurrent_dropout=hp.Float(&quot;recurrent_dropout2&quot;, min_value=0.15, max_value=0.6, step=0.025),
                                   kernel_regularizer=regularizers.L1L2(l1=0.0001, l2=0.001),
                                   bias_regularizer=regularizers.L2(0.0001))))
  
      # Neural Network/Classifier Layer
      model.add(Dense(units=hp.Int(&quot;dense_units1&quot;, min_value=64, max_value=256, step=16), kernel_regularizer=regularizers.L2(0.0005), activation='relu'))
      model.add(Dense(units=hp.Int(&quot;dense_units2&quot;, min_value=24, max_value=48, step=4), kernel_regularizer=regularizers.L2(0.0005), activation='relu'))
      model.add(Dense(units=hp.Int(&quot;dense_units3&quot;, min_value=8, max_value=16, step=2), kernel_regularizer=regularizers.L2(0.00025), activation='relu'))
      model.add(Dense(units=1, activation='sigmoid'))
  
      model.compile(optimizer=Adam(learning_rate=hp.Choice('learning_rate', [0.001, 0.0005, 0.0001, 0.00005, 0.00001])), loss='binary_crossentropy', metrics = ['accuracy'])
      return model
  
  
  tuner = kt.RandomSearch(
      build_model,
      overwrite=True,
      objective='val_loss',
      max_trials=50)
  
  
  keras.utils.set_random_seed(1)
  tuner.search(X_train, y_train, epochs=15, batch_size=32, validation_data=(X_val, y_val))
  
  
  best_model_clone = tuner.hypermodel.build(tuner.oracle.get_best_trials(1)[0].hyperparameters)
  
  
  best_model_clone.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
  
  
  tuned_log = best_model_clone.fit(X_train, y_train, epochs=15, batch_size=32, validation_data=(X_val, y_val))
  <span class="math-container">```</span>
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-29T08:56:21.667Z
Comments.Text: I don't know about your specific logic or step, but there's usually some randomness in training models. In particular, you may be sampling data from a dataset in order to create the mini-batches. If you're sampling this randomly, then you may get different behaviours. There might be other points of randomness in your procedure.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-29T20:19:43.440Z
Comments.Text: sure, @nbro, but I was hoping that the call to keras.utils.set_random_seed(1) would have accounted for that, but perhaps I'm not understanding all the sources of randomness here.
Comments.UserDisplayName: Archetupon
Tags: <convolutional-neural-networks><long-short-term-memory><keras><hyperparameter-optimization>
Title: Why can't I replicate the validation loss from a Keras tuner (LSTM)
UserDisplayName: Archetupon
ThreadText: # Why can't I replicate the validation loss from a Keras tuner (LSTM)
  
  (Asked by: username_0 on 2024-07-26)
  
  I feel like I'm doing a pretty straightforward sequence of tasks and must be making a simple mistake - I simply build a sequential model, tune it, build a clone of the optimal model by extracting the hyperparameters from the tuner, and re-train this "cloned" model. However, the behavior seems completely unreplicable. By the last epoch of re-training the cloned model, the validation loss has completely diverged from the supposed best val loss from the tuner, being anywhere from 1-3x higher.
  
  For the record, even though in this particular example I am doing an LSTM model, in the past I also had this issue with a convolutional neural network. This is why my hunch is I must be making a mistake in my logic, so I really appreciate any insights! A snippet of code that I hope fully illustrates my approach is below. The raw data is pretty standard text data you might find from any introductory NLP exercise, i.e. tweet data.
  
  ```
  def build_model(hp):
      model = Sequential()
      model.add(Embedding(input_dim=len(tokenizer.word_index) + 1, output_dim=128, input_length=max_length))
  
      # LSTM Layer 1
      model.add(Bidirectional(LSTM(units=hp.Int("lstm_units1", min_value=5, max_value=10, step=1), 
                                   dropout=hp.Float("dropout1", min_value=0.15, max_value=0.6, step=0.025), 
                                   recurrent_dropout=hp.Float("recurrent_dropout1", min_value=0.15, max_value=0.6, step=0.025),
                                   kernel_regularizer=regularizers.L1L2(l1=0.0001, l2=0.001),
                                   bias_regularizer=regularizers.L2(0.0001),
                                   return_sequences=True)))
  
      # LSTM Layer 2
      model.add(Bidirectional(LSTM(units=hp.Int("lstm_units2", min_value=5, max_value=10, step=1), 
                                   dropout=hp.Float("dropout2", min_value=0.15, max_value=0.6, step=0.025), 
                                   recurrent_dropout=hp.Float("recurrent_dropout2", min_value=0.15, max_value=0.6, step=0.025),
                                   kernel_regularizer=regularizers.L1L2(l1=0.0001, l2=0.001),
                                   bias_regularizer=regularizers.L2(0.0001))))
  
      # Neural Network/Classifier Layer
      model.add(Dense(units=hp.Int("dense_units1", min_value=64, max_value=256, step=16), kernel_regularizer=regularizers.L2(0.0005), activation='relu'))
      model.add(Dense(units=hp.Int("dense_units2", min_value=24, max_value=48, step=4), kernel_regularizer=regularizers.L2(0.0005), activation='relu'))
      model.add(Dense(units=hp.Int("dense_units3", min_value=8, max_value=16, step=2), kernel_regularizer=regularizers.L2(0.00025), activation='relu'))
      model.add(Dense(units=1, activation='sigmoid'))
  
      model.compile(optimizer=Adam(learning_rate=hp.Choice('learning_rate', [0.001, 0.0005, 0.0001, 0.00005, 0.00001])), loss='binary_crossentropy', metrics = ['accuracy'])
      return model
  
  tuner = kt.RandomSearch(
      build_model,
      overwrite=True,
      objective='val_loss',
      max_trials=50)
  
  keras.utils.set_random_seed(1)
  tuner.search(X_train, y_train, epochs=15, batch_size=32, validation_data=(X_val, y_val))
  
  best_model_clone = tuner.hypermodel.build(tuner.oracle.get_best_trials(1)[0].hyperparameters)
  
  best_model_clone.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
  
  tuned_log = best_model_clone.fit(X_train, y_train, epochs=15, batch_size=32, validation_data=(X_val, y_val))
  ```
  
  ```
  
  **Tags:** convolutional-neural-networks, long-short-term-memory, keras, hyperparameter-optimization
  
  > **Comments:**
  > **username_1**: I don't know about your specific logic or step, but there's usually some randomness in training models. In particular, you may be sampling data from a dataset in order to create the mini-batches. If you're sampling this randomly, then you may get different behaviours. There might be other points of randomness in your procedure.  
  > **username_0**: sure, @username_1, but I was hoping that the call to keras.utils.set_random_seed(1) would have accounted for that, but perhaps I'm not understanding all the sources of randomness here.  
  
  ---
  
  ## 0 answers
  
  