<p>You can save weights during training by passing checkpoint callback to model.fit() method.</p>
<pre><code># Instantiate your model here
model = create_model() 

# Set model configurations here
model.compile(loss=..., optimizer=..., metrics=...) 

# Set checkpoint path
checkpoint_path = &quot;model_weights.ckpt&quot;

# Create a callback that saves the model's weights
    filepath=checkpoint_path,
    save_weights_only=True,
    monitor='val_loss',
    mode='min',
    save_best_only=True)

# Train the model with the new callback
model.fit(train_images_1, 
          train_labels_1,
          epochs=50, 
          batch_size=batch_size, 
          callbacks=[cp_callback],
          validation_data=(test_images_1, test_labels_1),
          verbose=0)
</code></pre>
<p>After finishing training 1st dataset, model weights will be saved in file called <em>model_weights.ckpt</em>. Before starting training next dataset, load the model weights as below</p>
<pre><code># Create a new model instance
model = create_model()

# Set model configurations here
model.compile(loss=..., optimizer=..., metrics=...) 

# Set checkpoint path
checkpoint_path = &quot;model_weights.ckpt&quot;

# Load the previously saved weights
model.load_weights(checkpoint_path)

# Create a callback that saves the model's weights
cp_callback = tf.keras.callbacks.ModelCheckpoint(
    filepath=checkpoint_path,
    save_weights_only=True,
    monitor='val_loss',
    mode='min',
    save_best_only=True)

# Train the model with the new callback
model.fit(train_images_2, 
          train_labels_2,
          epochs=50, 
          batch_size=batch_size, 
          callbacks=[cp_callback],
          validation_data=(test_images_2, test_labels_2),
          verbose=0)
</code></pre>
<p>Repeat this for all datasets.</p>

CC BY-SA 4.0
2021-05-16T04:19:43.093Z
2021-05-16T05:45:03.297Z
2021-05-16T05:45:03.297Z
spb
<p>If you reinitialize your model weights before training it on a new subset, you erase everything it learned before; is it what you want to ?
If not, saving your model after each subset and loading it before training on the next subset isn't a good practice neither because your model will see a lot of times a subset of samples and then move on the next ones without seing again these samples.</p>
<p>A better solution is to load batches on-fly. This can be done using <a href="https://stanford.edu/%7Eshervine/blog/keras-how-to-generate-data-on-the-fly" rel="nofollow noreferrer">keras generators</a>. It allows you to load a batch and do a gradient descent step without changing the model.fit function.</p>
<p>It will be a bit slower but it will work.</p>

CC BY-SA 4.0
2021-05-18T12:43:22.000Z
2021-05-18T12:43:22.000Z
bguetarni
<p>I created a binary image classification model. The dataset contains about 500K images in each class, with ratio = Train : Validation : Test = 7 : 2 : 1. Total images = 1M</p>
<p>I split my dataset into 5 parts (compute constraints)—5 training subsets, 5 validation subsets, and 1 test subset.</p>
<p>I trained and evaluated my model stage by stage. In first stage (evaluation), my model's accuracy was 65%. I re-fitted it with 2nd dataset and the accuracy was 43%. I did same process with the rest, and my accuracies were: 65%, 43%, 57%, 21%, 30%.</p>
<p>How can I train my model in staged training?</p>
<p>I want to train models with different datasets without reinitialize the weight every training process.</p>

CC BY-SA 4.0
2021-05-16T10:52:57.150Z
Hello. I don't really understand what the question/doubt is here. Can you clarify that? I understand it has to do with `fit`, but apart from that I don't really understand the question.
nbro
<deep-learning><tensorflow><training>
What is the process working on Tensorflow model.fit()?
Kue Satu
# What is the process working on Tensorflow model.fit()?

(Asked by: username_0 on 2021-05-16)

I created a binary image classification model. The dataset contains about 500K images in each class, with ratio = Train : Validation : Test = 7 : 2 : 1. Total images = 1M

I split my dataset into 5 parts (compute constraints)—5 training subsets, 5 validation subsets, and 1 test subset.

I trained and evaluated my model stage by stage. In first stage (evaluation), my model's accuracy was 65%. I re-fitted it with 2nd dataset and the accuracy was 43%. I did same process with the rest, and my accuracies were: 65%, 43%, 57%, 21%, 30%.

How can I train my model in staged training?

I want to train models with different datasets without reinitialize the weight every training process.

**Tags:** deep-learning, tensorflow, training

> **Comments:**
> **username_1**: Hello. I don't really understand what the question/doubt is here. Can you clarify that? I understand it has to do with `fit`, but apart from that I don't really understand the question.  

---

## 2 answers

### Answer by username_2 on 2021-05-16. Score: 1

You can save weights during training by passing checkpoint callback to model.fit() method.
# Instantiate your model here
model = create_model() 

# Set model configurations here
model.compile(loss=..., optimizer=..., metrics=...) 

# Set checkpoint path
checkpoint_path = "model_weights.ckpt"

# Create a callback that saves the model's weights
    filepath=checkpoint_path,
    save_weights_only=True,
    monitor='val_loss',
    mode='min',
    save_best_only=True)

# Train the model with the new callback
model.fit(train_images_1, 
          train_labels_1,
          epochs=50, 
          batch_size=batch_size, 
          callbacks=[cp_callback],
          validation_data=(test_images_1, test_labels_1),
          verbose=0)

After finishing training 1st dataset, model weights will be saved in file called model_weights.ckpt. Before starting training next dataset, load the model weights as below
# Create a new model instance
model = create_model()

# Set model configurations here
model.compile(loss=..., optimizer=..., metrics=...) 

# Set checkpoint path
checkpoint_path = "model_weights.ckpt"

# Load the previously saved weights
model.load_weights(checkpoint_path)

# Create a callback that saves the model's weights
cp_callback = tf.keras.callbacks.ModelCheckpoint(
    filepath=checkpoint_path,
    save_weights_only=True,
    monitor='val_loss',
    mode='min',
    save_best_only=True)

# Train the model with the new callback
model.fit(train_images_2, 
          train_labels_2,
          epochs=50, 
          batch_size=batch_size, 
          callbacks=[cp_callback],
          validation_data=(test_images_2, test_labels_2),
          verbose=0)

Repeat this for all datasets.


---

### Answer by username_3 on 2021-05-18. Score: 0

If you reinitialize your model weights before training it on a new subset, you erase everything it learned before; is it what you want to ?
If not, saving your model after each subset and loading it before training on the next subset isn't a good practice neither because your model will see a lot of times a subset of samples and then move on the next ones without seing again these samples.

A better solution is to load batches on-fly. This can be done using [keras generators](https://stanford.edu/%7Eshervine/blog/keras-how-to-generate-data-on-the-fly). It allows you to load a batch and do a gradient descent step without changing the model.fit function.

It will be a bit slower but it will work.


---

