Answers.Body: <h3>Common practice</h3>
  <p>Model checkpoints are often saved to the HDD to keep the GPU memory free.</p>
  <p>At every epoch, a selection metric (e.g. validation loss) is evaluated after the training stage completes, and we save the:</p>
  <ul>
  <li>Top-k best models (by model selection metric)</li>
  <li>Latest model at end of epoch</li>
  </ul>
  <p>Example training loop:</p>
  <pre><code>best_val_loss = np.inf
  
  for epoch in range(num_epochs):
      out_train = train_epoch(model)
      out_val = validate_epoch(model)
  
      save(model, &quot;last.ckpt&quot;)
  
      if out_val[&quot;loss&quot;] &lt; best_val_loss:
          save(model, &quot;best.ckpt&quot;)
          best_val_loss = out_val[&quot;loss&quot;]
  </code></pre>
  <p>It is possible that there exist models that best minimize the selection metric during an epoch. However, since the gain is likely quite minimal, it is much more practical to choose the strategy mentioned above.</p>
  <hr />
  <h3>Is there something better?</h3>
  <p>You propose some interesting strategies. It is possible that there might be a better process than the typical &quot;train/validation/test over shuffled epoch with batches&quot;, but it needs to be shown that such a process leads to noticeable improvements. The convenience and comprehensibility of using a common approach should not be underestimated.</p>
  <blockquote>
  <p>You might also want to go back and restart from a &quot;better&quot; set of weights in the hopes of getting a better trajectory.</p>
  </blockquote>
  <p>Restarting from a better model is almost like training using the validation set, though admittedly not as severe as using validation gradients to optimize the model directly. Typical problems to consider when using such an approach are overfitting and difficulty in escaping local minima.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-14T05:39:26.367Z
Answers.LastActivityDate: 2022-07-14T05:52:27.917Z
Answers.LastEditDate: 2022-07-14T05:52:27.917Z
Answers.UserDisplayName: Mateen Ulhaq
Answers.Body: <blockquote>
  <p>Is this done in practice?</p>
  </blockquote>
  <p>Yes, this is done normally when using (lack of) improvements to validation metrics as a stop criterion, and many libraries support it as standard. Depending on the library, you may find you need to add a little code to keep a copy of the best-so-far weights, but some will do it automatically by default, or based on setting params on the <code>train</code> or <code>fit</code> function.</p>
  <p>For example, <a href="https://keras.io/api/callbacks/early_stopping/" rel="noreferrer">Keras' EarlyStopping class</a> has a <code>restore_best_weights</code> parameter. Using this class in your main fit function, and setting the param to <code>true</code> will do what you want automatically with no other code required.</p>
  <blockquote>
  <p>If not what are arguments against it?</p>
  </blockquote>
  <p>Over-fitting to the validation set is a possible concern, as running the validation checks 100s of times to decide the &quot;best&quot; model may lead to some maximisation bias, and make decisions between other hyperparameters than the number of epochs less reliable.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-13T17:58:50.587Z
Answers.Text: Interesting. Still kinda makes me wonder whether doing it more often than epoch-wise can be beneficial or is rather detrimental. But I suppose probably no one has checked this or maybe the possible benefit is too marginal to matter - I don't know.
Answers.UserDisplayName: Stefan Perko
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-13T18:08:28.770Z
Answers.Text: You would have to try it and see. I think the concerns about checking too often would apply more heavily (as well as cost of evaluating) if the validation score was tested every mini-batch (or after some number of mini-batches). And yes, maybe not worth it for expected marginal gains where changes to architecture or regularisation can usually make larger differences.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-13T17:26:09.353Z
Answers.LastActivityDate: 2022-07-13T17:53:02.933Z
Answers.LastEditDate: 2022-07-13T17:53:02.933Z
Answers.UserDisplayName: Neil Slater
Body: <p>It is my understanding that when training a Deep NN in Tensorflow/PyTorch/... we only keep the current state of the network in memory, except perhaps when we manually decide to save the current weights to the HDD/SSD.</p>
  <p>Now, naively speaking it may seem reasonable to not only remember the current state (i.e. the current values of the trained weights), i.e. &quot;where we are&quot;, but also the &quot;best&quot; weights &quot;so far&quot; by some metric, such as the validation error. Immediately, this approach doubles our memory requirements, especially if we want to keep everything in the GPU memory.</p>
  <p>Is this done in practice? If not what are arguments against it?</p>
  <hr />
  <p>Now there are certainly details that make this question more tricky. If you saved only each epoch it's probably fine to just save your weights to the HDD without significant loss in speed. If you remember the best weights for each <em>step</em> of SGD than you would have to compute the validation error over and over, which is costly. Or you use perhaps an estimate of the training error based on its gradient, which however could be leading to overfitting if you are not careful. You could also mix and match, or you could compute the validation error only on a subset of the data, or only remember the best every <span class="math-container">$k$</span> steps, etc. etc. You might also want to go back and restart from a &quot;better&quot; set of weights in the hopes of getting a better trajectory.</p>
  <p>I also don't know whether in practice at the of training people typically, deliberately, go back to a previous &quot;checkpoint&quot; they might have saved somewhere.</p>
  
Tags: <neural-networks><deep-learning><reference-request><training>
Title: Deep Learning with Best-so-far instead of Where-you-are
UserDisplayName: Stefan Perko
ThreadText: # Deep Learning with Best-so-far instead of Where-you-are
  
  (Asked by: username_0 on 2022-07-13)
  
  It is my understanding that when training a Deep NN in Tensorflow/PyTorch/... we only keep the current state of the network in memory, except perhaps when we manually decide to save the current weights to the HDD/SSD.
  
  Now, naively speaking it may seem reasonable to not only remember the current state (i.e. the current values of the trained weights), i.e. "where we are", but also the "best" weights "so far" by some metric, such as the validation error. Immediately, this approach doubles our memory requirements, especially if we want to keep everything in the GPU memory.
  
  Is this done in practice? If not what are arguments against it?
  
  ---
  
  Now there are certainly details that make this question more tricky. If you saved only each epoch it's probably fine to just save your weights to the HDD without significant loss in speed. If you remember the best weights for each *step* of SGD than you would have to compute the validation error over and over, which is costly. Or you use perhaps an estimate of the training error based on its gradient, which however could be leading to overfitting if you are not careful. You could also mix and match, or you could compute the validation error only on a subset of the data, or only remember the best every $k$ steps, etc. etc. You might also want to go back and restart from a "better" set of weights in the hopes of getting a better trajectory.
  
  I also don't know whether in practice at the of training people typically, deliberately, go back to a previous "checkpoint" they might have saved somewhere.
  
  **Tags:** neural-networks, deep-learning, reference-request, training
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-07-13. Score: 8 | Accepted answer
  
  > Is this done in practice?
  
  Yes, this is done normally when using (lack of) improvements to validation metrics as a stop criterion, and many libraries support it as standard. Depending on the library, you may find you need to add a little code to keep a copy of the best-so-far weights, but some will do it automatically by default, or based on setting params on the `train` or `fit` function.
  
  For example, [Keras' EarlyStopping class](https://keras.io/api/callbacks/early_stopping/) has a `restore_best_weights` parameter. Using this class in your main fit function, and setting the param to `true` will do what you want automatically with no other code required.
  
  > If not what are arguments against it?
  
  Over-fitting to the validation set is a possible concern, as running the validation checks 100s of times to decide the "best" model may lead to some maximisation bias, and make decisions between other hyperparameters than the number of epochs less reliable.
  
  > **Comments:**
  > **username_0**: Interesting. Still kinda makes me wonder whether doing it more often than epoch-wise can be beneficial or is rather detrimental. But I suppose probably no one has checked this or maybe the possible benefit is too marginal to matter - I don't know.  
  > **username_2**: You would have to try it and see. I think the concerns about checking too often would apply more heavily (as well as cost of evaluating) if the validation score was tested every mini-batch (or after some number of mini-batches). And yes, maybe not worth it for expected marginal gains where changes to architecture or regularisation can usually make larger differences.  
  
  ---
  
  ### Answer by username_1 on 2022-07-14. Score: 2
  
  ### Common practice
  
  Model checkpoints are often saved to the HDD to keep the GPU memory free.
  
  At every epoch, a selection metric (e.g. validation loss) is evaluated after the training stage completes, and we save the:
  
  * Top-k best models (by model selection metric)
  * Latest model at end of epoch
  
  Example training loop:
  
  ```
  best_val_loss = np.inf
  
  for epoch in range(num_epochs):
      out_train = train_epoch(model)
      out_val = validate_epoch(model)
  
      save(model, "last.ckpt")
  
      if out_val["loss"] < best_val_loss:
          save(model, "best.ckpt")
          best_val_loss = out_val["loss"]
  
  ```
  
  It is possible that there exist models that best minimize the selection metric during an epoch. However, since the gain is likely quite minimal, it is much more practical to choose the strategy mentioned above.
  
  ---
  
  ### Is there something better?
  
  You propose some interesting strategies. It is possible that there might be a better process than the typical "train/validation/test over shuffled epoch with batches", but it needs to be shown that such a process leads to noticeable improvements. The convenience and comprehensibility of using a common approach should not be underestimated.
  
  > You might also want to go back and restart from a "better" set of weights in the hopes of getting a better trajectory.
  
  Restarting from a better model is almost like training using the validation set, though admittedly not as severe as using validation gradients to optimize the model directly. Typical problems to consider when using such an approach are overfitting and difficulty in escaping local minima.
  
  
  ---
  
  