Answers.Body: <p>Yes, you should be able to save the weights and resume the training session at a later time and on a different GPU/server. This sort of functionality should be supported by most ML libraries (e.g. TF, PyTorch, JAX).</p>
  <p>You can use k-fold cross validation if you really want to, but that will increase the training time, because you need to train and validate on differnt splits. I think nowadays people tend to use a single train/val split, especially given that most datasets for pretraining are fairly large.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-21T06:30:28.317Z
Answers.LastActivityDate: 2022-12-21T06:30:28.317Z
Answers.UserDisplayName: PeaBrane
Body: <p>i want to pretrain some model on P100 which is provided by kaggle. Pretraining on 3 A100 is about 1.5 day. I have 2 questions:</p>
  <ol>
  <li>Can I put the same seed everywhere so that the results match, train the model for 12 hours, saving all the variables and weights at the end? Then, when the 12 hour session is over, continue pretraining in another session?</li>
  <li>I'm used to using KFold, should i use it for pretraining such huge model?</li>
  </ol>
  
Tags: <training><gpu><k-fold-cv>
Title: Pretrain large model on single GPU
UserDisplayName: kali_xyyali
ThreadText: # Pretrain large model on single GPU
  
  (Asked by: username_0 on 2022-12-20)
  
  i want to pretrain some model on P100 which is provided by kaggle. Pretraining on 3 A100 is about 1.5 day. I have 2 questions:
  
  1. Can I put the same seed everywhere so that the results match, train the model for 12 hours, saving all the variables and weights at the end? Then, when the 12 hour session is over, continue pretraining in another session?
  2. I'm used to using KFold, should i use it for pretraining such huge model?
  
  **Tags:** training, gpu, k-fold-cv
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-12-21. Score: 0 | Accepted answer
  
  Yes, you should be able to save the weights and resume the training session at a later time and on a different GPU/server. This sort of functionality should be supported by most ML libraries (e.g. TF, PyTorch, JAX).
  
  You can use k-fold cross validation if you really want to, but that will increase the training time, because you need to train and validate on differnt splits. I think nowadays people tend to use a single train/val split, especially given that most datasets for pretraining are fairly large.
  
  
  ---
  
  