Answers.Body: <p>In general, it is definitely very computationally expensive, so an exhaustive search is not performed in practice. However, there are some recent approaches for determining whether the architecture is &quot;fine&quot; without training the neural network first - by looking at the covariance matrix after forwarding the data, for example, in a recent paper <a href="https://arxiv.org/abs/2006.04647" rel="nofollow noreferrer">Neural Architecture Search without Training</a>. However, such an approach is very limited.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-24T19:11:29.807Z
Answers.LastActivityDate: 2021-01-18T15:12:32.217Z
Answers.LastEditDate: 2021-01-18T15:12:32.217Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Body: <p>When training a relatively small DL model, which takes several hours to train, I typically start with some starting points from literature and then use a trial-and-error or grid-search approach to fine-tune the values of the hyper-parameters, in order to prevent overfitting and achieve sufficient performance.</p>
  <p>However, it is not uncommon for large models to have training time measured in days or weeks [<a href="https://discovery.ucl.ac.uk/id/eprint/10045895/1/agz_unformatted_nature.pdf" rel="nofollow noreferrer">1</a>], [<a href="https://arxiv.org/pdf/1812.04948.pdf" rel="nofollow noreferrer">2</a>], [<a href="https://arxiv.org/pdf/1301.3781.pdf" rel="nofollow noreferrer">3</a>].</p>
  <p>How are hyperparameters determined in such cases?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-24T14:10:32.717Z
Comments.Text: I've seen some papers which are deriving a small dataset which has more or less the same characteristics as the large one and do the hyper-parameter selection on it.
Comments.UserDisplayName: razvanc92
Tags: <neural-networks><deep-learning><training><hyperparameter-optimization><hyper-parameters>
Title: How are training hyperparameters determined for large models?
UserDisplayName: Kao
ThreadText: # How are training hyperparameters determined for large models?
  
  (Asked by: username_0 on 2020-07-24)
  
  When training a relatively small DL model, which takes several hours to train, I typically start with some starting points from literature and then use a trial-and-error or grid-search approach to fine-tune the values of the hyper-parameters, in order to prevent overfitting and achieve sufficient performance.
  
  However, it is not uncommon for large models to have training time measured in days or weeks [[1](https://discovery.ucl.ac.uk/id/eprint/10045895/1/agz_unformatted_nature.pdf)], [[2](https://arxiv.org/pdf/1812.04948.pdf)], [[3](https://arxiv.org/pdf/1301.3781.pdf)].
  
  How are hyperparameters determined in such cases?
  
  **Tags:** neural-networks, deep-learning, training, hyperparameter-optimization, hyper-parameters
  
  > **Comments:**
  > **username_1**: I've seen some papers which are deriving a small dataset which has more or less the same characteristics as the large one and do the hyper-parameter selection on it.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-07-24. Score: 3 | Accepted answer
  
  In general, it is definitely very computationally expensive, so an exhaustive search is not performed in practice. However, there are some recent approaches for determining whether the architecture is "fine" without training the neural network first - by looking at the covariance matrix after forwarding the data, for example, in a recent paper [Neural Architecture Search without Training](https://arxiv.org/abs/2006.04647). However, such an approach is very limited.
  
  
  ---
  
  