Answers.Body: <p>Part of the answer is that new architectures are often variants of existing architectures. There are some rough heuristics people follow (e.g. using powers of 2 in layer sizes, changing layer sizes according to a schedule, adding normalization layers, etc.). However, even when building off an existing architecture, there can still be a lot of toying around with different configurations and hyperparameters to require copious training time.</p>
  <p>Researchers working for a company will likely have the luxury of some serious compute, e.g. GPU clusters or cloud services like AWS. This can really make a huge difference over training a model on your PC GPU. Further, many models can be trained at once. Many academic institutions also employ cluster computing on dedicated GPU clusters - for example, here at the University of Utah our department has a cluster with 8 really excellent GPUs.</p>
  <p>One last thing I will note, is that it's not always necessary to train the model to completion on the full training set to get a sense of which network configuration and hyperparameter settings are working better than others. If I have a massive dataset that will take days to train on, I may take a smaller subset of the training data and do some quick comparisons between different models to get a feel for what changes have the greatest impact on test performance. It's not completely reliable, but it can guide intuition to narrow down the number of models you want to train on the full dataset to convergence. For example, using this technique I was able to find that predictions from my model were fairly invariant to the exact layer sizes I was using and was far more sensitive to a hyperparameter in my optimization solver.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-11T07:06:48.347Z
Answers.LastActivityDate: 2020-12-11T07:06:48.347Z
Answers.UserDisplayName: adamconkey
Body: <p>After reading this topic on GitHub <a href="https://github.com/pjreddie/darknet/issues/742" rel="nofollow noreferrer">how long time it takes to train YOLOV3 on coco dataset</a> I was wondering how researchers deal with long training times while inventing new architectures.</p>
  <p>I imagine that to evaluate the model you need to train it first. How do they make tweaks in their architectures, e.g. tweaking layers, adding pooling, dropout, etc., if training can take a few days? Is it pure art and it is designed roughly or is it a more deliberate process?</p>
  <p>What are the steps of engineering new architecture using deep neural networks?</p>
  
Tags: <deep-learning><research>
Title: What is the process of inventing deep neural network models? How researchers deal with long training times?
UserDisplayName: karlosos
ThreadText: # What is the process of inventing deep neural network models? How researchers deal with long training times?
  
  (Asked by: username_0 on 2020-12-10)
  
  After reading this topic on GitHub [how long time it takes to train YOLOV3 on coco dataset](https://github.com/pjreddie/darknet/issues/742) I was wondering how researchers deal with long training times while inventing new architectures.
  
  I imagine that to evaluate the model you need to train it first. How do they make tweaks in their architectures, e.g. tweaking layers, adding pooling, dropout, etc., if training can take a few days? Is it pure art and it is designed roughly or is it a more deliberate process?
  
  What are the steps of engineering new architecture using deep neural networks?
  
  **Tags:** deep-learning, research
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-12-11. Score: 2
  
  Part of the answer is that new architectures are often variants of existing architectures. There are some rough heuristics people follow (e.g. using powers of 2 in layer sizes, changing layer sizes according to a schedule, adding normalization layers, etc.). However, even when building off an existing architecture, there can still be a lot of toying around with different configurations and hyperparameters to require copious training time.
  
  Researchers working for a company will likely have the luxury of some serious compute, e.g. GPU clusters or cloud services like AWS. This can really make a huge difference over training a model on your PC GPU. Further, many models can be trained at once. Many academic institutions also employ cluster computing on dedicated GPU clusters - for example, here at the University of Utah our department has a cluster with 8 really excellent GPUs.
  
  One last thing I will note, is that it's not always necessary to train the model to completion on the full training set to get a sense of which network configuration and hyperparameter settings are working better than others. If I have a massive dataset that will take days to train on, I may take a smaller subset of the training data and do some quick comparisons between different models to get a feel for what changes have the greatest impact on test performance. It's not completely reliable, but it can guide intuition to narrow down the number of models you want to train on the full dataset to convergence. For example, using this technique I was able to find that predictions from my model were fairly invariant to the exact layer sizes I was using and was far more sensitive to a hyperparameter in my optimization solver.
  
  
  ---
  
  