Body: <p>I noticed that there are many studies in recent years on how to train/update neural networks faster/quicker with equal or better performance. I find the following methods(except the chips arms race):</p>
  <ol>
  <li>using few-shot learning, for instance, pre-taining and etc.</li>
  <li>using the <a href="https://medium.com/appanion/the-minimum-viable-data-set-5deb45524726" rel="nofollow noreferrer">minimum viable dataset</a>, for instance using <a href="https://bmsce.ac.in/conferences/CS/PID5919853.pdf" rel="nofollow noreferrer">(guided) progressive sampling</a>.</li>
  <li>model compression, for instance, <a href="https://arxiv.org/pdf/2009.06732.pdf" rel="nofollow noreferrer">efficent transformers</a></li>
  <li><a href="https://ai.googleblog.com/2020/05/speeding-up-neural-network-training.html" rel="nofollow noreferrer">Data echoing</a>, or simply put let the data pass multiple times in the graph(or GPU)</li>
  </ol>
  <p>Is there a systematic structure on this topic and how can we update or train a model faster without loss of its capacity?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-16T11:27:32.170Z
Comments.Text: use validation set maybe?
Comments.UserDisplayName: Alex
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-16T11:33:39.443Z
Comments.Text: @Alex What do you mean by validation set? I cannot get your point, sorry. Could you please flesh your comment out into an answer?
Comments.UserDisplayName: Lerner Zhang
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-16T11:40:04.653Z
Comments.Text: Use val data to check if you are overfitting (in addition to other tricks), this will prevent the degradation of the model's performance
Comments.UserDisplayName: Alex
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-16T11:43:05.083Z
Comments.Text: @Alex Yes, but let's consider the model that takes N epochs to get the best performance over the develop set, can we speed the training by other methods? Make the N smaller than that for the original model or less time? For instance by reducing the amount of training data or compressing the model size?
Comments.UserDisplayName: Lerner Zhang
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-16T11:55:03.353Z
Comments.Text: what do you mean by 'compressing the model size'?
Comments.UserDisplayName: Alex
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-16T12:03:28.100Z
Comments.Text: @Alex Yes, using techniques like knowledge distilling or pruning and etc.
Comments.UserDisplayName: Lerner Zhang
Tags: <neural-networks><machine-learning><training><efficiency>
Title: How to train/update neural networks faster without a decrease in performance?
UserDisplayName: Lerner Zhang
ThreadText: # How to train/update neural networks faster without a decrease in performance?
  
  (Asked by: username_0 on 2021-01-16)
  
  I noticed that there are many studies in recent years on how to train/update neural networks faster/quicker with equal or better performance. I find the following methods(except the chips arms race):
  
  1. using few-shot learning, for instance, pre-taining and etc.
  2. using the [minimum viable dataset](https://medium.com/appanion/the-minimum-viable-data-set-5deb45524726), for instance using [(guided) progressive sampling](https://bmsce.ac.in/conferences/CS/PID5919853.pdf).
  3. model compression, for instance, [efficent transformers](https://arxiv.org/pdf/2009.06732.pdf)
  4. [Data echoing](https://ai.googleblog.com/2020/05/speeding-up-neural-network-training.html), or simply put let the data pass multiple times in the graph(or GPU)
  
  Is there a systematic structure on this topic and how can we update or train a model faster without loss of its capacity?
  
  **Tags:** neural-networks, machine-learning, training, efficiency
  > **Comments:**
  > **username_1**: use validation set maybe?  
  > **username_0**: @Alex What do you mean by validation set? I cannot get your point, sorry. Could you please flesh your comment out into an answer?  
  > **username_1**: Use val data to check if you are overfitting (in addition to other tricks), this will prevent the degradation of the model's performance  
  > **username_0**: @Alex Yes, but let's consider the model that takes N epochs to get the best performance over the develop set, can we speed the training by other methods? Make the N smaller than that for the original model or less time? For instance by reducing the amount of training data or compressing the model size?  
  > **username_1**: what do you mean by 'compressing the model size'?  
  > **username_0**: @Alex Yes, using techniques like knowledge distilling or pruning and etc.  
  
  ---
  
  ## 0 answers
  
  