<p>I noticed that there are many studies in recent years on how to train/update neural networks faster/quicker with equal or better performance. I find the following methods(except the chips arms race):</p>
<ol>
<li>using few-shot learning, for instance, pre-taining and etc.</li>
<li>using the <a href="https://medium.com/appanion/the-minimum-viable-data-set-5deb45524726" rel="nofollow noreferrer">minimum viable dataset</a>, for instance using <a href="https://bmsce.ac.in/conferences/CS/PID5919853.pdf" rel="nofollow noreferrer">(guided) progressive sampling</a>.</li>
<li>model compression, for instance, <a href="https://arxiv.org/pdf/2009.06732.pdf" rel="nofollow noreferrer">efficent transformers</a></li>
<li><a href="https://ai.googleblog.com/2020/05/speeding-up-neural-network-training.html" rel="nofollow noreferrer">Data echoing</a>, or simply put let the data pass multiple times in the graph(or GPU)</li>
</ol>
<p>Is there a systematic structure on this topic and how can we update or train a model faster without loss of its capacity?</p>

CC BY-SA 4.0
2021-01-16T11:27:32.170Z
use validation set maybe?
Alex
CC BY-SA 4.0
2021-01-16T11:33:39.443Z
@Alex What do you mean by validation set? I cannot get your point, sorry. Could you please flesh your comment out into an answer?
Lerner Zhang
CC BY-SA 4.0
2021-01-16T11:40:04.653Z
Use val data to check if you are overfitting (in addition to other tricks), this will prevent the degradation of the model's performance
Alex
CC BY-SA 4.0
2021-01-16T11:43:05.083Z
@Alex Yes, but let's consider the model that takes N epochs to get the best performance over the develop set, can we speed the training by other methods? Make the N smaller than that for the original model or less time? For instance by reducing the amount of training data or compressing the model size?
Lerner Zhang
CC BY-SA 4.0
2021-01-16T11:55:03.353Z
what do you mean by 'compressing the model size'?
Alex
CC BY-SA 4.0
2021-01-16T12:03:28.100Z
@Alex Yes, using techniques like knowledge distilling or pruning and etc.
Lerner Zhang
<neural-networks><machine-learning><training><efficiency>
How to train/update neural networks faster without a decrease in performance?
Lerner Zhang
# How to train/update neural networks faster without a decrease in performance?

(Asked by: username_0 on 2021-01-16)

I noticed that there are many studies in recent years on how to train/update neural networks faster/quicker with equal or better performance. I find the following methods(except the chips arms race):

1. using few-shot learning, for instance, pre-taining and etc.
2. using the [minimum viable dataset](https://medium.com/appanion/the-minimum-viable-data-set-5deb45524726), for instance using [(guided) progressive sampling](https://bmsce.ac.in/conferences/CS/PID5919853.pdf).
3. model compression, for instance, [efficent transformers](https://arxiv.org/pdf/2009.06732.pdf)
4. [Data echoing](https://ai.googleblog.com/2020/05/speeding-up-neural-network-training.html), or simply put let the data pass multiple times in the graph(or GPU)

Is there a systematic structure on this topic and how can we update or train a model faster without loss of its capacity?

**Tags:** neural-networks, machine-learning, training, efficiency

> **Comments:**
> **username_1**: use validation set maybe?  
> **username_0**: @username_1 What do you mean by validation set? I cannot get your point, sorry. Could you please flesh your comment out into an answer?  
> **username_1**: Use val data to check if you are overfitting (in addition to other tricks), this will prevent the degradation of the model's performance  
> **username_0**: @username_1 Yes, but let's consider the model that takes N epochs to get the best performance over the develop set, can we speed the training by other methods? Make the N smaller than that for the original model or less time? For instance by reducing the amount of training data or compressing the model size?  
> **username_1**: what do you mean by 'compressing the model size'?  
> **username_0**: @username_1 Yes, using techniques like knowledge distilling or pruning and etc.  

---

## 0 answers

