Changing the number of hidden layers for BERT
## ❓ Questions & Help

Hello, 

when reducing the number of hidden layers for BERT, say from 12 to 3, which layers are loaded from the pretrained model, the first 3 layers or the last 3 ones? and is there a way to control this?

Thanks in advance

Hi,
The first ones are loaded and there is currently no simple way to control this.<|||||>**Is there any evidence than the first layers is the best choice when reducing the number of layers ?**

For example in your article about Distil-Bert, you chose to initialize the student by taking the even layers. Why so ?<|||||>> For example in your article about Distil-Bert, you chose to initialize the student by taking the even layers. Why so ?

It empirically produces stronger performance.
There are some other empirical evidences in [this paper](https://arxiv.org/abs/1909.11556) from Angela Fan, Edouard Grave and Armand Joulin.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>I also want to ask this 