Reason behind the layers taken for distilbert-multilingual
Hi,
I went across the training code for distilbert. I can see that the distillation process is used to train the distilbert model from bert model.

What is the reason only the **layers [0, 2, 4, 7, 9, 11]** have been taken to train distilbert model?Is there any idea about the layers behind this?

And,what is the **last two layers in distilbert layer** corresponds and are they equivalently similar to the **9 and 11** layer of the original bert model?

This was the training code i referred to:
https://github.com/huggingface/transformers/blob/master/examples/distillation/scripts/extract_distilbert.py
With distillation, the initialization is really important in order to obtain good results. You can read the [paper](https://arxiv.org/pdf/1910.01108.pdf) to have more information, look for section 3 and "Student Initialization"!

The important part is the following:

_Student initialization In addition to the previously described optimization and architectural choices,
an important element in our training procedure is to find the right initialization for the sub-network to
converge. Taking advantage of the common dimensionality between teacher and student networks,
we initialize the student from the teacher by taking one layer out of two._<|||||>Why not [0, 2, 4, 6, 8, 10]?