Hi, have you seen this https://github.com/google-research/bert 15
It describes and provides several smaller Bert models, including evaluations of their performance.
By the way, I am not an expert.


I think the results would be OK, but the 6-layer Bert would be slower to train and use than the 6-layer DistilBert


I expect it would be possible to just remove some layers and then finetune.  After all, full training starts from randomly initialized weights, so I don’t suppose your cut-down model would be actually worse than that.  On the other hand, I don’t know if it would be better.   Certainly, if you do try it, you would want to cut off the last layers, not the first ones.    If you look at deep Convolutional networks for image recognition, the first layers detect simple patterns, and the later layers build those simple patterns into more complicated ones.


I don’t know for sure why Devlin et al chose 12 or 24 layers, but I assume they tried lots of different configurations and 12 or 24 were the best compromises.   They might also have wanted to create models that were roughly as complicated (expensive to run) as some of the previous state-of-the-art models, so that they could compare like for like in their evaluations.    It is also likely that even numbers are more efficient because of the way the hardware (GPU or TPU) is configured.  Notice that all the newly released  small Bert models have even numbers of Layers.