Question about BERT model size (transformer block number) 
# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
Hi,

Thank you for your interesting work! I have just started to learn BERT and distillation recently. I have some general questions regarding this topic.

1. I want to compare the performance of BERT with different model size (transformer block number). Is it necessary to do distillation? If I just train a BERT with 6 Layers without distillation, does the performance look bad?

2. Do you have to do pretraining every time you change the layer number of BERT? Is it possible to just remove some layers in an existing pre-trained model and finetune on tasks?

3. Why BERT has 12 blocks? Not 11 or 13 etc. ? I couldn't find any explanation.

Thanks,
ZLK
Hi ZLK, we're trying to keep Github issues on our repo strictly related to code. [Our forum](https://discuss.huggingface.co) is a much better place for general questions like yours, and the people over there will better be able to answer it! Cheers