Why you need DistilBertModel class?
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

You have `DistilBertModel`, `DistilBertForSequenceClassification`, etc. in `modeling_distilbert.py`. Why you need these classes? How about using `BertModel`, `BertForSequenceClassification`, etc.?

I found the weight names are different (e.g., `transformer.layer.0.attention.q_lin.weight`and `bert.encoder.layer.0.attention.self.query.weight`), but I think it's better to use the same weight names.

(A conversion script from distilbert weights to normal bert weights is useful to use `BertModel`.)

Thanks.
This is a smaller model compared to the original, and is thus better suitable for usage on embedded devices / devices without large gpu's. Their blog posts explains this: https://medium.com/huggingface/distilbert-8cf3380435b5.<|||||>Hello @tomohideshibata,
True, we could use the same code base for BERT and DistilBERT.
For now, I prefer keeping them separate mainly for clarity since the architectures are slightly different:
- No token_type_embeddings in DistilBERT
- No Sequence Pooler in DistilBERT
Handling these two in BertModel would unnecessarily (slightly) complexify the code and I'd like to keep it clean.
Another caveat: I use torch's `nn.LayerNorm` in DistilBERT while BERT uses a custom `BertLayerNorm`. There might be slightly different edge cases (I have to check).
But overall, you can totally implement a single class for BERT and DistilBERT on your side. I would suggest to have a look at [this script](https://github.com/huggingface/pytorch-transformers/blob/master/examples/distillation/scripts/extract_for_distil.py) to have the mapping between the names.<|||||>@VictorSanh Thanks for your comments. I will try to make a conversion script from DistilBERT to BERT weights.