Compatibility between DistilBert and Bert models
## ❓ Questions & Help

I have a regular classification task for sentences in russian language. 
I used to train `BertForSequenceClassification` with pretrained Bert from [DeepPavlov](http://docs.deeppavlov.ai/en/master/features/models/bert.html) [RuBERT](http://files.deeppavlov.ai/deeppavlov_data/bert/rubert_cased_L-12_H-768_A-12_v2.tar.gz) (using PyTorch). Then I switched to `DistilBertForSequenceClassification`, but still using pretrained RuBert (because there is no pretrained DistilBert with russian language). And it worked. 

Then after [this changing](https://github.com/huggingface/transformers/pull/1203/commits/465870c33fe4ade66863ca0edfe13616f9d24da5#diff-9dc1f6db4a89dbf13c19d02a9f27093dL178) there is impossible to load `DistilBertConfig` from `BertConfig` config.json. `DistilBertConfig` uses property decorator for compatibility between `DistilBertConfig` and `BertConfig`, that's why using setattr() causes error `AttributeError: can't set attribute`.

So my question is following: Is it a bug or feature? Is it OK to load DistilBert from pretrained Bert or not? Or maybe the best way for me is to distill RuBERT by myself using your script for distillation?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
