Loading SentenceTransformers (DistilBertModel) model using from_pretrained(...) HF function into a DPRQuestionEncoder model
## Environment info
```
- `transformers` version: 4.10.0
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.11
- PyTorch version (GPU?): 1.9.0+cu102 (True)
- Tensorflow version (GPU?): 2.6.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No
```

### Who can help
research_projects/rag: @patrickvonplaten, @lhoestq

## To reproduce

Steps to reproduce the behavior:

_./semanticsearch_model_ is a DistilBertModel created with SentenceTransformers it is structured as follow: 
![image](https://user-images.githubusercontent.com/10784227/132877566-0abb56d2-8657-43a1-a84e-cfdf98fece87.png)

```
from transformers import DPRQuestionEncoder, DPRQuestionEncoderTokenizer

model_name = './semanticsearch_model'
model = DPRQuestionEncoder.from_pretrained(model_name)
```

# Error
```
You are using a model of type distilbert to instantiate a model of type dpr. This is not supported for all configurations of models and can yield errors.

NotImplementedErrorTraceback (most recent call last)
<ipython-input-52-1f1b990b906b> in <module>
----> 1 model = DPRQuestionEncoder.from_pretrained(model_name)
      2 # https://github.com/huggingface/transformers/blob/41cd52a768a222a13da0c6aaae877a92fc6c783c/src/transformers/models/dpr/modeling_dpr.py#L520

/opt/conda/lib/python3.8/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   1358                 raise
   1359         elif from_pt:
-> 1360             model, missing_keys, unexpected_keys, mismatched_keys, error_msgs = cls._load_state_dict_into_model(
   1361                 model,
   1362                 state_dict,

/opt/conda/lib/python3.8/site-packages/transformers/modeling_utils.py in _load_state_dict_into_model(cls, model, state_dict, pretrained_model_name_or_path, ignore_mismatched_sizes, _fast_init)
   1462             )
   1463             for module in unintialized_modules:
-> 1464                 model._init_weights(module)
   1465 
   1466         # copy state_dict so _load_from_state_dict can modify it

/opt/conda/lib/python3.8/site-packages/transformers/modeling_utils.py in _init_weights(self, module)
    577         Initialize the weights. This method should be overridden by derived class.
    578         """
--> 579         raise NotImplementedError(f"Make sure `_init_weigths` is implemented for {self.__class__}")
    580 
    581     def tie_weights(self):

NotImplementedError: Make sure `_init_weigths` is implemented for <class 'transformers.models.dpr.modeling_dpr.DPRQuestionEncoder'>
```

## Another thing
It seems that _from_pretrained_ function is not able to load all the weights (i.e. it ignores the 1_Pooling, 2_Dense folders which contains the last two layers beyond the Transformer model).

## Expected behavior

Load correctly the model.

I don't fully understand here. If the model is a `DistilBertModel` - why is it loaded with `DPRQuestionEncoder` ? Shouldn't it be loaded as follows:

```python
from transformers import DistilBertModel

model_name = './semanticsearch_model'
model = DistilBertModel.from_pretrained(model_name)
```

?<|||||>Hi @patrickvonplaten, you're right. My aim was to load a DistilBERT as underlying model of DPRQuestionEncoder (rather than the standard BERT), but maybe this is not the right way.

However, by loading the model as DistilBERT (as in your code) it only loads the transformer (i.e. `pytorch_model.bin` and `config.json` in the root directory) while the pooling and dense layers added by SentenceTransformers are ignored.

The point is, is there a way to load a model created with SentenceTransformer - including pooling and dense layers - into an HuggingFace model?

<|||||>Can you try:

```
from transformers import DistilBertForMaskedLM

model_name = './semanticsearch_model'
model = DistilBertForMaskedLM.from_pretrained(model_name)
```
?
<|||||>> Can you try:
> 
> ```
> from transformers import DistilBertForMaskedLM
> 
> model_name = './semanticsearch_model'
> model = DistilBertForMaskedLM.from_pretrained(model_name)
> ```
> 
> ?

I get

```
Some weights of DistilBertForMaskedLM were not initialized from the model checkpoint at ./semanticsearch_model and are newly initialized: ['vocab_transform.weight', 'vocab_projector.bias', 'vocab_transform.bias', 'vocab_layer_norm.weight', 'vocab_projector.weight', 'vocab_layer_norm.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

Pooling and dense layers are still ignored since as final embedding dimension I get 768 instead of 512 as it should be due to the dense layer.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.