DPR usage of BertPooler
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.8.2
- Platform: Linux-5.8.0-50-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.4
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help
RAG, DPR: @patrickvonplaten, @lhoestq
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- encoder-decoder models (For example, BlenderBot, BART, Marian, Pegasus, T5, ByT5): @patrickvonplaten, @patil-suraj
- Longformer, Reformer, TransfoXL, XLNet, FNet: @patrickvonplaten
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj @patrickvonplaten 
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten
- Tokenizers: @LysandreJik
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information
DPR [initializes BertModel with a BertPooler](https://github.com/huggingface/transformers/blob/master/src/transformers/models/dpr/modeling_dpr.py#L178) module which [is not used in the end](https://github.com/huggingface/transformers/blob/master/src/transformers/models/dpr/modeling_dpr.py#L206)

Although this seems consistent with [the original implementation](https://github.com/facebookresearch/DPR/blob/main/dpr/models/hf_models.py#L164), it is confusing for the user. One would expect that the `pooled_output` will come from the BertPooler module, if it is present, and the last layer of the model. Moreover it wastes memory and compute.

## How to fix

Simply add the `add_pooling_layer=False` flag in https://github.com/huggingface/transformers/blob/master/src/transformers/models/dpr/modeling_dpr.py#L178
Some other parts of the code need also to be fixed, like https://github.com/huggingface/transformers/blob/master/src/transformers/models/dpr/modeling_dpr.py#L205
should be `sequence_output = outputs[0]`

Hi ! I think it was kept in case some version of DPR had a projection layer (but afaik none of the official versions have that and the paper doesn't mention that either).

I think it would also break the conversion script from DPR weights of the official repository to `transformers` weights.

Not sure exactly how such changes are welcomed in the library but @patrickvonplaten probably knows more?<|||||>DPR has an optional projection layer in the [original implementation](https://github.com/facebookresearch/DPR/blob/main/dpr/models/hf_models.py#L168) but it is [only applied on the sequence output](https://github.com/facebookresearch/DPR/blob/main/dpr/models/hf_models.py#L216), not on BertPooler's output.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Sorry to be so late on this. @PaulLerner - would you like to open a PR to fix it? Otherwise I can try to give it a shot<|||||>No problem, I’m not sure to be familiar enough with the library to be able to fix it by myself.
For example, at first I thought one should simply add the `add_pooling_layer=False` flag in https://github.com/huggingface/transformers/blob/master/src/transformers/models/dpr/modeling_dpr.py#L178
But actually, some other parts of the code need also to be fixed, like https://github.com/huggingface/transformers/blob/master/src/transformers/models/dpr/modeling_dpr.py#L205
should be `sequence_output = outputs[0]`

And I have no clue about the conversion script for the weights that @lhoestq is talking about.<|||||>Hey @PaulLerner, don't worry about the conversion script. I think it'll be enough to just fix everything as suggested by you in `modeling_dpr.py` - would you like to give it a try? <|||||>Ok, I’ll let you know. I’m quite busy atm.