DPR retriever module
I see https://github.com/huggingface/transformers/pull/5279 that describes the DPR flow.

Just checking to see when the retriever module will be available.
Many thanks for making DPR available !
Pinging @lhoestq!<|||||>Hi @mchari 

The retriever is now part of the `nlp` library.
You can install it with

```
pip install nlp
```

and load the retriever:
```python
from nlp import load_dataset

wiki = load_dataset("wiki_dpr", with_embeddings=False, with_index=True, split="train")
```

The retriever is basically a dense index over wikipedia passages.
To query it using the DPR question encoder you can do:
```python
from transformers import DPRQuestionEncoderTokenizer, DPRQuestionEncoder

question_tokenizer = DPRQuestionEncoderTokenizer.from_pretrained('facebook/dpr-question_encoder-single-nq-base')
question_encoder = DPRQuestionEncoder.from_pretrained('facebook/dpr-question_encoder-single-nq-base')

question = "What is love ?"
question_emb = question_encoder(**question_tokenizer(question, return_tensors="pt"))[0].detach().numpy()
passages_scores, passages = wiki.get_nearest_examples("embeddings", question_emb, k=20)  # get k nearest neighbors
```

Shall we make a blog post or something to show how to use it with `transformers` @thomwolf ?

EDIT: `nlp` is now renamed to `datasets`<|||||>Hi @lhoestq ,
Which metric does the `FaissIndex` use to compute vector similarity? (i.e. how `passages_scores` values are computed?)

It uses the _inner product_ (as described in DPR paper) or something else?
Thank you<|||||>It uses inner product.
You can see the code that creates the index here  https://github.com/huggingface/nlp/blob/6655008c738cb613c522deb3bd18e35a67b2a7e5/datasets/wiki_dpr/wiki_dpr.py#L171<|||||>Thanks for the retriever functionality !  Not sure how it works if I want to use it on my own documents.
<|||||>@lhoestq , any guidance for fine tuning the retriever module on another set of documents ?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
