Add REALM
# What does this PR do?

This PR adds REALM.

- Original paper: https://arxiv.org/abs/2002.08909
- Code and checkpoints: https://github.com/google-research/language/tree/master/language/realm

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Closes https://github.com/huggingface/transformers/issues/3497
Hi @qqaatw,

Thank you so much for putting effort on this PR, providing pre-trained REALM models in Pytorch Transformers API.

I am wondering whether your REALM models in pytorch can reproduce Table 2 of their original [paper](https://arxiv.org/pdf/2002.08909.pdf)?

Alternatively, do you verify their tensorflow pre-train model has the same embeddings as your converted pytorch models given arbitrary input sequence?

Thanks again for the awesome work! <|||||>Hello @OctoberChang, thanks for your reply!

This is my first time trying to port a model from Tensorflow, so I may need some time to clarify the structure and behavior of the original model. Currently, the retriever part was successfully converted.

Regarding your concerns, I've verified the retriever's behavior by feeding the same inputs to TensorFlow and PyTorch models respectively, and then checking their outputs that are nearly identical. For now, I may not have enough resources to complete those ablation experiments sadly, but I think it can be reproduced as long as the PyTorch model's behavior is nearly the same as that of Tensorflow.<|||||>> Hello @OctoberChang, thanks for your reply!
> 
> This is my first time trying to port a model from Tensorflow, so I may need some time to clarify the structure and behavior of the original model. Currently, the retriever part was successfully converted.
> 
> Regarding your concerns, I've verified the retriever's behavior by feeding the same inputs to TensorFlow and PyTorch models respectively, and then checking their outputs that are nearly identical. For now, I may not have enough resources to complete those ablation experiments sadly, but I think it can be reproduced as long as the PyTorch model's behavior is nearly the same as that of Tensorflow.

Awesome! Looking forward to this PR and the pre-trained Realm models in Pytorch Transformers!
<|||||>The reason I didn't add `RealmForQuestionAnswering` is the following:

1. The fine-tuning code is placed at another project, [ORQA](https://github.com/google-research/language/tree/master/language/orqa), which has its own [paper](https://arxiv.org/abs/1906.00300).
2. The architecture of fine-tuned models is not compatible with the existing question answering architecture in `transformers`.

Therefore, I think residing the fine-tuning code to research_project folder or making it a new model would be more appropriate.

 <|||||>Tests related to REALM have passed! Some failures seem related to Flax Big Bird model.
@sgugger @LysandreJik @patrickvonplaten <|||||>@OctoberChang Do you have any suggestion on this PR? :-)<|||||>@patrickvonplaten Thank you a lot for the comments and review. I've left replies on the threads.<|||||>Hi ! Sure we can add the index in `datasets`.
Do you know what data they used exactly ? Are the texts available ? If yes, did they also share the embeddings of the documents ?

Otherwise we can just build an index from scratch using Wikipedia and the model to encode the documents<|||||>Hey @qqaatw,

Sorry to answer that late. We just had a long discussion internally with @lhoestq on how to best integrate REALM into `transformers`.
Our understanding of [ORQA](https://arxiv.org/pdf/1906.00300.pdf) and [REALM](https://arxiv.org/pdf/2002.08909.pdf) and how it relates to the integration to `transformes` is the following:

- The ORQA paper was published before REALM. In ORQA only the retriever was pretrained. REALM was published afterwards and is (subjectively) an improved pre-training method for knowlegde-augmented language models. REALM compares its methods to ORQA by evaluating the models on open-ended question answering, *i.e.*:

```
We evaluate our approach by fine-tuning the models pre-trained with REALM on the task of Opendomain Question Answering (Open-QA), one of the most
knowledge-intensive tasks in natural language processing. We evaluate on three popular Open-QA benchmarks (NATURALQUESTIONS-OPEN, WEBQUESTIONS, and
CURATEDTREC) and compare to state-of-the-art Open-QA
models, including both extremely large models that store
knowledge implicitly (such as T5) as well as previous approaches that also use a knowledge retriever to access external knowledge, but implement retrieval in a more heuristic fashion (Lee et al., 2019 - ORQA; Min et al., 2019a; Asai et al.,
2019). REALM achieves new state-of-the-art results on all
three benchmarks, significantly outperforming all previous
systems by 4-16% absolute accuracy. We also demonstrate
qualitative benefits of REALM, including interpretability
and modularity.
```
(on page 2)

=> this means that the REALM paper does not just provide a pretraining method, but also fine-tuned checkpoints for Open-QA.

- As a conclusion REALM does not necessarly rely on the code of `ORQA`. REALM provides both pre-trained checkpoints: https://github.com/google-research/language/tree/master/language/realm#pre-trained-model-checkpoints as well as fine-tuned ones: https://github.com/google-research/language/tree/master/language/realm#pre-trained-model-checkpoints which from a logical point of view are 1-to-1 related to the REALM paper (since REALM evaluated its models on Open-QA). Therefore, in our opinion, the community should be able to load all of the following checkpoints within `modeling_realm.py`:
- 
```
REALM pre-trained with CC-News as the target corpus and Wikipedia as the knowledge corpus is available at gs://realm-data/cc_news_pretrained
REALM fine-tuned to perform open-domain QA:
on WebQuestions: gs://realm-data/orqa_wq_model_from_realm
on NaturalQuestions: gs://realm-data/orqa_nq_model_from_realm
```

Therefore, I think we should add all the logic of your [codebase](https://github.com/qqaatw/pytorch-realm-orqa): `RealmSearcher` and `RealmReader` as well as `RealmForOpenQA` in `modeling_realm.py`. This has a lot of advantages also from a community's point of view:

1. REALM without an implementation for QA cannot really be used by most people as pretraining is just to expensive
2. REALM with QA would allow us to nicely demo the model.

=> would it be ok for you to implement `RealmSearcher` and `RealmReader` similar to how you've done it in [your code-base](https://github.com/qqaatw/pytorch-realm-orqa) as well as a `RealmForOpenQA` class that wraps both of those classes so that QA can be done with a single model instance.

In a first step, I think we can transfer most of your code from https://github.com/qqaatw/pytorch-realm-orqa into `modeling_realm.py` and add the integration test - once that passes we can refactor a bit.

In a second step, I think we should think a bit about a better abstraction of the retrieval part - ideally we can implement a `RealmRetriever` similar in design to the `RagRetriever` in https://github.com/huggingface/transformers/blob/master/src/transformers/models/rag/retrieval_rag.py (@lhoestq and I are happy to help with that!)

Does this make sense to you? 

cc @lhoestq 
<|||||>Thank you very much for your detailed replies @patrickvonplaten, I agree with you that we should add `RealmSearcher`, `RealmReader`, and `RealmQA` together.

In fact, I initially think that because *REALM* and *ORQA* belong to different codebases, even if fine-tuned *ORQA* checkpoints of *REALM* are provided, they should be put into different models. However, considering UX and costly *REALM* pre-training procedure to most users, integrating them into a single model in `transformers` totally makes sense!

The branch of `transformers` codebase in https://github.com/qqaatw/pytorch-realm-orqa is based on the branch of this PR, so I can merge it here seamlessly. 

Before doing so, there are some concerns that we should clarify first:

1. The term `Searcher` in ORQA's codebase is actually called `Retriever`, and I changed it to `Searcher` in order to prevent the name conflict between REALM's retriever and ORQA's retriever (Their logics are slightly different but have the same purpose.). Do you think this naming is OK?
2. Currently `RealmSearcher` is leveraging `ScaNN` python package as the vector similarity searcher, would it be OK to add this package into `transformers`' requirements in `setup.py`? If we can't, there are two possible alternatives:
    -  Asking users to install it manually in the model's docs.
    -  Implementing brute-force matrix multiplication to replace `ScaNN`, (mentioned [here](https://github.com/google-research/language/tree/master/language/realm#code)).
3. Since `block records` and `block embs` are bound together, `block records` is the corpus containing billions of documents, and `block embs` is a tensor with shape (num_block_records, retriever_proj_size) pre-computed from the corpus, if we upload `block records` into `datasets`'s index, we should also notify users that which model checkpoints in the hub containing `block embs` are corresponded to the `block records` in `datasets`.
4. Would it be suitable to upload Google's official pre-trained and fine-tuned checkpoints to the hub under my HF account? or uploading it under Google's org account is a better choice? If so, do I need to get permission from them first? (Although I've uploaded some checkpoints to my account for testing)<|||||>Hey @qqaatw,

Thanks for your answer:

1) I think both "Searcher" and "Retriever" is fine! - happy to stick with "Searcher"
2) Yes good question - we'll probably have to add an optional dependency to `setup.py` indeed. We can take care of this later though. The idea is to verify that scann is installed as soon as `Realm....from_pretrained(...)` is called. For now, feel free to just add it and I think we can later take care of it. It would be nice to have both the possibility to use Scann as well as brute-force matrix multiplication (maybe we can add a switch to the REALM config) - something like `use_scann`?
3) cc @lhoestq what do you think here? Should we add both `block records` and `block embeds` to `datasets` with `block embeds` being model-specific or add just `block records` to `datasets` and `block embeds` to the corresponding model repo?
4) Yes for now feel free to upload under your account name.


Thanks a lot for your work on this :-)<|||||>Hello @patrickvonplaten, thanks for your answer.

I've added `RealmSearcher`, `RealmReader`, and `RealmForOpenQA` into the model; also, I've added `use_scann` option in the config for search method switching (brute-force matrix multiplication searcher has been implemented).

We can complete integration tests as soon as the strategy of storing block records is decided.<|||||>> Hello @patrickvonplaten, thanks for your answer.
> 
> I've added `RealmSearcher`, `RealmReader`, and `RealmForOpenQA` into the model; also, I've added `use_scann` option in the config for search method switching (brute-force matrix multiplication searcher has been implemented).
> 
> We can complete integration tests as soon as the strategy of storing block records is decided.

Awesome work @qqaatw :-) Did you already push the additions? I can't see `RealmForOpenQA` in `modeling_realm.py` yet. @lhoestq - I think we should store the block records in `datasets` no?<|||||>@patrickvonplaten The changes have been merged, thanks!<|||||>@qqaatw - ok I think we have a general outline of how to implement everything in mind now. I think this will be a slighly bigger PR than originally expected but the result should be a very nice model :-) I'll try to help you with the integration as much as possible in the coming days/weeks! 

In a first step, could you add a very hacky way of making an integration test pass? This way I can see exactly how each of the components interact with each other and can reproduce the results locally - feel free to retrieve the block records what is most sutiable for you right now. We'll iterate from this first version then :-)<|||||>I believe @patrickvonplaten and @lhoestq are on it, but it's a very big contribution (thanks @qqaatw!!) so it might take a bit of time. Sorry about that!<|||||>Hey @qqaatw, 

I freed up some time next week to dive into this! Very sorry for the delay<|||||>@patrickvonplaten,

Thanks for taking time to dive into this, especially in the holiday. I'll keep tracking this thread to discuss more in detail.

Merry Christmas!<|||||>> @patrickvonplaten,
> 
> Thanks for taking time to dive into this, especially in the holiday. I'll keep tracking this thread to discuss more in detail.
> 
> Merry Christmas!

Hey @qqaatw,

Thanks a lot for the nice words! Merry Christmas to you as well! 

The docs are now cleaned and now we can start to look how to best integrate REALM :-) 
In a first step, it would be amazing if we could make sure that the performance of REALM matches more or less the paper. I see that you have some very nice analysis in your repo here: https://github.com/qqaatw/pytorch-realm-orqa#naturalquestionsnq

Do you think you could post a command here that allows to reproduce those results with the current code, *e.g.* using this format:

```python
        model = RealmForOpenQA.from_pretrained( 
            r"qqaatw/realm-orqa-nq-searcher",
            r"qqaatw/realm-orqa-nq-reader",                                                    
            BLOCK_RECORDS_PATH,                                                                
        )                                                                                      

        question = "Who is the pioneer in modern computer science?"
        searcher_output, reader_output, predicted_answer = model(question)

        self.assertEqual(predicted_answer, "alan mathison turing")
```
just like in the test: `test_inference_open_qa`. Once we have verified this, I think I'm able to quickly onboard @lhoestq and others to get this PR merged. Let me know if this is not very clear here :-)

In a first step I downloaded `https://storage.cloud.google.com/orqa-data/enwiki-20181220/blocks.tfr` and ran `test_inference_open_qa`. The weird thing is that when running it multiple times, sometimes I'm getting the correct answer, but unfortunately I also sometimes (20% of the time) get the following error: 

```bash
E       AssertionError: 'charles babbage' != 'alan mathison turing'                                                                                                                           
E       - charles babbage                                                                                                                                                                     
E       + alan mathison turing                                                                                                                                                                                                                                                                                                                                                          
tests/test_modeling_realm.py:489: AssertionError 
```

So there seems to be some kind of randomness in the forward pass - could you verify whether this is the case for you as well? 80% of the time I'm however getting the correct solution. 
Note that the reason could also be tiny differences in logits precision for multiple forward passes which I've previously seen with RAG as well. So if you can't reproduce the result it's fine as is I think :-) The important part would now be to verify your eval results here: https://github.com/qqaatw/pytorch-realm-orqa#naturalquestionsnq with this HF implementation .

BTW, I'm using the following envs (transformers on this branch):

```
- `transformers` version: 4.16.0.dev0 (this branch)
- Platform: Linux-5.3.0-64-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyTorch version (GPU?): 1.10.0+cu102 (True)
- Tensorflow version (GPU?): 2.7.0 (False)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No
```

and 

```
scann version: 1.2.4
```

in case this might be a reason for the randomness in the test.<|||||>Hi @patrickvonplaten,

Thanks for testing this out. I cannot reproduce the failure with brute-force searcher but can reproduce with ScaNN searcher.

It seems that the result from ScaNN is sometimes not deterministic using the same ScaNN parameters as that of in the `orqa` codebase. In contrast, because brute-force searcher always computes all inner products and finds the top K highest scores, it produces consistent results. 

Here is the way that uses brute-force searcher:
```python
    realm_config=RealmConfig(use_scann=False)
    model = RealmForOpenQA.from_pretrained( 
        r"qqaatw/realm-orqa-nq-searcher",
        r"qqaatw/realm-orqa-nq-reader",                                                    
        BLOCK_RECORDS_PATH,
        config=realm_config,                         
    )                                                                                      

    question = "Who is the pioneer in modern computer science?"
    searcher_output, reader_output, predicted_answer = model(question)

    assert predicted_answer == "alan mathison turing"
```

My testing env:
```
- `transformers` version: 4.16.0.dev0
- Platform: Linux-5.4.0-91-generic-x86_64-with-glibc2.17
- Python version: 3.8.10
- PyTorch version (GPU?): 1.9.0+cu111 (True)
- Tensorflow version (GPU?): 2.6.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.3.6 (cpu)
- Jax version: 0.2.24
- JaxLib version: 0.1.69
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False

ScaNN version:
scann                         1.2.3
```


For benchmark reproductions, because it requires some data loading, preprocessing, and evaluation function, there might not be practical to paste the entire code here. I just wrote a compact [script](https://github.com/qqaatw/pytorch-realm-orqa/blob/master/benchmark.py) that reproduces both NQ and WQ benchmark results using current up-to-date HF impl.

To run NQ:

```bash
python benchmark.py \
    --dataset_name_path natural_questions \
    --retriever_pretrained_name qqaatw/realm-orqa-nq-searcher \
    --checkpoint_pretrained_name qqaatw/realm-orqa-nq-reader \
    --block_records_path ./data/enwiki-20181220/blocks.tfr \
    --device cuda
```

To run WQ:

```bash
python benchmark.py \
    --dataset_name_path web_questions \
    --retriever_pretrained_name qqaatw/realm-orqa-wq-searcher \
    --checkpoint_pretrained_name qqaatw/realm-orqa-wq-reader \
    --block_records_path ./data/enwiki-20181220/blocks.tfr \
    --device cuda
```<|||||>> Hi @patrickvonplaten,
> 
> Thanks for testing this out. I cannot reproduce the failure with brute-force searcher but can reproduce with ScaNN searcher.
> 
> It seems that the result from ScaNN is sometimes not deterministic using the same ScaNN parameters as that of in the `orqa` codebase. In contrast, because brute-force searcher always computes all inner products and finds the top K highest scores, it produces consistent results.
> 
> Here is the way that uses brute-force searcher:
> 
> ```python
>     realm_config=RealmConfig(use_scann=False)
>     model = RealmForOpenQA.from_pretrained( 
>         r"qqaatw/realm-orqa-nq-searcher",
>         r"qqaatw/realm-orqa-nq-reader",                                                    
>         BLOCK_RECORDS_PATH,
>         config=realm_config,                         
>     )                                                                                      
> 
>     question = "Who is the pioneer in modern computer science?"
>     searcher_output, reader_output, predicted_answer = model(question)
> 
>     assert predicted_answer == "alan mathison turing"
> ```
> 
> My testing env:
> 
> ```
> - `transformers` version: 4.16.0.dev0
> - Platform: Linux-5.4.0-91-generic-x86_64-with-glibc2.17
> - Python version: 3.8.10
> - PyTorch version (GPU?): 1.9.0+cu111 (True)
> - Tensorflow version (GPU?): 2.6.0 (False)
> - Flax version (CPU?/GPU?/TPU?): 0.3.6 (cpu)
> - Jax version: 0.2.24
> - JaxLib version: 0.1.69
> - Using GPU in script?: False
> - Using distributed or parallel set-up in script?: False
> 
> ScaNN version:
> scann                         1.2.3
> ```
> 
> For benchmark reproductions, because it requires some data loading, preprocessing, and evaluation function, there might not be practical to paste the entire code here. I just wrote a compact [script](https://github.com/qqaatw/pytorch-realm-orqa/blob/master/benchmark.py) that reproduces both NQ and WQ benchmark results using current up-to-date HF impl.
> 
> To run NQ:
> 
> ```shell
> python benchmark.py \
>     --dataset_name_path natural_questions \
>     --retriever_pretrained_name qqaatw/realm-orqa-nq-searcher \
>     --checkpoint_pretrained_name qqaatw/realm-orqa-nq-reader \
>     --block_records_path ./data/enwiki-20181220/blocks.tfr \
>     --device cuda
> ```
> 
> To run WQ:
> 
> ```shell
> python benchmark.py \
>     --dataset_name_path web_questions \
>     --retriever_pretrained_name qqaatw/realm-orqa-wq-searcher \
>     --checkpoint_pretrained_name qqaatw/realm-orqa-wq-reader \
>     --block_records_path ./data/enwiki-20181220/blocks.tfr \
>     --device cuda
> ```

Thanks for diving into the random output story. I indeed only tried the "ScaNN" approach. Given that "ScaNN" has a TF dependency I think it's better anyways to switch to `use_scann=False`. Just tried it and it seems to work well. 

Thanks a lot for providing the benchmarking script - I'll run it today and start to think about how we can best integrate the model :-)
That's great! That's exactly what I was looking for :-) <|||||>@qqaatw - I can reproduce the eval results which is great. I've started to modify the structure of the elements a bit. I still need to do some modifications tomorrow, but after that I could maybe hand back to you :-) <|||||>Hey @qqaatw,

I think we are pretty close to have a version that can be integrated into `transformers`. I've now finished the main changes, which were:
- No `tokenizer` should ever be saved in a model, which is why I moved all tokenizer logic out of the model's forward pass
- Non-PyTorch related code to the retrieval should be handled by a `RealmRetriever` class which I have added and integrated into the model.
- The model `RealmForOpenQA` now **only** does PyTorch matrix multiplication and no tokenizer or retrieval-like operations. This way `RealmForOpenQA` keeps the same format of all other Transformer models.

For now, I think it is enough to ensure whenever commits are finished that the test `test_inference_open_qa` still works correctly. 

@qqaatw - the next big step now is to fully remove the `RealmSearcher` class and to load all necessary weights directly in `RealmForOpenQA` . The class `RealmForOpenQA` also should not have a special `from_pretrained(...)` or `save_pretrained(...)` method but should be able to use the default ones. 

We are looking for the following API in the end:

```python
question = "some question"

model_id = "/path/to/full/realm/model/on/hub/"

tokenizer = RealmTokenizer.from_pretrained(model_id)
retriever = RealmRetriever.from_pretrained(model_id)  # will load the heavy tf records file

model = RealmForOpenQA.from_pretrained(model_id, retriever=retriever)

question_ids = tokenizer(question, return_tensors="pt").input_ids
predicted_ids = model(question_ids)[0]

answer = tokenizer.decode(predicted_ids)
```

which is very similar to https://huggingface.co/facebook/rag-token-nq#usage<|||||>To get ```model = RealmForOpenQA.from_pretrained(model_id, retriever=retriever)``` you will probably have to do the following:

1. Delete the special `save_pretrained(...)` method in `RealmForQA`
2. Load `RealmForQA` as done in the `test_inference_open_qa` now.
3. Then save the model as **one** using the default `save_pretrained(...)` method (since you deleted the old one) in `/temp_dir`
4. Delete the special `from_pretrained(...)` method in `RealmForQA` 
5. Now try to load the model with `model = RealmForOpenQA.from_pretrained(model_id, retriever=retriever)` where `model_id` is your just saved `/temp_dir` directory.
6. Debug this until `test_inference_open_qa` works and then upload the single `pytorch_model.bin` file to a repo<|||||>Let me know if you have difficulties with this and I can try to take a look :-)<|||||>Hey @patrickvonplaten,

Thanks a lot for doing these modifications. I will work on it tomorrow.<|||||>Hey @qqaatw,

I've done the main modifications now to form the structure into what is common for retrieval-augmented models in `transformers`. See my comments here: 
https://github.com/huggingface/transformers/pull/13292#discussion_r777452449 and here: https://github.com/huggingface/transformers/pull/13292/files#r777456805. Is this ok for you? Does that make sense? Do you want to discuss anything? 

Could you maybe copy the checkpoint: https://huggingface.co/patrickvonplaten/realm-open-qa under your namespace and play around with it to see if you're ok with those changes? 

The next steps would be: 
- refactor the code (make it cleaner, more comments, treat the suggestions above, add `RealmForOpenQAOutputs`, etc...)
- Then we also need to wait for @lhoestq to see how we can best store the tf records in a retriever

Let me know if you have any questions or would like to discuss anything :-) <|||||>Hi @patrickvonplaten,

I've completed the most modifications and still need to do some improvements.

Let me know if there is anything that needs to be improved or fixed.
<|||||>Great work @qqaatw! We are very close with merging this PR I think. Everything fits very well now IMO and there is not much left to to:

- Rename `bert` to `realm` 
- Add a `test_realm_retrieval.py` file (I can take care of it after talking to @lhoestq)
- See how to best integrate the block records in the retriever `.from_pretrained(...)` method. (I'll also take care of this).

We could then think also a bit about how to best demo this model :-) Maybe in a space (depending on how big the required RAM is). Also I think a blog post could be a great idea<|||||>@qqaatw - I just discussed with @lhoestq that we should store the block records as a numpy file on the hub. I've done some final changes to the retriever and uploaded the block records in numpy format here: https://huggingface.co/qqaatw/realm-orqa-nq-openqa/commit/ea416e495785cd9612f659b69af3a7857c91fe2a 
Note that this has the **big** advantage that your PyTorch code is now fully independent of TF -> there is no need to install TF anymore :-)

I think the API is not more or less complete. The test `test_inference_open_qa` should now work without having to call upon any hard-coded path. Could you try it out on your side?

I noticed that you already corrected your checkpoint to have `realm` in the weight names -> that's great. I've done some updates in the corresponding modeling file (hope you see this before doing the same thing yourself). 

Think we can merge the PR now by this week :-) It would be great if you could do some final clean-ups (fixing potentially failing tests) and if you want you could also give the `test_retrieval_realm.py` file a try (otherwise I can to it tomorrow or Friday). 

Let me know if you have any questions or need help :-)<|||||>Hey @patrickvonplaten,

Currently all the tests pass, and the new `block_records` handling looks working well. Thank you for making these changes.
Now I think we should upload the `.npy` file to every related model repo right? Could you help this out as the file is so huge that might take a long time to upload from my side. (should upload to the repo with suffix `openqa` I think)

Also, I've written a standalone checkpoint conversion script [here](https://github.com/qqaatw/pytorch-realm-orqa/blob/master/checkpoint_converter.py) as well as some instructions on the [readme](https://github.com/qqaatw/pytorch-realm-orqa/blob/master/README.md). Do you think we just provide a link directing to my repo in the model docs or should place it into `transformers`?

By the way, I'm not sure how to preview the generated docs website (I see this feature is under development in `CONTRIBUTING.md`).


<|||||>> We could then think also a bit about how to best demo this model :-) Maybe in a space (depending on how big the required RAM is). Also I think a blog post could be a great idea

Both sound appealing! Let me know if I could help with this.<|||||>> Hey @patrickvonplaten,
> 
> Currently all the tests pass, and the new `block_records` handling looks working well. Thank you for making these changes. Now I think we should upload the `.npy` file to every related model repo right? Could you help this out as the file is so huge that might take a long time to upload from my side. (should upload to the repo with suffix `openqa` I think)
> 
> Also, I've written a standalone checkpoint conversion script [here](https://github.com/qqaatw/pytorch-realm-orqa/blob/master/checkpoint_converter.py) as well as some instructions on the [readme](https://github.com/qqaatw/pytorch-realm-orqa/blob/master/README.md). Do you think we just provide a link directing to my repo in the model docs or should place it into `transformers`?
> 
> By the way, I'm not sure how to preview the generated docs website (I see this feature is under development in `CONTRIBUTING.md`).

I can definitely help with the uploading part once the PR is merged :-)<|||||>Amazing job ! Thanks for converting the records blocks to numpy format, it's more convenient this way. Later we can even allow to memory-map the numpy file to avoid filling up the RAM :)<|||||>I think this PR is more or less ready to be merged. @sgugger @LysandreJik could you give it a quick look maybe?<|||||>All test failures are due to the latest release of Tokenizers. A quick rebase on master should get rid of them.<|||||>Thanks for all your work on this!