REALM
# 🌟 New model addition

REALM is from some of the authors of BERT (I like to think of it as the next BERT :) ) that have found a way to incorporate world knowledge (from Wikipedia) into the model.

They do this by having the concept of a retriever module that retrieves information from wikipedia articles.

<!-- Important information -->

## Open source status

Code not released at the moment but will probably be released by Google soon I'd imagine.

https://arxiv.org/abs/2002.08909

Code is released for the preceding paper: Latent Retrieval for Weakly Supervised Open Domain Question Answering" https://www.aclweb.org/anthology/P19-1612/
https://github.com/google-research/language/tree/master/language/orqa<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>I am also interested on it. Any news? Thanks<|||||>I'm also interested on this, are you planning to include this model in the library??<|||||>Finally the REALM code has been released here https://github.com/google-research/language/tree/master/language/realm.
I think that this issue should be re-opened. @aced125  are you able to re-open it?<|||||>I hope somebody can reopen this and make it happen<|||||>Any updates on this ? <|||||>Any update? 
<|||||>I think this could be implemented easily with RAG by making sure that we
both finetune doc encoder and question encoder of the retriever model. This
might be very useful for us.

On Wed, Feb 10, 2021, 23:42 Lysandre Debut <notifications@github.com> wrote:

> Reopened #3497 <https://github.com/huggingface/transformers/issues/3497>.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/3497#event-4314060510>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGULJ5VEDTMJD6F4ZXLS6JPJPANCNFSM4LVQ3UVQ>
> .
>
<|||||>Any Update on this?<|||||>@OctoberChang  I did extend the RAG in a way that can be used to experiment REALM stuff. 

https://paperswithcode.com/paper/fine-tune-the-entire-rag-architecture<|||||>@shamanez , thanks for the pointer to your RAG project [(link)](https://github.com/huggingface/transformers/tree/master/examples/research_projects/rag-end2end-retriever)
Do you have any pre-trained (not fine-tuned on downstream tasks) models with ICT or Salient span masking [[link]](https://arxiv.org/pdf/2002.08909.pdf) that can be load from Huggingface Transformers?<|||||>I haven't extended into the ICT. Sorry.
@OctoberChang  I think to get REALM models loaded with the current HF retrieval framework, we might need some workarounds. This is mainly because RAG uses a generative model and REALM consists of extractive LM. <|||||>@qqaatw's WIP PR (https://github.com/huggingface/transformers/pull/13292) adds REALM.