DeepMind Retro
# 🌟 New model addition

Basically a retrieval augmented model like RAG, but without expensive retriever end2end training



## Open source status

* [ ] the model implementation is available: Not available 
* [ ] the model weights are available:  Not available 
* [ ] who are the authors:(https://arxiv.org/pdf/2112.04426.pdf)

Hi,

No weights are available at the moment. I'd suggest to take a look at this repo: https://github.com/lucidrains/RETRO-pytorch<|||||>Yeah, I saw it. Anyways I am thinking of combining Retro with RAG model. The results seem very promising also the model is very practical as recent work suggests end-to-end retriever training is very expensive and impractical.
<|||||>I would love to get some help :)<|||||>> I would love to get some help :)

Hi! I am interested in this implementation and I can help for it!<|||||>@halixness  do you think we should modify the RAG implementation ? <|||||>@shamanez @halixness @NielsRogge  I would like to participate in the implementation, please let me know<|||||>Hi All,

I think we can use the RAG pipeline.  But we need to implement the reader and its loss function. 

We might be able to use [this](https://github.com/lucidrains/RETRO-pytorch).

any suggestions? <|||||>> I would love to get some help :)

I'd like to help as well!<|||||>yeah this model seems cool. feel like it's getting closer to true open source LLMs. anyone aware if there are similar open-source efforts as bloom to reproduce and publish this?<|||||>Hi @shamanez, very I'm interested in RETRO. How far along are you in an implementation?<|||||>I did some digging and apparently Nvidia is picking up this idea again. I found it via an [article](https://www.marktechpost.com/2023/04/22/this-ai-paper-from-nvidia-provides-the-recipe-to-reproduce-retro-up-to-9-5b-parameters-while-retrieving-a-text-corpus-with-330b-tokens/) from April '23. It links to their [GitHub repo](https://github.com/NVIDIA/Megatron-LM/tree/main/tools/retro) and the corresponding [paper](https://arxiv.org/pdf/2304.06762.pdf) by Nvidia. In this paper they describe how they reproduce RETRO including the generation of the pretraining corpus.
However, in the paper they mention that the implementation and the weights are not open-sourced, but just being able to regenerate the pretraining corpus would get us one step ahead.