RAG-2nd2end-revamp
same as [shamanez:rag-retriever-end2end](https://github.com/huggingface/transformers/pull/11655) PR. In the previous version, I got some version control problems.

Really sorry to come up with duplicate PRs :(.   

@lhoestq  @patrickvonplaten 

I conducted an experiment to check the difference and added a simple test run code. 


Finally I added two test functions to test_modeling_rag.py and test_retrieval_rag.py.  

![image](https://user-images.githubusercontent.com/16892570/119752765-ad5ed080-bef1-11eb-9579-28d3b7477446.png)


I added all the minor changes and I would like to thank @patrickvonplaten and @lhoestq for the enormous amounts of support and advice.  :)<|||||> Hey 
Thanks for giving this end to end version. Am trying to run it and see its performance on my own dataset (much smaller and domain specific to see if get performance gains with end to end) but at the moment it is throwing a Pickling error with dummy dataset in the code. Am still stuck trying to understand how to fix this. Any idea how this can be dealt with

  
      File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 580, in dump
        return Pickler.dump(self, obj)
      File "pyarrow/io.pxi", line 1021, in pyarrow.lib.Buffer.__reduce_ex__
    AttributeError: module 'pickle' has no attribute 'PickleBuffer

Cheers
<|||||>Could you please add this as a new issue. Also I would like to see the
entire error log. Seems like something is wrong with your RAY installation.

On Fri, Jun 4, 2021, 00:49 Shraey Bhatia ***@***.***> wrote:

> Hey
> Thanks for giving this end to end version. Am trying to run it and see its
> performance on my own dataset (much smaller and domain specific to see if
> get performance gains with end to end) but at the moment it is throwing a
> Pickling error with dummy dataset in the code. Am still stuck trying to
> understand how to fix this. Any idea how this can be dealt with
>
>   File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 580, in dump
>     return Pickler.dump(self, obj)
>   File "pyarrow/io.pxi", line 1021, in pyarrow.lib.Buffer.__reduce_ex__
> AttributeError: module 'pickle' has no attribute 'PickleBuffer
>
> Cheers
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/pull/11893#issuecomment-853842534>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGWLRNB4P234HGI4JOLTQ5255ANCNFSM45TDZJNA>
> .
>
<|||||>Sure, just created a new issue with full log