RAG end to end with RAY throws pickling error
**Environmnet info:**

Transformers:4.5.1
Paltform:Ubuntu
Python:3.7
Torch 1.6.0 
Gpus = yes
Distributed: Ray (1.3.0)

**Information**

Am using RAG end2end-retriever from examples code
examples/research_projects/rag-end2end-retriever

For the time being just trying on dummy data and just trying the script given there
sh ./test_run/test_finetune.sh 
Used the config there except just changes number of gpus to 4 and changed the gpu order

**Who can help**
@shamanez 

**Error**

Throws pickling error

      Loading passages from test_run/dummy-kb/my_knowledge_dataset
      Traceback (most recent call last):
        File "finetune_rag.py", line 790, in <module>
          main(args)
        File "finetune_rag.py", line 727, in main
          model: GenerativeQAModule = GenerativeQAModule(args)
        File "finetune_rag.py", line 124, in __init__
          hparams.model_name_or_path, hparams.actor_handles, config=config
        File "/home/shraeyb/transformers/examples/research_projects/rag-end2end-retriever/distributed_ray_retriever.py", line 165, in from_pretrained
          index=index,
        File "/home/shraeyb/transformers/examples/research_projects/rag-end2end-retriever/distributed_ray_retriever.py", line 93, in __init__
          for worker in self.retrieval_workers
        File "/home/shraeyb/transformers/examples/research_projects/rag-end2end-retriever/distributed_ray_retriever.py", line 93, in <listcomp>
          for worker in self.retrieval_workers
        File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/actor.py", line 112, in remote
          return self._remote(args, kwargs)
        File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/actor.py", line 153, in _remote
          return invocation(args, kwargs)
        File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/actor.py", line 147, in invocation
          num_returns=num_returns)
        File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/actor.py", line 865, in _actor_method_call
          list_args, name, num_returns, self._ray_actor_method_cpus)
        File "python/ray/_raylet.pyx", line 1359, in ray._raylet.CoreWorker.submit_actor_task
        File "python/ray/_raylet.pyx", line 1364, in ray._raylet.CoreWorker.submit_actor_task
        File "python/ray/_raylet.pyx", line 304, in ray._raylet.prepare_args
        File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/serialization.py", line 324, in serialize
          return self._serialize_to_msgpack(value)
        File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/serialization.py", line 304, in _serialize_to_msgpack
          self._serialize_to_pickle5(metadata, python_objects)
        File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/serialization.py", line 264, in _serialize_to_pickle5
          raise e
        File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/serialization.py", line 261, in _serialize_to_pickle5
          value, protocol=5, buffer_callback=writer.buffer_callback)
        File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 73, in dumps
          cp.dump(obj)
        File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 580, in dump
          return Pickler.dump(self, obj)
        File "pyarrow/io.pxi", line 1021, in pyarrow.lib.Buffer.__reduce_ex__
      AttributeError: module 'pickle' has no attribute 'PickleBuffer'
      INFO:wandb.internal.internal:Internal process exited
      /home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/autoscaler/_private/cli_logger.py:61: FutureWarning: Not all Ray CLI dependencies were found. In Ray 1.4+, the Ray CLI, autoscaler, and dashboard will only be usable via `pip install 'ray[default]'`. Please update your install command.
        "update your install command.", FutureWarning)
      Stopped all 13 Ray processes.
Check this[ StackOverflow question](https://stackoverflow.com/questions/67798070/raytune-is-throwing-error-module-pickle-has-no-attribute-picklebuffer-whe).  btw I use python 3.8 and it worked perfectly for me.<|||||>Yup i did but that is also just the question without any answer unfortunately.<|||||>Hi there, I executed the code again by installing RAY just to double-check. It is working perfectly for me.   I also use an anaconda and RAY with 1.3.0. 

The only difference is python 3.8 instead of 3.7. 

I would suggest you to update the anaconda and trying to run the script.<|||||>Alright thank you, i used python 3.8 now ( though my cuda is 10.1), That error looks resolved but something new pops up, Still trying on dummy data

        File "finetune_rag.py", line 790, in <module>
          main(args)
        File "finetune_rag.py", line 727, in main
          model: GenerativeQAModule = GenerativeQAModule(args)
        File "finetune_rag.py", line 131, in __init__
          retriever.set_ctx_encoder_tokenizer(ctx_encoder_tokenizer)
      AttributeError: 'RagRayDistributedRetriever' object has no attribute 'set_ctx_encoder_tokenizer'
      Exception ignored in: <function ActorHandle.__del__ at 0x7fc4c143aee0>
      Traceback (most recent call last):
        File "/home/shraeyb/anaconda3/envs/py38/lib/python3.8/site-packages/ray/actor.py", line 809, in __del__
      AttributeError: 'NoneType' object has no attribute 'global_worker'
      Exception ignored in: <function ActorHandle.__del__ at 0x7fc4c143aee0>
      Traceback (most recent call last):
        File "/home/shraeyb/anaconda3/envs/py38/lib/python3.8/site-packages/ray/actor.py", line 809, in __del__
      AttributeError: 'NoneType' object has no attribute 'global_worker'
      Exception ignored in: <function ActorHandle.__del__ at 0x7fc4c143aee0>
      Traceback (most recent call last):
        File "/home/shraeyb/anaconda3/envs/py38/lib/python3.8/site-packages/ray/actor.py", line 809, in __del__
      AttributeError: 'NoneType' object has no attribute 'global_worker'
      Exception ignored in: <function ActorHandle.__del__ at 0x7fc4c143aee0>
      Traceback (most recent call last):
        File "/home/shraeyb/anaconda3/envs/py38/lib/python3.8/site-packages/ray/actor.py", line 809, in __del__
      AttributeError: 'NoneType' object has no attribute 'global_worker'
      Stopped all 13 Ray processes.<|||||>Ah, I think you have to install the library from sources. Still, this is not in the pip version.<|||||>You mean ray 1.3 from source?<|||||>Tranformers library.

On Fri, Jun 4, 2021, 02:32 Shraey Bhatia ***@***.***> wrote:

> You mean ray 1.3 from source?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/12012#issuecomment-853916373>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGRFOV4TWZSIZNP3IODTQ6HARANCNFSM46AZYZBQ>
> .
>
<|||||>Thank you. This solves it. Indded needs to be python 3.8
Though i have a few 8gb gpus so cannot really fit on my box, goes out of memory. What gpu config did you use?<|||||>Oh.  I am working with 32Gb gpus. This has nearly 800 M parameters. Bart
large model and two Bert base models.

On Fri, Jun 4, 2021, 02:58 Shraey Bhatia ***@***.***> wrote:

> Thank you. This solves it. Indded needs to be python 3.8
> Though i have a few 8gb gpus so cannot really fit on my box, goes out of
> memory. What gpu config did you use?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/12012#issuecomment-853935772>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGS4TI7XJMAZAK3NOXDTQ6KCNANCNFSM46AZYZBQ>
> .
>
<|||||>No worries. thats fine, I will have to submit jobs to a cluster ( is just tedious to do that). Thank you