RAG with RAY implementation: Ray workers memory slowly increase over time.
I fine-tune the RAG with RAY workers from 50 000 steps. When I checked the MEM% with the top command, I can see the memory consumption keep growing slowly. Usually, it should use around 20GB. After 50000 steps it raises up to 24 GB. 

This could eventually lead to crash the system with OOM error. I did a background check and found the Redis server usually keeps increasing its memory consumption. 

So is it ok to set a value  **object_store_memory**?


I get something similar to this... (found this from the [issue](https://github.com/ray-project/ray/issues/10431) )

![image](https://user-images.githubusercontent.com/16892570/115200801-b5845d00-a148-11eb-8447-b0c71b33b937.png)


@lhoestq  @amogkam 
I think I solved the memory leakage issue.  In my case, I simply wanted to update the index parameters in **self.retriever  object**. So I used **def set_index function**.  But I observe for some reason ray works completely cannot flush out the old index and its related objects.

So now when I want to update it, I delete the self.retriever object and re-initialize it. 

```
class RayRetriever:
    def __init__(self):
        self.initialized = False

    def create_rag_retriever(self, config, question_encoder_tokenizer, ctx_encoder_tokenizer,generator_tokenizer, index):
        if not self.initialized:
            self.retriever = RagRetriever(
                config,
                question_encoder_tokenizer=question_encoder_tokenizer,
                ctx_encoder_tokenizer=ctx_encoder_tokenizer,
                generator_tokenizer=generator_tokenizer,
                index=index,
                init_retrieval=False,
            )
            self.initialized = True

    def init_retrieval(self):
        self.retriever.index.init_index()

    def set_index(self,index):
        self.retriever.index=index #with this new index class all the paramters in HFindex class get updated
   

    def clear_object(self):  #we can call this first delete all thing things and again call create_rag_retriever
        del self.retriever
        self.initialized = False
     

    def retrieve(self, question_hidden_states, n_docs):

        doc_ids, retrieved_doc_embeds = self.retriever._main_retrieve(question_hidden_states, n_docs)
        doc_dicts= self.retriever.index.get_doc_dicts(doc_ids)

        return doc_ids, retrieved_doc_embeds,doc_dicts
``````