Errors when fine-tuning RAG on cloud env
Hi the team,

I'm trying to fine-tune RAG with [the scripts you provided](https://github.com/huggingface/transformers/tree/9ec0f01b6c3aff4636869aee735859fb6f89aa98/examples/research_projects/rag). My env is cloud servers (4 V100 with 48G GRAM), and I always have these errors when do the fine-tuning:

> RuntimeError: Error in faiss::Index* faiss::read_index(faiss::IOReader*, int) at /__w/faiss-wheels/faiss-wheels/faiss/faiss/impl/index_read.cpp:480: Error: 'ret == (size)' failed: read error in <cache path>: 6907889358 != 16160765700 (Success)

It seems like errors are from faiss (and I don't know how to interpret it. Sizes do not macth?). I used this command to do the fine-tuning:

```bash
- python run_rag_ft.py
        --data_dir /msmarco
        --output_dir ./msmarco_rag
        --model_name_or_path facebook/rag-sequence-nq
        --model_type rag_sequence
        --fp16
        --gpus 4
        --distributed_retriever pytorch
        --num_retrieval_workers 4
        --fp16
        --profile
        --do_train
        --do_predict
        --n_val -1
        --train_batch_size 8
        --eval_batch_size 1
        --max_source_length 128
        --max_target_length 40
        --val_max_target_length 40
        --test_max_target_length 40
        --label_smoothing 0.1
        --dropout 0.1
        --attention_dropout 0.1
        --weight_decay 0.001
        --adam_epsilon 1e-08
        --max_grad_norm 0.1
        --lr_scheduler polynomial
        --learning_rate 3e-05
        --num_train_epochs 2
        --warmup_steps 500
        --gradient_accumulation_steps 1
```

Nothing special but just use my own data (MSMARCO). I sticked to the pytorch for the distributed retriever, and have not yet tested the ray version. Is that the problem?

I cannot run this on my local machines because of OOM errors (two 24GRAM GPUs).


I think @patrickvonplaten could help me on this. Thanks!

Hey @DapangLiu,

We sadly don't actively maintain the `research_projects` folder except for Wav2Vec2. Could you try to use the forum: https://discuss.huggingface.co/ instead? <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.