seq2seq trainer gives OOM error while evaluating
Hi,

I'm trying t5-base for translation with source and target lengths of 320 and 256 respectively. I'm using Seq2SeqTrainer on A100-40GB GPU. For training, it is consuming not more than 20GB of GPU memory with batch_size of 8. But for evaluation with batch_size of 4, it is giving CUDA Out Of Memory error.

Later I figured that if I remove `compute_metrics` from trainer, everything is working fine. So could you plz suggest what's wrong with my below `compute_metrics` function ..?

```
def compute_metrics(pred):
    sacrebleu = datasets.load_metric("metrics/sacrebleu.py")    
    labels_ids = pred.label_ids
    pred_ids = pred.predictions

    labels_ids[labels_ids == -100] = tokenizer.pad_token_id
    label_str = tokenizer.batch_decode(labels_ids, skip_special_tokens=True)
    pred_str = tokenizer.batch_decode(np.argmax(pred_ids[0],axis=-1), skip_special_tokens=True)

    label_string = []
    for l in label_str:
        label_string.append([l])

    sacrebleu_output = sacrebleu.compute(predictions=pred_str, references=label_string)

    return {
        "bleu": sacrebleu_output["score"]
    }
```
Hi, this could be because during evaluation `.generate` is used with beam search which uses more memory. Also by default, all predictions are stored on the device, so if the eval dataset is large it could result in OOM.

You could use `--eval_accumulation_steps` argument if it's passed the predictions are offloaded on CPU after every `eval_accumulation_steps` which reduced GPU memory.<|||||>Thanks a lot @patil-suraj , it worked!