You can avoid the metric computation by just not passing the compute_metrics function to your Trainer. Otherwise, yes, you should use a portion of your evaluation dataset only to avoid the crash, then run a manual evaluation loop on the whole evaluation dataset when you’re done. If you don’t use Trainer, you can use the fact that Datasets metric store the intermediate results on disk and avoid your OOM error by doing:
for batch in eval_dataloder:
    batch = {k: v.to(device) for k, v in batch.items()}
    with torch.no_grad():
        outputs = model(**bach)
    pred_ids = np.argmax(outputs.logits.cpu().numpy(), axis=-1)
    labels = batch["labels"].cpu().numpy()
    labels[labels == -100] = processor.tokenizer.pad_token_id

    pred_str = processor.batch_decode(pred_ids)
    # we do not want to group tokens when computing the metrics
    label_str = processor.batch_decode(labels group_tokens=False)

    wer = wer_metric.add_batch(predictions=pred_str, references=label_str)

final_result = wer.compute()