Evaluate on subset of evaluation dataset during training
# 🚀 Feature request

Allow evaluating the model on a random subset of the evaluation dataset at each evaluation step during training.

## Motivation

While participating in the Robust Speech Recognition Challenge in January / February 2022, I noticed that evaluating the model on the whole evaluation dataset increased the training time considerably (Swedish Common Voice, in that specific case).

Sampling a subset of the evaluation set before training is not the optimal approach as it would lead to a less realistic model evaluation.

The same issue was discussed [here](https://discuss.huggingface.co/t/evaluate-subset-of-data-during-training/10952).

## Your contribution

I will update the Trainer and TrainingArguments classes, HFArgumentParser (if needed), tests, and docs to allow this feature, before submitting a Pull Request.

## Usage example

```
from transformers import TrainingArguments

[...]

training_args = TrainingArguments(
    ...,
    eval_subset_ratio=0.1,
    eval_subset_seed=0
)

[...]




cc @patrickvonplaten @anton-l <|||||>Hey @marinone94, 

Thanks for your issue! Do you think the https://github.com/huggingface/transformers/blob/b7e292aebdb638e2238cd9febf8c09253195fb5d/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py#L181 option is not enough here?<|||||>Hej @patrickvonplaten, by using `max_eval_samples` one always evaluates on the same subset (in the example, the top `max_eval_samples` examples, but it could well be a random selection, it would not change my point).

What I would like to offer (and use myself) is to dynamically sample `max_eval_samples` before each eval step, so that the model is not always evaluated on the same subset during training, and then run a full evaluation at the end of the training. I think it would expand the library's offer while keeping it fully compatible with previous versions.<|||||>It's pretty difficult to compare randomly sampled eval datasets to each other no? E.g. if epoch 1 gives me 40% WER on a subset 1 and epoch 2 gives me 40% on a subset 2 which is completely different from 1 then I can't really use this information this much no? <|||||>Well, if the model generalises well the trend should be similar, but I see your point.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.