[WIP] Trainer supporting evaluation on multiple datasets
# What does this PR do?
With this PR `Trainer` and `Seq2SeqTrainer` support evaluating on multiple datasets. For this, the `eval_dataset` and `compute_metrics` parameters have been updated. In order to evaluate on multiple datasets, `eval_dataset` should be a dict mapping a dataset name to a Dataset. In `_maybe_log_save_evaluate` we then loop over the dict, calling `evaluate` with each Dataset. The metric prefix is also updated to contain the dataset name. Furthermore, each eval dataset can optionally have its own `compute_metrics` function. For this, `compute_metrics` should be a dict where the keys match with `eval_dataset`.

Fixes #15857


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
 @sgugger 
_The documentation is not available anymore as the PR was closed or merged._<|||||>Hey @sgugger, I mostly followed your suggestion in #15857, except instead of having a list of eval_datasets and another training arg, I solved it via passing a dict of eval_datasets. I thought a dict would work better because we also need multiple compute_metric functions. This way it is all lined up and less error-prone. However, let me know if you think otherwise.

Also, could you suggest what tests to write for this PR? I am not really sure, since the major change is in `_maybe_log_save_evaluate` and I didn't find a test for that.<|||||>Thanks for checking it so quickly!

In my case, I am training a seq2seq QA model and evaluating it on multiple datasets. However, they have different formats (eg extractive qa like SQuAD, or multiple-choice qa like commonsese QA). Using a seq2seq model for multiple formats has been for example proposed in the [UnifiedQA paper](https://arxiv.org/abs/2005.00700). Having multiple trainers has the limitation that I could only train on a single dataset at a time, but not train on multiple ones at the same time. However, note that if you pass multiple eval_datasets as a dict, but only a single compute_metric callable, the same compute_metrics function will be called on all the eval_datasets. That's what [this if statement](https://github.com/huggingface/transformers/pull/19158/files#diff-ed55888e6665791fe92cc8fc0c499da54f4ace6738551cd9a2591881cda076deR2048) is doing. So the original scenario described in the Issue is also solved.<|||||>It's too niche of a use-case to allow for support, especially when we have other tools that easily let you more customizable training/evaluation loops like Accelerate.<|||||>Alright, I have reverted the change. Let me know in case of anything else:)<|||||>I'm trying to take advantage of the feature to include multiple eval_datasets in the trainer. Maybe I'm misreading the documentation, I've tried several ways to present the eval_dataset, but keep getting a KeyError when I include a DatasetDict / dict with datasets for the eval_dataset parameter. Am I doing something wrong? Do I need to specify the compute_metrics differently? Couldn't find anything on that. 

Here's an example notebook resulting in the ValueError: https://colab.research.google.com/drive/1yLo9iqY4Cz9_h8BtAvcYRCtK5O_xa5jP?usp=sharing<|||||>> Thanks for your PR! Having the multiple datasets as a dict solves the problem of distinguishing a single dataset that is a list or a list of datasets. So I like this part.
> 
> However I didn't see anything in the issue regarding using several `compute_metrics` function. If there is a need for different metrics, it probably means different Trainer should be built as it represents different tasks/problems. That change should be reverted, as the part where `compute_metrics` can be passed along to the `evaluate`/`predict` function.

@sgugger passing multiple `compute_metrics` functions for evaluation purposes can actually be more general than stated by @timbmg. For example, suppose we are doing multi-task training and we wish to evaluate on the same or held-out tasks as we train. This is common in recent research publications (eg FLAN-T5). Would you accept to support the multiple compute metrics functions? Or would your advice be to not use the trainer altogether and look towards using `accelerate`? I was worried that `accelerate` for training is a big step back towards writing a lot of boilerplate and code that the `Trainer` saves us.<|||||>I'd recommend using Accelerate instead of the Trainer for this use case.<|||||>@sgugger Any examples out there of using Accelerate for this? I would also like to evaluate on multiple datasets while training. Thanks!