Trainer accumulates logits
Hi,

I am using `transformers.Trainer` to pre-train a model with MLM. From line 2213 in `trainer.py` I can see the logits obtained on the evaluation step are accumulated: 
`preds_host = logits if preds_host is None else nested_concat(preds_host, logits, padding_index=-100)`

This makes it impossible to use a reasonable validation dataset, since already 1000 examples with `max_length = 512` and `vocab_size = 30522` occupy 1000 * 512 * 30522 / 1024^3 > 14 Gb of memory (e.g. **c4** dataset has validation dataset of size 365 000). This can be corrected if the additional metrics are calculated on each validation step for each batch separately rather than in the end.

This implies lines 2272-2275 should be moved inside the _for loop_. If you agree with all I have stated, I can do it by my own and come up with the merge request.
@sgugger <|||||>No, metrics usually can't be computed on a batch-per-batch basis as it usually gives the wrong result when the metric is not a mean (like precision, recall or F1 score).

For metrics in language modeling, you should use your own manual evaluation loop after training.<|||||>Fair enough, but if I do not need any metrics at all (need to only track
the loss value)? I still cannot use the validation sample since the logits
will be accumulated anyway.

On Mon, 26 Jul 2021 at 15:39, Sylvain Gugger ***@***.***>
wrote:

> No, metrics usually can't be computed on a batch-per-batch basis as it
> usually gives the wrong result when the metric is not a mean (like
> precision, recall or F1 score).
>
> For metrics in language modeling, you should use your own manual
> evaluation loop after training.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/12878#issuecomment-886666573>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AIXZKXMPSQLGIY2ZROP7ZQ3TZVJRTANCNFSM5A6RXYNA>
> .
>
<|||||>If you don't pass any `compute_metrics` function, they won't be accumulated, or you can force it with `prediction_loss_only=True` in your `TrainingArguments`.<|||||>Thanks a lot! Did not know about this parameter. To sum up, while pre-training a model, we need to write a custom evaluation loop to use custom metrics? Probably it is worth adding a special parameter for the metrics, indicating whether its computation can be split into batches or not? I can handle this.<|||||>Yes, we don't have anything setup in the Trainer for metric accumulation, so basically any time you want to avoid accumulating logits (so all language modeling tasks basically), you will need a custom training loop.

We'll try to come up with an API to make it easier to do a batch-by-batch accumulation but that probably will need a rewrite of some pieces of the Trainer, which in turn might cause some breaking changes. So it's probably going to be for v5