Prediction loop: work with batches of variable length (fixed per batch)
In the current form, the `prediction_loop` doesn't handle batches with samples of varying lengths (but fixed length per batch). This patch adds this capability.
This is great because it can save a lot of time during training and inference, where using the full length every time is a big sacrifice, knowing self-attention scales in n**2.
Disclaimer: This is a strictly personal contribution, not linked to my professional affiliation in any way.

@sgugger 

https://github.com/huggingface/transformers/issues/8483

Please feel free to modify this in any shape or form of course.<|||||>This will cause a regression for code expecting a NumPy output, obviously :/<|||||>I don't have much time, if any, to dedicate to this. If you're not interested by the idea, it's completely fine, I will close.<|||||>I think @jplu is redesigning the TFTrainer. Maybe this should be reopened once that design has been merged in master?<|||||>This won't be compliant anymore because the redisign doesn't use custom loops.<|||||>Do you support different batch lengths in the new one? @jplu <|||||>It is not on top of the list but, yes for sure, we plan to support it, including for training.