hi @berkayberabi
to use Seq2SeqTrainer for prediction, you should pass predict_with_generate=True to  Seq2SeqTrainingArguments.  The trainer only does generation when that argument is True .
If it’s true then predictions returned by the predict method will contain the generated token ids.
Also if you want to do distributed evaluation, you could try this script transformers/run_distributed_eval.py at master · huggingface/transformers · GitHub 27