Hello everybody,
I am trying to use my own metric for a summarization task passing the compute_metrics to the Trainer class. I would like to calculate rouge 1, 2, L between the predictions of my model (fine-tuned T5) and the labels.
However, I have a problem understanding what the Trainer gives to the function. The EvalPrediction object should be composed of predictions and label_ids. To my understanding, since I am truncating the summaries to 150 characters, the predictions and label_ids should be vectors of size 150, or (batch_size, 150). Surprisingly, I get the predictions to be nested tuples, of size (23, 150, 32128) or (23, 12, 150, 64) or (23, 12, 512, 64), are these logits vectors? While the label_ids is a tuple with 23 vectors of size 150.
Can you kindly help me to understand what I should expect from the object EvalPrediction?
Thank you.