Hi. I have a similar problem – I get one less class predicted (e.g. y_true = [2, 3, 4], y_hat = [3 ,4]).
I modified the Trainer class as following, because I want to add “weight” argument to my CrossEntropyLoss. But, like in your case, I get one less prediction from the .evaluate() method.
class TrainerCustom(transformers.Trainer):

    def __init__(self, weights, *args, **kwargs):
        super().__init__(*args, **kwargs)

        # initialize weights from argument
        self.weights = weights

    def compute_loss(self, model, inputs, return_outputs=False):

        labels = inputs.pop("labels")

        outputs = model(**inputs).logits

        self.weights = self.weights.to(labels.device)

        cross_entropy_loss_func = torch.nn.CrossEntropyLoss(weight = self.weights)
   
        loss = cross_entropy_loss_func(outputs, labels.long())
        
        return (loss, outputs) if return_outputs else loss

You would know how to modify the Trainer class so I get rid of the bug of unequal predictions, but keep the weight argument (i.e. modified loss function)?
Also, how would I go about using your “dirty” solution, i.e. appending a [1] that gets then removed - where would I add it? I use the Trainer class like you, but I need to append it to the tensor that goes into EvalPrediction, because I want it to return +1  predictions so that the first gets removed (since it’s just a dummy). Not sure where to do that.
Thanks!