I figured out your problem: the Trainer is optimized for the Transformers models so it expects the model outputs to always be a tuple, with the loss first and logits second if labels are provided, or just the logits if no labels are provided.
Therefore, in you custom compute_loss of your subclass of Trainer, the last line should be:
    return (loss, (loss, logits)) if return_outputs else loss