Hi, sorry for the late reply.
I was able to fix it with this definition for compute_metrics:
def compute_metrics(p: EvalPrediction):

    metric_acc = datasets.load_metric("accuracy")    
    preds = p.predictions[0] if isinstance(p.predictions, tuple) else p.predictions
    preds = np.argmax(preds, axis=1)
    result = {}
    result["accuracy"] = metric_acc.compute(predictions=preds, references=p.label_ids)["accuracy"]
    return result

I think the issue is that BartForSequenceClassification returns sequence-to-sequence-classifications instead of a single classification for an input sequence (As Bert does) and I copied the metrics-computation from a script which utilized BertForSequenceClassification.