Note that you need to tokenize your labels in the target context manager, otherwise they will be tokenized as English and not German:
with tokenizer.as_target_tokenizer():
    labels = tokenizer(“Ich stecke mit text_pair Argument von Tokenizer fest.”, return_tensors="pt").input_ids