 Quantitative  | model | Rouge1 | Rouge2 | RougeL | RougeLsum | |-|-|-|-|-| | [T-Systems-onsite/mt5-small-sum-de-en-v2](https://huggingface.co/T-Systems-onsite/mt5-small-sum-de-en-v2)| 0.107 | 0.0297 | 0.098 | 0.098 | | aiautomationlab/german-news-title-gen-mt5 | 0.3131 | 0.0873 | 0.1997 | 0.1997 |  For evaluating the factuality of the generated headlines concerning the input text, we use 3 state-of-the-art metrics for summary evaluation (the parameters were chosen according to the recommendations from the respective papers or GitHub repositories). Because these metrics are only available for the English language the texts and generated headlines were translated from German to English using the [DeepL API](https://www.deepl.com/en/docs-api/) in an additional preprocessing step for this factuality evaluation.  - **SummaC-CZ** [^summac]     Yields a score between -1 and 1, representing the difference between entailment probability and contradiction probability (-1: the headline is not entailed in text and is completely contradicted by it, 1: the headline is fully entailed in text and not contradicted by it).    Parameters:   - `model_name`: [vitc](https://huggingface.co/tals/albert-xlarge-vitaminc-mnli) - **QAFactEval** [^qafacteval]     Using Lerc Quip score, which is reported to perform best in the corresponding paper. The score yields a value between 0 and 5 representing the overlap between answers based on the headline and text to questions generated from the headline (0: no overlap, 5: perfect overlap).    Parameters:   - `use_lerc_quip`: True  - **DAE (dependency arc entailment)** [^dae]     Yields a binary value of either 0 or 1, representing whether all dependency arcs in the headline are entailed in the text (0: at least one dependency arc is not entailed, 1: all dependency arcs are entailed).    Parameters:   - model checkpoint: DAE_xsum_human_best_ckpt   - `model_type`: model_type   - `max_seq_length`: 512   Each metric is calculated for all article-headline pairs in the test set and the respective mean score over the test set is reported.  | model | SummacCZ | QAFactEval | DAE |  |-|-|-|-| | [T-Systems-onsite/mt5-small-sum-de-en-v2](https://huggingface.co/T-Systems-onsite/mt5-small-sum-de-en-v2) | 0.6969 | 3.3023 | 0.8292 | | aiautomationlab/german-news-title-gen-mt5 | 0.4419 | 1.9265 | 0.7438 |  It can be observed that our model scores consistently lower than the T-Systems one. Following human evaluation, it seems that to match the structure and style specific to headlines the headline generation model has to be more abstractive than a model for summarization which leads to a higher frequency of hallucinations in the generated output.  