  Evaluating large summarization corpora using humans has proven to be
expensive from both the organizational and the financial perspective.
Therefore, many automatic evaluation metrics have been developed to measure the
summarization quality in a fast and reproducible way. However, most of the
metrics still rely on humans and need gold standard summaries generated by
linguistic experts. Since BLANC does not require golden summaries and
supposedly can use any underlying language model, we consider its application
to the evaluation of summarization in German. This work demonstrates how to
adjust the BLANC metric to a language other than English. We compare BLANC
scores with the crowd and expert ratings, as well as with commonly used
automatic metrics on a German summarization data set. Our results show that
BLANC in German is especially good in evaluating informativeness.

Computation and Language