2

AI21 Summarize API

TECHNICAL EVALUATION

3 Results

3.1 Human evaluation

Figure 1 shows the pass rate for both human evaluation experiments, comparing AI21 Summarize API to davinci-003
with Simple and Detailed prompting. AI21 Summarize API outperforms davinci-003 in both experiments.
Table 1 shows the distribution of labels for each model in both experiments. In both experiments, AI21 Summarize API
had signiﬁcantly fewer Very Bad summaries compared to davinci-003 (p = .05 and p = .008 respectively). Very Bad
labels correspond to completely unreliable summaries that contradict the source text or deviate from it substantially (see
Appendix B).

Figure 1: Human evaluation pass rates on real-world data. Left - Experiment #1 comparing AI21 Summarize API
versus OpenAI davinci-003 with a simple prompt. Right - Experiment #2 comparing AI21 Summarize API versus
OpenAI davinci-003 with a detailed prompt.

Experiment

Model / Method

AI21 Summarize API

davinci-003 Simple Prompt

Okay
AI21 SUMMARIZE API- TECHNICAL EVALUATION