\section{Introduction}
Fine-tuning pre-trained models on domain-specific datasets has been the leading paradigm in text summarization research in recent years \cite{lewis2020bart, zhang2020pegasus, raffel2020exploring}. These models generate high-quality summaries on standard benchmarks, but still require sizeable training datasets to adapt to new settings, e.g., summarizing data from a new source domain or producing a summary in a different style.
The success of prompt-based models (GPT-3 \cite{brown2020language}, T0 \cite{sanh2022multitask}, PaLM \cite{chowdhery2022palm}, etc.) provides an alternative approach, namely learning from natural language task instructions and/or a few demonstrative examples in the context without updating model parameters. While recent work \cite{zhao2021calibrate, min2022rethinking, ye2022unreliability} has analyzed and evaluated this paradigm across a number of tasks, it has only been studied for text summarization with unreliable automatic metrics \cite{he2022z, chowdhery2022palm, ouyang2022training} or in non-standard settings \cite{saunders2022self}.

\begin{figure}
    \centering
    \includegraphics[scale=0.236, trim=50mm 95mm 70mm 30mm, clip]{length-keyword-constraint.pdf}
    \caption{Examples of zero-shot summarization using GPT-3. We can generate summaries following style constraints or queries included in the prompts, allowing us to emulate a range of existing fine-tuned systems.}
    \label{fig:length-keyword}
\end{figure}

In this paper, we conduct the first systematic study of the impact of prompt-based models on the text summarization research space, using an Instruct-tuned 175B GPT-3 model (text-davinci-002) \cite{brown2020language, ouyang2022training} as a case study. Figure~\ref{fig:length-keyword} shows that GPT-3 summaries are extremely high-quality and adaptable to different summarization settings. Starting from these observations, we aim to answer three main questions. First, how do prompt-based GPT-3 summaries compare to those obtained from state-of-the-art fine-tuned summarization models \cite{zhang2020pegasus, liu2022brio}?
We compare these approaches using A/B testing on a new corpus of recent news articles, and find that our study participants overwhelmingly prefer zero-shot GPT-3 summaries across two different ``styles'' with different prompts (three-sentence and single-sentence).
Moreover, these zero-shot summaries do not suffer from limitations due to low-quality training data that plague fine-tuned generic summarization models \cite{maynez2020faithfulness, goyal2022training}.

Second, are existing automatic metrics well-suited to evaluating zero-shot summaries? Recent work has shown that classic reference-based such as \texttt{ROUGE} \cite{lin2004rouge} and \texttt{BERTScore} \cite{bertscore2020} are unreliable when small improvements are reported \cite{peyrard2019studying, fabbri2021summeval}; however large differences, on the order of say $5$ \textsc{Rouge} points or greater, are considered to be correlated with human preferences \cite{bhandari2020metrics, deutsch-etal-2022-examining}. However, we find that the same is no longer true when evaluating zero-shot GPT-3 summaries. These summaries score much lower on automatic metrics ($7$ \texttt{ROUGE-L} points on average) than all prior state-of-the-art models while comfortably outperforming them on human evaluation. Furthermore, we show that recent reference-free metrics, e.g. QA-based metrics \cite{fabbri-etal-2022-qafacteval, durmus2020feqa} and trained factuality models \cite{kryscinski2020evaluating, goyal2020evaluating}, similarly fail to adapt to this shift from the fine-tuned to zero-shot summarization space and need to be re-visited.

Finally, how can zero-shot summaries be used for use cases beyond generic summarization? We focus on keyword-based and aspect-based summarization. For keyword-based summarization, we find that zero-shot GPT-3 consistently generates more coherent and keyword-relevant summaries compared to current fine-tuned alternatives: crowd annotators prefer GPT-3 summaries over a baseline model \cite{he2020ctrlsum} 70\% of the time. We observe mixed results for the aspect-based setting, where GPT-3 summaries show frequent failure cases when prompted with simple prompts for aspect-based summarization

Taken together, this evidence suggests that GPT-3 represents a fundamental paradigm shift in summarization, changing the what data we need (or don't need) and what approaches we can now explore. Evaluating these systems as they progress further will require a new framework distinct from the automatic metrics that have dominated the last decade of summarization research.

\section{Models and Setup}
\label{sec:background}

\subsection{Current Paradigms for Summarization}

Recent zero- and few-shot prompting based models \cite{brown2020language, sanh2022multitask}, have shown impressive generalization capabilities on unseen tasks specified using prompts alone and without performing any gradient updates \cite{mishra2022cross}. In this work, we want to compare their text summarization performance against the current state-of-the-art models.

\begin{figure}[h]
    \centering
    \includegraphics[scale=0.23, trim=25mm 190mm 40mm 40mm, clip]{landscape.pdf}
    \caption{Broad categorization of available summarization systems; those compared in this work are highlighted in red.}
    \label{fig:landscape}
\end{figure}

Figure \ref{fig:landscape} shows the broad categories of all available summarization approaches, including current SOTA models and prompting-based models. The former set consists of \colorbox[HTML]{F7EBCC}{\textbf{fine-tuned}} language models, trained on a large number of article-summary pairs (e.g. BART \cite{lewis2020bart}, PEGASUS \cite{zhang2020pegasus}, BRIO \cite{liu2022brio}) to obtain dataset-specific systems. This category also includes models aimed at tasks beyond generic summarization, such as keyword- or query-based summarization, that still rely on standard datasets for training \cite{he2020ctrlsum}.

\begin{table}[t]
    \centering
    \small

    \begin{tabular}{c|cc|cc}
    \toprule
     \multirow{2}{*}{\textbf{Dataset}}   & \multicolumn{2}{c|}{\textbf{Avg. Words}} & \multicolumn{2}{c}{\textbf{\% novel n-grams}}  \\
        & Article & Summ & $n=1$ & $n=2$ \\ \midrule
        \textbf{\texttt{CNN}} & 760.5 & 45.7 & 16.7 & 54.3 \\
        \textbf{\texttt{DailyMail}} & 653.3 & 54.6 & 17.0 & 53.8 \\
        \textbf{\texttt{XSum (BBC)}} & 431.1 & 23.2 & 35.7 & 82.4 \\

        \textbf{\texttt{Newsroom}} & 658.6 & 26.7 & 18.9 & 47.5\\
    \bottomrule
    \end{tabular}
    \caption{Basic statistics of standard summarization datasets: \texttt{CNN/DM}~\cite{hermann2015teaching, nallapati2016abstractive}, \texttt{XSum} \cite{narayan2018don},

    Newsroom \cite{grusky2018newsroom}. These show large variance in their summary properties and fundamentally differ in their definition of the ``gold'' standard. }
    \label{tab:news_dataset_statistics}
\end{table}

On the other extreme are \colorbox[HTML]{D3DEE1}{\textbf{zero- or few-shot}} models, (e.g. GPT3 \cite{brown2020language}, PaLM \cite{chowdhery2022palm}), that are not explicitly trained for any particular task, as discussed above. Recent work \cite{ouyang2022training, wei2022finetuned, sanh2022multitask} has improved on these models by introducing \colorbox[HTML]{DCE5D2}{\textbf{instruction-tuned}} models. Here, pre-trained language models are fine-tuned on multiple tasks (which may include summarization) using instruction templates in order to align their training with inference time usage.

In this work, we compare the summarization performance of three  models that are representative of this space of options:
\begin{enumerate}[leftmargin=*]
    \item \texttt{\textbf{OpenAI's text-davinci-002}}, a GPT-3 model \cite{brown2020language} from the Instruct series \cite{ouyang2022training}. While we do not know the exact training details for this release of the model, the previous in the series (text-davinci-001) was fine-tuned on a combination of prompts submitted to their API and labeler prompts spanning multiple tasks. These tasks include summarization but not (to our knowledge) standard summarization datasets like \texttt{CNN/DM}~\cite{hermann2015teaching, nallapati2016abstractive} or \texttt{XSum} \cite{narayan2018don}.

    We choose the text-davinci-002 version for our experiments in order to benchmark the best available prompt-based model.\footnote{We did not observe obvious quality differences in generated summaries between text-davinci-001 and text-davinci-002. Examples are included in Appendix~\ref{app:d1vsd2}.} We refer to this approach as \texttt{GPT3-D2}{}.

    \item \textbf{\texttt{BRIO}} \cite{liu2022brio}, a fine-tuned summarization model that reports state-of-the art results on both \texttt{CNN/DM}~and \texttt{XSum}. We will use versions of this model fine-tuned on each of these two datasets.
    \item \textbf{\texttt{T0}}~\cite{sanh2022multitask}, a prompt-based model fine-tuned on multiple tasks including standard summarization datasets. This provides a useful point of comparison between task-specific fine-tuned (\texttt{BRIO}) and zero-shot models (\texttt{GPT3-D2}).
\end{enumerate}

\subsection{Using \texttt{GPT3-D2}~for zero-shot summarization } Fine-tuned models largely follow the ``style'' of reference summaries in their training data, and hence, generated summaries show large variance between datasets (see Table \ref{tab:news_dataset_statistics} for basic summary statistics of standard summarization datasets). To ensure fair comparison between these and zero-shot \texttt{GPT3-D2}{}, we adapt the latter's prompt to align with dataset-specific styles.

\begin{figure}[t]
    \centering
    \includegraphics[scale=0.26, trim=60mm 110mm 25mm 50mm, clip]{length-constraints.pdf}
    \caption{Illustration of length control using the task description / prompt for \texttt{GPT3-D2}. We found that the generated summaries followed the given sentence length constraint 98\% of the time, allowing us to generate different length summaries emulating different datasets. }
    \label{fig:length_ex}
\end{figure}

Specifically, we follow prior work \cite{sanh2022multitask} and use sentence-count length prompts to adapt to each dataset. Although these datasets also differ along other attributes, e.g. \texttt{CNN/DM}~is lead-biased whereas \texttt{XSum} requires inferences drawing from a whole article, we do not attempt to control any other attributed of the summary. Figure~\ref{fig:length_ex} shows an example of different length \texttt{GPT3-D2}~summaries for the same news article, using the following prompt format:\footnote{Although prior work has shown that few-shot learning works better than zero-shot across tasks like inference, textual reasoning, and translation \cite{brown2020language, chowdhery2022palm, ouyang2022training, sanh2022multitask}, we opt for zero-shot prompting as news articles are generally longer and multiple demonstration may not fit within the context.}

\paragraph{}\textit{Article: \{\{article\}\}}

\textit{Summarize the above article in {{N}} sentences.}

\paragraph{}We found that \texttt{GPT3-D2}~summaries faithfully follow the given length constraint in 98\% of the test instances used in our human study data in Section~\ref{sec:human-study}.

Given this setup, we first compare the summary quality of the three summarization models through a human annotation study (Section \ref{sec:human-study}). Then, we evaluate the current suite of summarization metrics for zero-shot summarization (Section \ref{sec:automatic-eval}). Finally, in Section \ref{sec:keyword}, we briefly discuss \texttt{GPT3-D2}~performance on summarization tasks beyond generic summarization and new challenges.

\begin{figure*}[t]
    \centering
    \includegraphics[width=\textwidth, trim=10mm 50mm 10mm 10mm, clip]{qualitative_ex.pdf}
    \caption{Examples of CNN-style and BBC/XSum-style summaries for the three systems. For CNN, we observe that models fine-tuned on the \texttt{CNN/DM}~training set reflect its dataset biases; summaries are highly extractive, specific and lead-biased. On the other hand, \texttt{GPT3-D2}~summaries contain fewer specific details but cover more content.}
    \label{fig:qualitative_ex}
\end{figure*}

\section{Human evaluation of \texttt{GPT3-D2}{} summaries}
\label{sec:human-study}

Generated summaries of fine-tuned models \cite{lewis2020bart, zhang2020pegasus, liu2022brio} emulate gold-standard summaries in their training datasets. In contrast, prompt-based \texttt{GPT3-D2}~models generate summaries based on how the given task description surfaces behavior learned during pre-training or instruction-tuning. In this section, we ask: how do these paradigms compare? Does learning from gold summaries lead to a better summarization model? To answer this, we conduct a human study to compare  outputs of our 3 representative models and collect human preferences of quality.

\subsection{Experimental Setup}
\paragraph{Datasets for fine-tuning} We choose two standard fine-tuning datasets whose summaries differ along multiple dimensions such as length and abstractiveness:

\begin{enumerate}[leftmargin=*]
    \item \textbf{CNN/DM} \cite{hermann2015teaching, nallapati2016abstractive} contains reference summaries that are approximately 3-4 sentences long. Summaries in this dataset are highly extractive and lead-biased.
    \item \textbf{XSum} \cite{narayan2018don} contains 1 sentence summaries of BBC news articles. In this dataset, references summaries, and consequently generated summaries from fine-tuned models are highly abstractive.
\end{enumerate}

\paragraph{Datasets for evaluation} Because \texttt{GPT3-D2}{}'s pre-training and instruction-tuning datasets are unknown, it may have been trained on existing articles and summaries in the test splits of these standard benchmarks.
We therefore run our human study on 100 recent articles from CNN\footnote{Although the \texttt{BRIO}'s \texttt{CNN/DM}~model also includes DailyMail data in its training, we do not use this news source in our study as it is now widely considered to be unreliable. E.g. according to Media Bias / Fact Check site, \texttt{DM}'s factual reporting is rated `low' \url{https://mediabiasfactcheck.com/daily-mail/}. } and BBC, collected between March 1, 2022 and June 31, 2022. We call these CNN-2022 and BBC-2022 respectively.

\paragraph{Model details} We use the publicly released \texttt{BRIO-XSum} and \texttt{BRIO-CNN/DM} models to generate summaries.\footnote{Models at: \url{https://github.com/yixinL7/BRIO}} For \texttt{T0}, we use a prompt we selected from its prompt repository for \texttt{CNN/DM}~and \texttt{XSum} datasets.\footnote{Repository with T0 prompts: \url{https://github.com/bigscience-workshop/promptsource}} Finally, to generate \texttt{GPT3-D2}~summaries, we set $N=3$ for CNN and $N=1$ for BBC in our standard sentence-count prompt template from Section~\ref{sec:background}.

For a maximally fair comparison in this ``realistic'' setting, we take some additional steps to improve the output of \texttt{BRIO-XSum}. In order to automate dataset creation, \texttt{XSum} removes the first sentence from news articles to use as the gold summary for training, then treats the rest of the sentences as the article to summarize. This setup differs from the real world usage of summarization systems where the complete article is summarized. Due to this mismatch, \texttt{BRIO-XSum} often generates very low quality outputs, e.g. \textit{All images: Strule Shared Education Campus} in Figure \ref{fig:qualitative_ex}, for around 30\% of the articles. We manually identify these examples and first attempt to fix them by selecting a summary without such obvious failures from further down the beam (we use beam size $=10$). However, if we cannot find a ``better'' summary, we remove the first sentence of the article and re-sample a new summary to align with its noisy training. This latter strategy often results in factually incorrect summary generations, as is well documented in prior research \cite{maynez2020faithfulness, goyal2021annotating}.

\paragraph{Design of the human study} We design an A/B test to collect preference annotations. For each given article, annotators are shown summaries from all three summarization systems (\texttt{BRIO}, \texttt{T0}~and \texttt{GPT3-D2}{}). They are then asked to select their most and least preferred summary or summaries. In addition to these multiple choice questions, we also ask for a free-text justification of both choices.

We make two design decisions for our human study: first, we do not provide annotators with specific definitions of summary quality to avoid introducing our own biases. It is also quite challenging to produce a unified definition of quality for the very different ``styles'' of summaries evaluated in this study.  Instead, we ask them to rely on their own preferences based on summaries they would like to see if they were browsing the web, which we believe to be a representative scenario for non-expert consumers of news summaries. Detailed task instructions are included in Appendix~\ref{app:instructions}.

Second, we allow multiple selections for both the best and worst summary questions to cater to scenarios in which different summarization systems output similar quality summaries without meaningful differences.

We hire crowd annotators through Prolific. For both CNN and BBC, we recruit 60 unique participants to annotate the 100 summaries in each dataset. Each annotator was asked to annotate 5 articles and each article was annotated by 3 annotators. Additionally, we use the Prolific's demographic filters to restrict participation to USA (or UK) residents for CNN (or BBC). We anticipate that residents from these respective countries are better positioned to understand country-specific news events and evaluate their summaries. Participants were paid approximately \$11/hr for their work.

\begin{table}[t]
    \centering
    \small
    \renewcommand{\tabcolsep}{1mm}
    \begin{tabular}{r|cc|cc|c}
    \toprule
      \multirow{2}{*}{\textbf{Model}}  & \multicolumn{2}{c|}{\textbf{Length Statistics}}  & \multicolumn{2}{c|}{\textbf{\% novel n-gms}} & \textbf{\#NEs per} \\
       & \#sent & \#words/sent & $n=1$ & $n=2$ & \textbf{100 words} \\ \midrule
       \multicolumn{6}{c}{\textbf{CNN}} \\ \midrule
       \texttt{BRIO}{} & 3.7 & 15.8 & 12.1 & 36.2 & 12.9 \\
       \texttt{T0}{} & 2.7 & 14.9 & 16.4 & 45.2 & 12.8 \\
       \texttt{GPT3-D2}{} & 2.9 & 23.4 & 16.3 & 40.7 & 10.5 \\ \midrule
       \multicolumn{6}{c}{\textbf{BBC}} \\ \midrule
       \texttt{BRIO}{} & 1.0 & 20.2 & 24.6 & 61.2 & 9.1\\
       \texttt{T0}{} & 1.0 & 20.0 & 26.3 & 66.7 & 9.8 \\
       \texttt{GPT3-D2}{} & 1.0 & 27.7 & 16.4 & 42.3 & 8.5 \\
    \bottomrule
    \end{tabular}
    \caption{Statistics for generated summaries evaluated in the human study across all  datasets and summarization systems. We observe that \texttt{GPT3-D2}{} generated summaries nearly always follow the sentence length constraints in their prompts.}
    \label{tab:quantitative}
\end{table}

\subsection{Results}

\paragraph{Differences between summarization systems}
Figure \ref{fig:qualitative_ex} shows examples of generated summaries from all three summarization systems for both CNN and BBC articles. For CNN, we observe that fine-tuned \texttt{BRIO}~summaries tend to be highly extractive and generally include a high number of named entities (dates, percentages, names), reflecting the data it was trained on. In contrast, \texttt{GPT3-D2}~summaries are more abstractive and less specific, but provide a more exhaustive overview of the article content. Table~\ref{tab:quantitative} provides quantitative evidence of this; we use percentage of novel n-grams to measure abstractiveness, and number of named entities per 100 words to measure specificity.

For BBC, we observe inverse trends where \texttt{BRIO}~and \texttt{T0}~are more abstractive compared to \texttt{GPT3-D2}. Again, this can be attributed to the \texttt{XSum} training data used to train both these prior models. For \texttt{GPT3-D2}~summaries, on the other hand, the level of abstractiveness does not differ between datasets. Finally, Table~\ref{tab:quantitative} shows that \texttt{GPT3-D2}~summaries tend to have longer sentences, and therefore similar number of summary sentences often results in a longer summary for both datasets. We study the effect of this length difference on human preference judgments in Appendix~\ref{app:length}.

\begin{table}[t]
\renewcommand{\tabcolsep}{1mm}
    \centering
    \small
    \begin{tabular}{c|cc|cc|cc}
    \toprule
        \multirow{2}{*}{\textbf{Dataset}} & \multicolumn{2}{c|}{\textsc{\textbf{Brio}}} & \multicolumn{2}{c|}{\textsc{\textbf{T0}}} & \multicolumn{2}{c}{\textsc{\textbf{GPT3}}} \\
        & Best $\uparrow$ & Worst $\downarrow$ & Best $\uparrow$ & Worst $\downarrow$ & Best $\uparrow$ & Worst $\downarrow$ \\ \midrule
        \textsc{\textbf{CNN}} & 36 & 24 & \cellcolor{red!20}8 & \cellcolor{red!20}67 & \cellcolor{green!20} 58 &  \cellcolor{green!20}9 \\
        \textsc{\textbf{BBC}} & \cellcolor{red!20}20 & \cellcolor{red!20}56 & 30 & 29 & \cellcolor{green!20} 57 &  \cellcolor{green!20}15 \\
    \bottomrule
    \end{tabular}
    \caption{Percentage of times a summarization system is selected as the best or worst according to majority vote (may be tied). Human annotators have a clear preference for zero-shot \texttt{GPT3-D2}{} for both CNN and BBC style summaries.}
    \label{tab:human-study-vanilla}
\end{table}

\paragraph{Which systems do humans prefer?} Results of our human study are summarized in Table \ref{tab:human-study-vanilla}. We report the percentage of times a particular system is the most/least preferred model according to majority vote combining all three annotator's choices.\footnote{As we allow multiple system selections, note that more that one system could be the majority. However, this is rare after majority vote: only 2\% of the articles in CNN and 7\% in BBC have multiple best summaries.}
Across both datasets and styles, we observe a clear preference for \texttt{GPT3-D2}~summaries compared to the other two models. In fact, in both scenarios, the \texttt{GPT3-D2}~outperforms the next best model by at least 20 percentage points. This improvement is statistically significant according to a paired bootstrap test (CNN $p-$value $= 2 \times 10^{-3}$, BBC $p-$value $= 6 \times 10^{-4}$).

Note that the next best model differs between the two datasets. For BBC, annotators prefer \texttt{T0}~summaries over \texttt{BRIO}. Annotator rationales often mentioned misleading or incorrect information as the primarily reason for selecting \texttt{BRIO}~as the worst summary, confirming the issues that have been observed with \texttt{XSum}-trained models \cite{maynez2020faithfulness, pagnoni2021understanding, goyal2021annotating}. Although \texttt{T0}~also includes \texttt{XSum} training data, we hypothesize that its multi-task framework helps offset the noisy signal from \texttt{XSum}.

In contrast, annotators rate \texttt{T0}~as the worst summarization system for CNN. The most common rationales for these were shorter length and inclusion of irrelevant details, e.g. long quotes, while missing key points. Some annotators also commented that these  \texttt{T0}~summaries were less coherent compared to the other models. Interestingly, we did not observe similar complaints for the single-sentence \texttt{T0}{} summaries for BBC.

\begin{figure}[t]
    \centering
    \includegraphics[scale=0.18, trim=48mm 80mm 40mm 20mm, clip]{basic_humanstudy_votes.pdf}
    \caption{Annotator vote distribution for best and worst summaries across all datasets and models. Although \texttt{GPT3-D2}~is the clear winner according to majority vote, this choice is unanimous for less than 30\% of the articles. This demonstrates the inherent variance in different annotators' definitions of ``best summary'', especially when comparing high-quality summaries from strong models. }
    \label{fig:vote_share}
\end{figure}

\paragraph{Do annotators agree with each other?} To study this, we plot the distribution of annotator votes for each summarization system and dataset in Figure~\ref{fig:vote_share}. Additionally, we report the inter-annotator agreement, measured using Krippendorff's alpha with MASI distance \cite{passonneau2006measuring}, to account for multiple selections of best or worst summary allowed in our study design.

The vote distribution shows that although more annotators prefer \texttt{GPT3-D2}~summaries, this choice is only unanimous, i.e. supported by all three annotators, for less that 30\% of the annotated articles. Conversely, although \texttt{BRIO}~(or \texttt{T0}) summaries are less preferred than \texttt{GPT3-D2}~for the CNN (or BBC) dataset on aggregate, they were voted as the best summary by at least one annotator for more than 60\% of the articles. This demonstrate two things: first, when comparing summaries from two strong models, the choice is inherently ambiguous (similar observations in \citet{clark2021all}). Second, these results and the diversity in the written rationales, show that there does not exist a universal definition of a ``good'' summary and that different summary properties appeal to different annotators. Regardless, the aggregate preference for \texttt{GPT3-D2}{} is high enough across the board to give us confidence in its strength.

\begin{table*}[t]
    \centering
    \small
    \begin{tabular}{c|c|ccc|cc|cc}
    \toprule
       \multirow{2}{*}{\textbf{Dataset}} & \multirow{2}{*}{\textbf{Model}} & \multicolumn{3}{c|}{\textbf{Overlap-Based}} & \multicolumn{2}{c|}{\textbf{Similarity-Based}} & \multicolumn{2}{c}{\textbf{QAEval}} \\
        & & \textbf{\textsc{ROUGE(1/2/L)}} & \textsc{\textbf{Meteor}} & \textsc{\textbf{Bleu}} & \textbf{BERTScore} & \textbf{MoverScore} & \textbf{EM} & \textbf{F1} \\ \midrule
        \multirow{4}{*}{\texttt{\textbf{CNN}}}
        & \texttt{PEGASUS} & 34.85/14.62/28.23 & .24 & \cellcolor{green!20}{7.1} & \cellcolor{red!20}.858 & .229 & .105 & .160 \\
        & \texttt{BRIO}{}  & \cellcolor{green!20}38.49/17.08/31.44 & \cellcolor{green!20}{.31} & 6.6 & \cellcolor{green!20}{.864} & \cellcolor{green!20}.261 & \cellcolor{green!20}.137 & \cellcolor{green!20}.211 \\
        & \texttt{T0}{}  & 35.06/13.84/28.46 & .25 & 5.9 & .859 & .238  & .099 & .163 \\
        & \texttt{GPT3-D2}{}   & \cellcolor{red!20} 31.86/11.31/24.71 & \cellcolor{red!20}.25 & \cellcolor{red!20} 3.8 & \cellcolor{red!20}.858 & \cellcolor{red!20}.216 & \cellcolor{red!20}.098 & \cellcolor{red!20}.159 \\\midrule
        \multirow{4}{*}{\texttt{\textbf{DailyMail}}}
        & \texttt{PEGASUS} & 45.77/23.00/36.65 & .33 & \cellcolor{green!20}12.2 & .865 & .308 & .159 & .229 \\
        & \texttt{BRIO}{}  & \cellcolor{green!20}{49.27/24.76/39.21} & \cellcolor{green!20}{.37} & 11.7 & \cellcolor{green!20}.871 & \cellcolor{green!20}.331 & \cellcolor{green!20}.175 & \cellcolor{green!20}.259 \\
        & \texttt{T0}{} & 42.97/19.04/33.95 & .28 & 8.9 & .863 & .290 & .121 & .184 \\
        & \texttt{GPT3-D2}{}  & \cellcolor{red!20} 38.68/14.24/28.08 & \cellcolor{red!20}.26 & \cellcolor{red!20}6.6 & \cellcolor{red!20}.859 & \cellcolor{red!20}.248 & \cellcolor{red!20}.101 & \cellcolor{red!20}.159 \\\midrule
        \multirow{4}{*}{\texttt{\textbf{XSum}}}
        & \texttt{PEGASUS} & 47.97/24.82/39.63 & .36 & 9.8 & \cellcolor{green!20}{.901} & .362 & \cellcolor{green!20}.145 & .221 \\
        & \texttt{BRIO}{}  & \cellcolor{green!20}{49.66/25.97/41.04} & \cellcolor{green!20}{.39} & \cellcolor{green!20}{10.6} & \cellcolor{green!20}{.901} & \cellcolor{green!20}.372 & .139 & \cellcolor{green!20}.224 \\
        & \texttt{T0}{}  &  44.20/20.72/35.84 & .34 & 8.0 & .896 & .340  & .125 & .208\\
        & \texttt{GPT3-D2}{}   & \cellcolor{red!20}28.78/7.64/20.60 & \cellcolor{red!20}.19 & \cellcolor{red!20}2.2 & \cellcolor{red!20}.869 & \cellcolor{red!20}.197 & \cellcolor{red!20}.066 & \cellcolor{red!20}.119 \\\midrule
        \multirow{4}{*}{\texttt{\textbf{Newsroom}}}
        & \texttt{PEGASUS} &  \cellcolor{green!20}39.21/27.73/35.68 &  \cellcolor{green!20}.39 &  \cellcolor{green!20}.14 & \cellcolor{green!20}.873 & \cellcolor{green!20}.272 & \cellcolor{green!20}0.182 & \cellcolor{green!20}0.253 \\
        & \texttt{BRIO}{} & - & - & - & - & - & - & - \\
        & \texttt{T0}{}  & \cellcolor{red!20}25.64/9.49/21.41 & \cellcolor{red!20}.20 & \cellcolor{red!20}.04 & \cellcolor{red!20}.849 & \cellcolor{red!20}.145 & \cellcolor{red!20}.080 & \cellcolor{red!20}0.125 \\
        & \texttt{GPT3-D2}{}   & 27.44/10.67/22.18 & .22 & .05 & .859 & .159 & .089 & 0.142 \\
    \bottomrule
    \end{tabular}
    \caption{Performance of different summarization systems measured using reference-based automatic metrics. Across all datasets, we observe that automatic metrics report substantially worse results for \texttt{GPT3-D2}~summaries compared to fine-tuned models. This directly contradicts the human preference results from Section \ref{sec:human-study}, demonstrating that these reference-based metrics cannot reliably compare the quality of zero-shot summaries against fine-tuned summaries.}
    \label{tab:reference-based-automatic}
\end{table*}

\paragraph{How do these results impact the field?} Progress in text summarization research in the last five years has been enabled by the construction of large-scale text summarization datasets that involved scraping news articles and pairing them with any available summary-like data \cite{hermann2015teaching, narayan2018don, grusky2018newsroom}. The \texttt{CNN/DM}~dataset considers bullet points accompanying news articles as its summary. These ``gold'' standard summaries provided useful training signal to train impressive supervised models \cite{lewis2020bart, zhang2020pegasus, liu2022brio} and hence, their quality or alignment with human preferences was largely ignored.

We found that, despite its popularity, \texttt{XSum} is largely unsuitable for fine-tuning models like \texttt{BRIO}{} for realistic summarization settings.
Even though a \texttt{CNN/DM}-trained \texttt{BRIO}{} model performed better, the results of our human study question the continued utility of hill-climbing on this dataset, as it seems users may simply prefer a different style of summary altogether. In fact, this preference for \texttt{GPT3-D2}{} is much larger than incremental improvements reported in other human evaluation settings, e.g. improvements on XSum on the \texttt{GENIE} leaderboard \cite{Khashabi2021GENIEAL}. Furthermore, as we we will see in Section~\ref{sec:keyword}, the greater flexibility of \texttt{GPT3-D2}{} compared to these systems makes it more suitable for news summarization tasks beyond generic summarization.

If a system designer collects a large-scale dataset of high-quality summaries that they wish to emulate, we believe a fine-tuned system may outperform \texttt{GPT3-D2}{}. However, better-trained models on datasets collected via ``incidental'' supervision are less likely to help.

\section{Can current automatic metrics evaluate \texttt{GPT3-D2}~summaries?}
\label{sec:automatic-eval}

Automatic metrics proposed for summarization evaluation can be broadly divided into two categories: (1) \textbf{reference-based}, that compare generated summaries against available gold summaries, and (2) \textbf{reference-free} that only rely on the input document. Here, we compare their performance at evaluating zero-shot \texttt{GPT3-D2}~summaries.

\begin{table*}[t]
    \centering
    \small
    \begin{tabular}{c|c|cc|cc|ccc}
    \toprule
        \multirow{2}{*}{\textbf{Dataset}} & \multirow{2}{*}{\textbf{Model}} & \multicolumn{2}{c|}{\textbf{Overall Quality}} & \multicolumn{2}{c|}{\textbf{Factuality (QA-based)}} & \multicolumn{3}{c}{\textbf{Factuality (NLI-based)}} \\
         & & \textbf{SUPERT} & \textbf{BLANC} & \textbf{QuestEval} & \textbf{QAFactEval} & \textbf{FactCC} & \textbf{DAE} & \textbf{SummaC} \\ \midrule
        \multirow{4}{*}{\texttt{\textbf{CNN}}}
        & \texttt{PEGASUS} & .5466 & .0605 & .7373 & \cellcolor{green!20}4.4071 & \cellcolor{green!20}.3743 & \cellcolor{green!20}.8223 & \cellcolor{green!20}.1138 \\
        & \texttt{BRIO}{}  & \cellcolor{green!20}.5586 & \cellcolor{green!20}.0802 & .7334 & 3.8332 & \cellcolor{red!20}.1817 &  .7577 & -.0532 \\
        & \texttt{T0}{}  & \cellcolor{red!20}.5330 & \cellcolor{red!20}.0558 & \cellcolor{green!20}.7799 & 3.7517 & .2012 & .7556 & -.0605 \\
        & \texttt{GPT3-D2}{}   & .5560 & .0749 & \cellcolor{red!20}.7249 & \cellcolor{red!20}3.6399 & .2428 & \cellcolor{red!20}.6671 & \cellcolor{red!20}-.0729 \\\midrule
        \multirow{4}{*}{\texttt{\textbf{DailyMail}}}
        & \texttt{PEGASUS} & \cellcolor{green!20}.6433 & .1137 & .7536 & \cellcolor{green!20}4.4677 & \cellcolor{green!20}.5152 & \cellcolor{green!20}.8497 & \cellcolor{green!20}.2402 \\
        & \texttt{BRIO}{}  & .6360 & \cellcolor{green!20}.1217 & \cellcolor{red!20}.7415 & 4.1362 & .3699 & .8118 & \cellcolor{red!20}.0153\\
        & \texttt{T0}{}  & \cellcolor{red!20}.5995 & \cellcolor{red!20}.0889 & \cellcolor{green!20}.7803 & 3.9827 & \cellcolor{red!20}.2431 & .8043 & .0478\\
        & \texttt{GPT3-D2}{}   & .6118 & .0983 & .7461 & \cellcolor{red!20}3.8279 & .2697 & \cellcolor{red!20}.6990 & .0365 \\\midrule
        \multirow{4}{*}{\texttt{\textbf{XSum}}}
        & \texttt{PEGASUS} & \cellcolor{red!20}.4439 & .0249 & .8233  & 2.0089 & .2465 & \cellcolor{red!20}.3598 & -.2993\\
        & \texttt{BRIO}{} & .4459 & \cellcolor{red!20}.0230 & \cellcolor{green!20}.8305  & \cellcolor{red!20}1.8626 & \cellcolor{red!20}.2031 & .3040 & \cellcolor{red!20}-.3292\\
        & \texttt{T0}{}  & .4538 & .0238 & \cellcolor{red!20}.7957  & 2.0330 & .2219 & .3392 & -.3037\\
        & \texttt{GPT3-D2}{}   & \cellcolor{green!20}.5060 & \cellcolor{green!20}.0594 & .8064  & \cellcolor{green!20}2.9492 & \cellcolor{green!20}.3977 & \cellcolor{green!20}.6372 & \cellcolor{green!20}-.2626\\\midrule
        \multirow{4}{*}{\texttt{\textbf{Newsroom}}}
        & \texttt{PEGASUS} & \cellcolor{green!20}.6286 & \cellcolor{green!20}.1131 & \cellcolor{red!20}.7118  & \cellcolor{green!20}4.2120 & \cellcolor{green!20}.7218 & \cellcolor{green!20}.7956 & \cellcolor{green!20}.2418 \\
        & \texttt{BRIO}{}  & -&-&-&-&-&-&- \\
        & \texttt{T0}{}  & .5433 & .0640 & \cellcolor{green!20}.7511  & 3.5799 & \cellcolor{red!20}.2828 & .7376 & .0261 \\
        & \texttt{GPT3-D2}{}   & \cellcolor{red!20}.5408 & \cellcolor{red!20}.0599 & .7160 & \cellcolor{red!20}3.2336 & .3988 & \cellcolor{red!20}.6564 & \cellcolor{red!20}-.0729\\
    \bottomrule
    \end{tabular}
    \caption{Performance of different summarization systems, as scored by automatic reference-free evaluation metrics from the summarization literature. Similar to reference-based metrics, these also generally fail to produce the same system rankings as human preferences reliably across datasets.}
    \label{tab:reference-free}
\end{table*}

\paragraph{Experimental Setup} We evaluate automatic metrics using summaries from 4 different summarization datasets, listed in Table~\ref{tab:news_dataset_statistics}. For each dataset, we construct our evaluation sets by randomly sampling 500\footnote{This size is chosen to give sufficient statistical power \cite{card2020little} while keeping costs for \texttt{GPT3-D2}{} evaluation low to enable others to compare on this subset. We outline costs in Appendix \ref{sec:cost}.} articles from the standard test split.\footnote{Note that these standard datasets were released before 2020. Therefore, it is possible that some article-summary pairs in our test set overlap with \texttt{GPT3-D2}'s training data. However, we do not observe a qualitative difference in \texttt{GPT3-D2}{}'s performance on these older articles.} We compare the same 3 summarization systems from Section \ref{sec:human-study} in our analysis. Additionally, we also report results using the fine-tuned \texttt{PEGASUS} model \cite{zhang2020pegasus}, as \texttt{BRIO}~fine-tuned models are not available for all datasets.

We publicly release this corpus of summarization outputs to standardize the test sets and support future research into \texttt{GPT3-D2}{} based summarization. Link: \url{https://tagoyal.github.io/zeroshot-news-annotations.html}.

\subsection{Reference-based metrics}
\label{sec:reference-based}

Here, we study if the gold summaries of the standard datasets are useful for evaluation, especially when evaluating zero-shot summaries that are not trained to emulate the gold. We benchmark the performance of 3 different summarization metrics: (1) \textbf{overlap-based} metrics, specifically \texttt{ROUGE} \cite{lin2004rouge} \texttt{METEOR} \cite{banerjee2005meteor} and  \texttt{BLEU} \cite{papineni2002bleu}. (2)~\textbf{similarity-based} metrics, that compute similarity between embeddings representations of generated and reference summaries. Specifically, we report \texttt{BERTScore} \cite{bertscore2020} and \texttt{MoverScore} \cite{zhao2019moverscore}. (3) a \textbf{QA-based} metric, specifically QAEval \cite{deutsch2021towards}. Although most QA-metrics are reference-free (discussed in Section \ref{sec:reference-free}), QAEval uses the reference summaries to indicate saliency. We report both exact match (EM) and F1 components of QAEval.

\paragraph{Results} Table \ref{tab:reference-based-automatic} outlines the results. It shows that \texttt{BRIO}~and \texttt{PEGASUS} models, fine-tuned to emulate the reference summaries, outperform \texttt{GPT3-D2}~summaries according to all reference-based automatic metrics. The difference in their assigned scores is very high, e.g. >7 \texttt{ROUGE-L} points between \texttt{GPT3-D2}~and \texttt{BRIO}. For comparison, these reported scores for \texttt{GPT3-D2}~are even lower than the trivial Lead-3 baseline reported in prior work \cite{fabbri2021summeval, grusky2018newsroom}. This clearly demonstrates that \textbf{current automatic reference-based metrics cannot be used to reliably measure  summary quality under the zero-shot prompting paradigm}.

Amongst prompting-based models, we observe that \texttt{T0}~summaries report better metric scores than \texttt{GPT3-D2}~for all datasets except Newsroom. Interestingly, out of the four datasets evaluated here, Newsroom is the only one not used to train the \texttt{T0}~model. This further shows that access to dataset-specific reference summaries during training improves performance according to these metrics, rendering them unsuitable for zero-shot evaluation.

\subsection{Reference-free metrics}
\label{sec:reference-free}

Next, we investigate whether current reference-free evaluation metrics reflect the human preference rankings between summarization systems, as observed in Section~\ref{sec:human-study}. Here, we study 2 categories of metrics: (1) \textbf{quality metrics}, specifically SUPERT \cite{gao2020supert}, which evaluates generated summaries against automatically identified salient sentences in the input, and BLANC \cite{vasilyev2020fill}, which evaluates summaries on language understanding tasks. We refer readers to the original papers for detailed explanation of these. (2) \textbf{factuality metrics}, that are evaluate whether generated summaries contain incorrect information with respect to the source article. We report the performance of summarization systems using two QA-based metrics: QuestEval \cite{scialom2021questeval} and QAFactEval \cite{fabbri-etal-2022-qafacteval}. Additionally, we also benchmark entailment-based metrics: FactCC \cite{kryscinski2020evaluating}, DAE \cite{goyal2020evaluating, goyal2021annotating} and SummaC \cite{laban-etal-2022-summac}.\footnote{Exact model versions and configurations used for these are outlined in Appendix \ref{app:config}.} These entailment-based models are designed for classification into factual or non-factual; therefore, we use $P(\mathrm{factual \mid article, summary})$ to score generated summaries.

\paragraph{Results} Table \ref{tab:reference-free} outlines the scores for each summarization system according to the above reference-free metrics. Ideally, we want the relative rankings of different systems according to these metrics to correspond to human preferences, i.e. \texttt{GPT3-D2}{} > \texttt{BRIO}{} > \texttt{T0}{} for \texttt{CNN/DM}\footnote{Although the human study in Section~\ref{sec:human-study} is only run on CNN articles, the underlying fine-tuned model is same for both CNN and DM. Therefore, it we can reasonably expect it to display similar quality differences with respect to \texttt{GPT3-D2}.} and \texttt{GPT3-D2}{} > \texttt{T0}{} > \texttt{BRIO}{} for \texttt{XSum}.\footnote{Note that while annotators were not explicitly asked to rate factuality, we instructed them to carefully check factuality and appropriately downvote non-factual summaries.}

Overall, we observe that none of the reference-free metrics we evaluate follow these trends for both \texttt{CNN/DM} and \texttt{XSum} datasets. In particular, we observe that \texttt{GPT3-D2}{} summaries report low factuality scores (except \texttt{XSum}) even though we rarely found any factual errors in our qualitative analysis of its generated summaries.

Interestingly, we noticed a roughly inverse relation to abstractiveness; summarization systems that generated more abstractive summaries (see Table \ref{tab:quantitative}) were generally scored lower by all automatic reference-based metrics. For instance, \texttt{GPT3-D2}{} is scored lower than \texttt{BRIO}{} by both quality metrics for all datasets except \texttt{XSum}; the latter is the only dataset for which \texttt{GPT3-D2}{} summaries are less abstractive. Such shortcomings of reference-free evaluation metrics due to spurious correlations have also been studied in prior work \cite{durmus2022spurious}. These issues become more exaggerated when the summarization systems being compared exhibit very different properties.

\paragraph{Discussion} On the surface, the failure of reference-free metrics at evaluating \texttt{GPT3-D2}{} summaries is more surprising that reference-based metrics as the later explicitly compares generated summaries with references that \texttt{GPT3-D2}{} is not trained to imitate. Therefore, \texttt{GPT3-D2}{} understandably scores lower than fine-tuned systems.

However, we note two different issues with reference-free metrics: (1) Some of these, e.g. FactCC and DAE, use reference summaries as positive examples to train the metric. Therefore, although ``reference-free'' at test time, they are still trained to reward the summary properties seen in the standard summarization benchmarks. (2) Even completely reference-free metrics, e.g. QuestEval and QAFactEval, have only been evaluated on reference-based benchmarks and fine-tuned models. Therefore, the choice of different components, such as question answering or question generation models to use, etc. has been dictated by the error space of prior fine-tuned models \cite{tang2022understanding}. These decisions also now need to be re-visited to incorporate \texttt{GPT3-D2}{} evaluation; we leave this for future work.

\section{Beyond Generic Summarization}
\label{sec:keyword}
Previously, we observed that \texttt{GPT3-D2}{} models faithfully follow simple ``style'' instructions in the given prompts. This provides a promising direction to tackle other use cases in news summarization beyond the generic summarization task from Section~\ref{sec:human-study}.

\begin{figure*}[t]
    \centering
    \includegraphics[scale=0.30, trim=5mm 40mm 0mm 0mm, clip]{keyword_aspect_combined.pdf}
    \caption{Comparison of keyword- and aspect-based summaries using \texttt{GPT3-D2}~and \texttt{CTRLSum} models. The \texttt{GPT3-D2}~prompt is shown on the left with the corresponding keyword or aspect bolded. For keyword-based summarization, the \texttt{GPT3-D2}~summary presents appropriate context before the keyword-specific information. However, for aspect-based summarization, it does not always generate factually correct summaries, as shown in the first aspect example. We observe that \texttt{CTRLSum} performs poorly for both these settings.}
    \label{fig:keyword_focused}
\end{figure*}

Different users can have very different information needs from the same article, all of which cannot be satisfied with a single generic summary. Prior work has introduced several task formulations to address this gap, including keyword-focused \cite{he2020ctrlsum}, query-focused \cite{baumel2014query, he2020ctrlsum}, or aspect-focused summarization \cite{krishna2018generating, ahuja2022aspectnews}, amongst others. Here, we evaluate \texttt{GPT3-D2}{} performance at two of these use cases.

In \textbf{keyword-based summarization}, the output summaries must succinctly summarize the input document focusing on a given keyword; these generally correspond to specific entities or events directly mentioned in the document. In contrast, the control units in \textbf{aspect-based summarization} are high-level topics that can be common across multiple similar types of documents. For e.g., for the input article in Figure~\ref{fig:length-keyword}, \textit{Donald Trump} or \textit{Russian interference in 2016 elections} are keyword controls whereas \textit{charges against the defendants} is a higher-level aspect that can serve as the query for any news article discussing a lawsuit or investigation.

\subsection{Qualitative Analysis}

\paragraph{Baseline Model for comparison} We use the recently proposed \texttt{CTRLSum} \cite{he2020ctrlsum}, a fine-tuned \texttt{BART} model, as our baseline. It can be flexibly adapted for both keyword- and aspect-based settings by including a prompt as additional input to the encoder. We use the prompt template recommended in the original paper.\footnote{Trained model publicly released at: \url{https://github.com/salesforce/ctrl-sum}.}

\paragraph{Control Units} For the keyword-focused setting, we use named entities extracted from the input article as the control units. For aspect-focused summarization, we directly use the aspects introduced in the guided summarization task from TAC 2011.\footnote{\url{https://tac.nist.gov/2011/Summarization/Guided-
Summ.2011.guidelines.html}} It defined 5 broad categories of newswire articles, such as accidents and natural disasters, investigations and trial, etc., and multiple aspects for each category. For example, the \textit{``investigations and trials''} category includes aspects such as \textit{``who is the defendant or under trial?''}, \textit{``who is investigating, prosecuting, judging?''}, and so on.

\paragraph{Qualitative Analysis} Figure~\ref{fig:keyword_focused} shows examples of keyword- and aspect-focused summaries using \texttt{GPT3-D2}~and the baseline \texttt{CTRLSum} model. The keywords or aspects are highlighted in bold within the \texttt{GPT3-D2}~prompt displayed on the left.

In this example, representative of average \texttt{GPT3-D2}{} quality, the keyword-focused \texttt{GPT3-D2}~summary first gives a brief overview of the article setting before providing keyword-relevant information. In contrast, the  \texttt{CTRLSum} summary exhibits poor discourse structure and reads like a list of facts stapled together.

The figure also shows aspect-focused summaries for two aspects associated with the ``investigations and trial'' category most appropriate for the chosen article. We see mixed results here for \texttt{GPT3-D2}; it generates a factually incorrect summary for the first aspect, listing multiple people from the input article as defendants instead of only ``Donald Trump''. For the second aspect, it correctly maps the high-level concept ``defendant'' to ``Donald Trump'' in the input article and generates the correct answer to the input query: \textit{``The defendant's reaction to charges in the above article is denial of charges''}.

On the other hand, \texttt{CTRLSum} fails to generate aspect-focused summaries for both cases. We believe that it struggles to align high-level concepts and explicit entities in the article due to a lack of such aspect-specific examples in its training data. Instead, it generates summaries focusing on lexically similar words, i.e. ``defenders'' for both cases.

Based off of \texttt{GPT3-D2}'s promising keyword-focused summarization capabilities observed above, we next conduct a human study to systematically compare it against the \texttt{CTRLSum} baseline. We leave further explorations of aspect-based summarization to future work, given the mixed to poor results for both models at this task.

\subsection{Human Study: Keyword-focused summarization}

\paragraph{Task Setup} Similar to Section~\ref{sec:human-study}, we design an A/B test to compare the two models. We use the same set of 100 CNN\footnote{We run this study using only CNN articles as the baseline \texttt{CTRLSum} model is trained on \texttt{CNN/DM}.} articles as Section~\ref{sec:human-study}. We randomly extract 2 distinct named entities from each article. In the study interface, the annotator is shown the article-keyword pair and \texttt{GPT3-D2}{} and \texttt{CTRLSum} summaries corresponding to it. They are asked to select the summary that best summarizes the input article while focusing on the given keyword. Exact task instructions are included in Appendix \ref{app:instructions}.

Again, we run this study using the Prolific platform. We recruit 60 participants to annotate the 100 articles; each article is annotated by 3 annotators which includes annotations for 2 separate keywords. Each annotator evaluates 5 articles.

\begin{figure}[t]
    \centering
    \includegraphics[scale=0.24, trim=60mm 130mm 0mm 100mm, clip]{keyword_human.pdf}
    \caption{Distribution of annotator votes for the keyword-focused summarization task. Annotators prefer \texttt{GPT3-D2}{} summaries over \texttt{CTRLSum} for approximately 70\% of all article-keyword pairs, showing unanimous preference more than half the time. }
    \label{fig:keyword}
\end{figure}

\paragraph{Results} Figure \ref{fig:keyword} shows the distribution of annotator votes between the \texttt{GPT3-D2}{} and \texttt{CTRLSum} models. Annotators show a clear preference for \texttt{GPT3-D2}{}. In fact, for nearly 70\% of all article-keyword pairs, \texttt{GPT3-D2}{} is preferred over \texttt{CTRLSum} by a majority of the annotators. The main rationales given for this choice were better contextualization of keyword-related information and better coherence in \texttt{GPT3-D2}{} summaries.

\paragraph{Impact} These results show that zero-shot \texttt{GPT-3} models present a promising alternative to fine-tuned models for such specialized summarization tasks that can be easily described using textual prompts. One of the major drawbacks of fine-tuned models is that they are constrained by what data is available and how it can be transformed to create new task-specific training data. \texttt{CTRLSum} relied on the SQuAD question answering dataset \cite{rajpurkar2016squad} because the required ``queries'' or ``questions'' were unavailable at scale for summaries in standard summarization datasets. In contrast, zero-shot models are not constrained by the availability of task-specific data and can flexibly adapt to new tasks. Future research should focus on further exploring these capabilities and possible improvements on currently ``unsolved'' tasks such as aspect-based or plan-based summarization.

\section{Discussion and Related Work}
In recent years, research in text summarization  \cite{rush2015neural, nallapati2016abstractive, see2017get, lewis2020bart,zhang2020pegasus, liu2022brio} has typically relied on comparisons with gold test sets for evaluation, possibly augmented with reference-free metrics for dimensions like factuality. This paper shows that \textbf{all these metrics are completely ineffective at evaluating GPT-3 summaries}. Although issues with these metrics, particularly low correlation with human judgments, have also been studied earlier \cite{fabbri2021summeval, deutsch2021understanding}, they are considered reliable when comparing systems in different score ranges \cite{peyrard2019studying, deutsch-etal-2022-examining}. However, \texttt{GPT-3} challenges these established practices and evaluation protocols, and poses an urgent need for better evaluation.

This brings us to manual evaluation, generally considered to be the gold standard for generation evaluation. The majority of summarization research now reports results from a human study in addition to automatic metrics, but  there is a general lack of consensus on what dimensions to evaluate, task design, and other factors \cite{hardy2019highres}. This presents difficulties in conducting reliable and reproducible comparisons between systems \cite{karpinska2021perils}, another factor contributing to the popularity of automatic metrics. Although recent efforts like \texttt{GENIE} \cite{Khashabi2021GENIEAL} have taken steps to standardize manual evaluation protocols across systems, its annotation is not universally affordable and the quality is not strictly monitored. We hope that future work addresses these challenges and democratizes human evaluations.

The ultimate test of summarization systems is with actual users using the systems in practice. \citet{jones2007automatic} discusses the need to align task formulations with actual applications scenarios (``purpose factors''). However, the research in text summarization until now has been constrained to certain problems or domains by the heavy dependence on large-scale training data: for example, producing a bullet-point summary of a news article has emerged as standard due to availability of data from CNN, not because it is shown to be the best way to present information.

Now, the success of zero-shot models like \texttt{GPT3-D2}{} can allow realistic use-cases to drive research in a more top-down way. We already show that \texttt{GPT3-D2}{} improves upon prior keyword-focused summarization systems that were trained on artificially adapted training data. In future research, we are interested in tackling other real world use cases, such as update summarization and plan- or aspect-based summarization. Additionally, adapting \texttt{GPT3-D2}{} to documents longer than the allowed context, or structured inputs such as tables, presents research challenges beyond the current capabilities of \texttt{GPT-3} and would be interesting to study.\footnote{We very briefly discuss long document summarization with GPT-3 in Appendix \ref{app:long-doc}.}

\section{Conclusion}
In this work, we performed the first systematic study comparing zero-shot \texttt{GPT-3}  and fine-tuned models at the news summarization task. We analyzed the impact of zero-shot models on the summarization field, including training paradigms and evaluation practices. Finally, to support further research and analysis into zero-shot summarization, we release a large corpus of generated summaries for multiple zero-shot and fine-tuned models, as well as human preference judgments comparing these systems.


\section{references}


\section{Introduction}
Social media's growing popularity amongst children and young adults raises serious concerns for their safety. The threat of online sexual grooming is an increasing problem in the digital age \citep{greene2020experiences}. In 2021 alone, UK police forces recorded over 5000 offences relating to sexual communications with a child, with social media apps such as Instagram and Snapchat popular amongst online predators \citep{nspcc}. Minors who fall victim to offenders suffer considerably, with many abusers seeking to establish physical contact offline \citep{shelton2016online}. To assist law enforcement, academics have been trying to identify in advance those predators that steer the relationship towards physical encounters \citep{briggs2011exploratory, winters2017sexual, o2003typology, williams2013identifying}. This important work predominantly relies on time-consuming human-effort to manually annotate conversations involving child sex offenders. In this work, we explore the extent to which automated annotation can mimic an expert human annotator.

Automatically detecting predatory behavior is a challenging task. Offenders use a variety of subtle behaviors to manipulate the flow of the conversation. Predators may use flattery to build trust \citep{barber2021exposing}, make threats or bribe a child as a coercion tactic \citep{joleby2021offender}. Whilst human experts can identify these contextual psychological behaviors in text, their implicitness can be problematic for machines \citep{buckingham2020extreme}. In computational social sciences, dictionary-based approaches \citep{tausczik2010psychological} are often used to identify psychological characteristics such as neuroticism \citep{bogdanova2014exploring}. However, these methods heavily rely on the included vocabulary, causing a large numbers of false positives for some behaviors \citep{kaur2021abusive} while overlooking others \citep{broome2020psycho}. Prior work utilising machine learning has focused on identifying predators from a mixed corpus of illicit and everyday conversations \citep{inches2012overview,pendar2007toward,miah2011detection,EBRAHIMI201633,gupta2012characterizing}. While valuable in its own right, this line of research does not offer significant value to grooming experts as it identifies grooming behavior after-the-fact \citep{razi2021human} and does not link or contribute to psychological insight.

This work is novel in its use of supervised and deep learning methods to analyse an expert-annotated corpus of real conversations between sexual predators and decoys pretending to be early teens. We report results and highlight the performance possible without incurring the cost of expert-labelling.
The paper is organised as follows: the methods and experimental setup are described in Section~\ref{sec:method}; Section~\ref{sec:results} then summarises results (with further details provided in the appendices) before Section~\ref{sec:concl} concludes the paper.

\section{Method and Experimental Setup}
\label{sec:method}

\begin{table}[h]
\centering
\begin{tabular}{lcc}
\hline
\textbf{Region} & \textbf{Num. Msgs} & \textbf{\% of Corpus}\\
\hline
Train & 4712 & 70\% \\
Test & 1355 & 20\% \\
Validation & 704 & 10\%\\
\hline
\end{tabular}
\caption{The number of messages included for training, testing, and validation regions.}
\label{tab:data-splits}
\end{table}

\begin{table*}[h]
\centering
\small{
\begin{tabular}{lccccc}
\hline
\textbf{Category} & \textbf{Coverage (\%)}& \textbf{Model}& \textbf{Precision}& \textbf{Recall} & \textbf{F1} \\
\hline
Communication/Coordination & 73.1 & NLI (1) & $84.0$ ($\pm0.6$) & $87.1$ ($\pm0.5$) & $85.5$ ($\pm0.2$)\\
Rapport Building & 15.2 & NLI (5) & $83.0$ ($\pm9.3$)  & $80.1$ ($\pm15.7$)  & $81.3$ ($\pm12.6$) \\
Control & 20.8 & NLI (5) & $61.5$ ($\pm1.7$) & $59.5$ ($\pm1.4$) & $60.4$ ($\pm0.2$)\\
Challenge & 4.5 & NLI (5) & $36.1$ ($\pm1.7$) & $24.2$ ($\pm4.1$) & $28.9$ ($\pm3.5$)\\
Negotiation & 20.9 & NLI (5) & $62.9$ ($\pm2.9$) & $66.7$ ($\pm1.7$) & $64.7$ ($\pm2.3$)\\
Use of Emotions & 16.4 & NLI (5) & $58.2$ ($\pm1.4$) & $53.1$ ($\pm1.65$) & $55.6$ ($\pm1.6$)\\
Testing Boundaries & 31.2 & NLI (5) & $69.6$ ($\pm0.8$) & $76.1$ ($\pm2.7$) & $72.7$ ($\pm0.9$)\\
Use of Sexual Topics & 18.3 & NLI (5) & $64.5$ ($\pm1.6$) & $69.3$ ($\pm3.6$) & $66.8$ ($\pm2.1$)\\
Mitigation & 3.0 & NLI (3) &   $62.2$ ($\pm10.7$) & $38.4$ ($\pm7.0$) & $47.5$ ($\pm8.5$)\\
Encouragement & 8.0 & NLI (1) & $43.2$ ($\pm4.8$) & $22.0$ ($\pm1.6$) & $29.1$ ($\pm2.0$)\\
Risk Management & 4.6 & NLI (1) & $63.1$ ($\pm4.0$) & $49.5$ ($\pm4.1$) & $55.4$ ($\pm3.5$)\\
\hline
\end{tabular}}
\caption{Eleven offender behavior categories derived and labelled by domain-experts, alongside their prevalence within the corpus. We report performance for the best performing model (based on F1) for each category. NLI (5) denotes the 5-message-input Natural Language Inference model, and similarly for NLI (3) and NLI (1). }
\label{tab:main-results}
\end{table*}

\paragraph{Dataset and Labelling.}
The \citet{pjwebsite} website  is an online repository of real, chat-based conversations between adults who were later convicted of grooming offences and decoys posing as children. Twenty-four chats, comprising of 12,942 messages, were labelled by a domain-expert with a background in forensic psychology. Chats were annotated in concordance with a theory of child grooming known as ``self-regulation'' \citep{elliott2017self} -- the notion that online predation contains a \textit{potentiality phase}, where the predator attempts to form a positive relationship with the victim; and a \textit{disclosure phase}, where the predator becomes more explicitly goal-oriented.
The following behaviors were identified and applied to the offender messages: (1) communication/coordination, (2) rapport building, (3) control, (4) challenges, (5) negotiation, (6) use of emotions, (7) testing boundaries, (8) use of sexual topics, (9) mitigation, (10) encouragement, (11) risk management. A qualitative description of these behaviors is given in Section~\ref{app-label-desc} of the Appendix. Overall 6,772 messages sent by the offender were labelled, with each message assigned a `Yes' or `No' for each of the above categories, based on the annotator's judgment. Due to the subjective nature of this assessment, labelling performed by different expert annotators may not be in complete agreement.

We construct binary classification tasks and predict whether each message is an example of each behavior category. Offender messages were split into training, testing, and validation regions (see Table~\ref{tab:data-splits} for information on data splits), and were stratified to ensure equal distribution of behaviors per region. To increase model confidence, we cross validated each experiment three times through random re-sampling. In our results, we report the average ($\pm SD$) scores for precision, recall and F1 as evaluation metrics.

\paragraph{Supervised Machine Learning.} Offender messages were tokenized, part-of-speech (POS) tagged, and lemmatized with \texttt{spaCy}. We removed stop-words (i.e., the, a, am, at, be, is), and used the frequency counts of the remaining unigrams as input features. Consistent with previous work \citep{miah2011detection, bogdanova2014exploring}, we compared the performance of four classifiers: Random Forest, Logistic Regression, Support Vector Machines, and Naive Bayes. For each algorithm, we optimized hyperparameters\footnote{Hyperparameters and the range of values explored are included in Section~\ref{hyp-app} of the Appendix.} using the GridSearch class in \texttt{Scikit-learn} \citep{pedregosa2011scikit} with 3-fold cross validation.

\paragraph{Deep Learning.}
To make predictions using transformer-based deep learning, we formulate the problem as a natural language inference (NLI) task. In NLI, the objective is to determine whether two sentences logically complement or contradict one another \citep{bowman2015large}. For our purposes, we compare each message with a standardized sentence crafted from each behavior label (i.e., ``This message is an example of \textit{building rapport}''). Each message/label sentence pair is then used as an input to the deep learning model, where a softmax activation function calculates the probability the two sentences are logically related. For training\footnote{We used a Tesla P100 GPU for training.}, we use \texttt{RoBERTa-large} \citep{liu2019roberta} with an implementation built in Pytorch. This model has been fine-tuned for NLI tasks using the Multi-Genre Natural Language Inference corpus \citep{N18-1101}. We ran both zero-shot and few-shot training setups. For the zero-shot model, predictions were made on the test set without any domain-specific training. In few-shot settings, we experiment with different amounts of positive training examples up to the full training set. As per \citep{wang2021entailment}, models were trained for 10 epochs with batch size 32 and a learning rate of $10^{-5}$.

\paragraph{Multi-Message Input.}
To explore whether the surrounding messages increase the contextual understanding of the transformer, we repeat full-shot experiments but expand the message window to include multiple prior messages sent by both speakers. To do this, we concatenate multiple messages into a single input which is then passed to the transformer. We compare the performance of NLI models with an input of 1, 3, and 5 messages.

\paragraph{Human-in-the-Loop Validation.}

To establish how comparable the model's output is to labelling performed by a human annotator, we scored the predictions made by the best performing model for $\sim10\%$ of the corpus ($645$ predator messages). Each message was scored on a 1-3 scale of agreement (1=disagree, 2=uncertain, 3=agree). Sections of chats were chosen at random. To mitigate potential bias, annotations generated during the initial labelling were shielded from view.

We use Cohen's $\kappa$ \citep{cohen1960coefficient} to measure pairwise agreement between the two human raters and the predictions generated by the best performing model for each behavior. We report the mean of these scores as an index of overall agreement.

\section{Results}
\label{sec:results}

Transfomer-based deep learning models achieved best performance across all categories, with the 5 message input NLI model performing best for most categories. Evaluation metrics for the highest performing model for each behavior are shown in Table~\ref{tab:main-results}. The performance of all models on all categories is reported in Section~\ref{full-results} of the Appendix. Eight of the eleven behaviors obtain a maximum F1 score above 50\%, with the most prevalent category, \textit{communication/coordination}, performing best with F1=$86\%$. Automatically labelling \textit{encouragement} and \textit{challenge} was least successful, with a maximum F1 score of $29\%$ for both.
For all eleven categories, a transformer-based model, trained on the full training set, was more successful than the traditional supervised machine learning algorithms. Averaging across behaviors, transformers improved performance of single message classification from $33\%$ to $53\%$ compared to the best performing supervised model. In 8 out of 11 cases, the supervised models, trained on the full training set, outperformed the zero-shot transformer. However, in 10 out of 11 cases, only 50 labelled instances were required by the transformer to exceed the best performing bag-of-words models. As many of the behaviors rely on contextual understanding, we increased the input size from a single message to 3 and 5 messages. The first contains two messages sent by the offender and one by the decoy, and the latter contains three by the offender and two by the decoy. Increasing the window size indeed improved performance for all behaviors\footnote{The majority of messages ($73\%$) were labelled as \textit{communication/coordination}. As a result, this behavior was not tested in a multi-message setting as grouping messages meant that all inputs would be positively labelled.} except for \textit{encouragement} and \textit{risk management}. On average, while 3-message classification did not improve single-message performance, 5-message classification improved on single-message F1 from $50\%$ to $56\%$.

\begin{figure}[h]
\centerline{\includegraphics[scale=0.55]{train_size_change_singlecol.pdf}}
\caption{Model performance (F1) on the eleven behavioral categories for single message classification, as the number of positive training examples increases (up to full training set). Error bars are standard  deviations.}
\label{fewshot-perf-change}
\end{figure}

Figure~\ref{fewshot-perf-change} shows the change in F1 for single message classification as the number of training samples increases. Whilst predictions for all behaviors improved with additional training, large variation in performance between the categories remains even for a fixed amount of positive training examples.
To better understand this variation, we manually examined the predictions made by the best performing single message model for $\sim 10\%$ of the corpus. We observed that the transformer has learned some rules that caused a number of false positives. For example, the \textit{testing boundaries} model correctly predicted a number of questions aimed at assessing the victim's willingness to engage, e.g., "what are you looking for?", but also included several logistical questions, e.g., "how about i come by at 8pm?". The \textit{rapport building} model correctly recognised complements and sweet talk as positive examples, but missed more everyday examples of rapport building, e.g., "how was your spring break?". Some aspects of \textit{control} required coverage over longer ranges than we have focused on. For example, persistently asking the same question is a sub-category of \textit{control} that was often misclassified. In this case, increasing the the message window to 3 messages was not enough to improve performance, however, the larger 5 message window did notably improve performance, with the F1 increasing from $54\%$ to $60\%$ due to better recall. In trying to predict \textit{encouragement}, the worst performing category, the transformer appeared to overfit on short verbal nods (i.e., "kool" and "sure", which appeared in the corpus either as evidence of encouragement or simple linguistic fillers) which increased the false positive rate. Over-reliance on this simple rule would explain why the model did not utilise the additional information provided in the multi-message setting. \textit{Risk management} achieved reasonable performance (F1=$55\%$) despite being one of the rarest behaviors in the corpus. This was largely a consequence of the transformer recognising instances where the offender attempts to establish the presence of a parent, e.g., ``when are they getting home?''.

In addition to drawing qualitative observations on the ability of the transformer-based model to recognise different offender behaviors, we quantify how comparable the best performing single-message model is to a human annotator. We compare the labelled classes between the original labels (Rater 1), with the predictions made by the best performing models (Model), and the labels verified during post-validation (Rater 2). We report a mean $\kappa$ score of 0.68, indicating `substantial agreement' between raters \citep{mchugh2012interrater}. Pairwise agreement is reported in Table~\ref{tab:post-val}. Agreement was highest between the two human raters ($\kappa=0.8$). The average agreement between each rater and the predictions generated by the model was $\kappa=0.62$ (see Section~\ref{app:agree} in the Appendix for pairwise agreement per behavior).

\begin{table}[h]
\centering
\begin{tabular}{lccc}
\hline
 & \textbf{Rater 1} & \textbf{Rater 2} & \textbf{Model}\\
\hline
\textbf{Rater 1} & - & 0.8 (0.78) & 0.59 \\
\textbf{Rater 2} &  & - & 0.65 (0.66)\\
\hline
\end{tabular}
\caption{Pairwise agreement (Cohen's $\kappa$) between raters over all behaviors. A score of 1.0 indicates perfect agreement. Value in parentheses indicate Rater 2 agreement when `uncertain' validation ratings are included.}
\label{tab:post-val}
\end{table}

\section{Conclusion}
\label{sec:concl}

Manually labeling the 24 chat logs used in this work took over 600 hours. Given that the full Perverted-Justice corpus contains 850 chat logs, it would be infeasible to label the entire corpus without the help of automated methods. We find that a transformer-based deep learning approach yields promising results when applied to the detection of online predatory behavior. However, even with training, the agreement between the model and a human annotator is not comparable to the agreement between two human annotators. The model's success in predicting behaviors varies significantly between the categories. Performance is better for more common behaviors, however, we show that the variation is not only due to the number of positive examples in the training data. Extending the prediction task to include multiple messages, increasing the contextual window, boosted detection for certain behaviors. The F1 score for \textit{rapport building}, for example, increased from $61\%$ for single messages to $81\%$ when classification was based on five-messages. Performing post-validation on the automatic classifications allowed us to gain qualitative insight about the model's performance, which may be used to design better prompts and improve performance further. Overall, our results are an encouraging step towards building an automated, psychology-informed model to detect online sexual exploitation.


\section{Introduction}
In reinforcement learning (RL), agents must solve sequential decision making tasks to maximise some notion of expected cumulative reward, known as the value function. Often the solutions to these tasks share elements in terms of their structure - different tasks can be decomposed into sequences of commonly shared sub-tasks. These sub-tasks link the timescales of primitive actions that take place at every step, and that of the broader abstract task, facilitating planning over these extended timescales. As such, \emph{temporal abstraction} is the process by which an agent learns the temporal structure of a task in a way that can both reduce cognitive load and improve generalisation across tasks with shared structure.

The canonical approach to address this form of abstraction is the Options framework \cite{sutton1999between}. Agents using Options seek to learn a set of policies associated with distinct sub-tasks, as well as the initiation and termination conditions for these policies. While this successfully quantifies the temporal abstraction problem, without predefined handcrafted domain-specific Options this process can be more complex than learning simple value functions \cite{pateria2021hierarchical}.

Rather than learning a specific policy associated with each sub-task, an alternative approach to temporal abstraction is to directly consider the time horizon of the actions taken in tandem with the policy for selecting them. This approach of temporally extending primitive actions by repeating them has been successfully applied to model-free RL; an agent's action space can be augmented with temporally extended actions \cite{lakshminarayanan2017dynamic}, or an additional policy can be learnt over the space of possible action repeats \cite{DBLP:conf/iclr/SharmaLR17, biedenkapp-icml21}. This is in contrast to the typical usage of fixed action repetition \cite{Braylan2015FrameSI, hessel2019inductive} in popular RL environments such as the Atari Arcade Learning Environment \cite{bellemare2013}, where action repetition is a fixed property of the environment rather than a process determined by the agent.

Given the success of temporally extended model-free methods, we set out to investigate whether temporally extending the successor representation (SR) could improve generalisation in a similar way. The SR linearly decomposes the problem of learning the value function into independently learning two components: a matrix of expected discounted state occupancies, and a vector of state-dependant expected rewards \cite{dayan1993improving}. Thus, the SR is able to improve upon the generalisation of model-free methods in tasks where the transition or reward structures changes \cite{russek2017predictive}.

We show that a temporally extended variant of the SR, which we term \emph{t-SR}, performs comparably to temporally-extended model-free approaches in static, unchanging environments. Further, t-SR exhibits dramatically improved performance over temporally-extended model-free methods when in environments with dynamic reward structure - specifically we assess reward revaluation.
\section{Preliminaries}
\textbf{Learnt Action Repetition}
In RL, a given task is characterised by a Markov Decision Process (MDP)  $\mathcal{M} = \langle \mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R} \rangle $. Here, the state space $\mathcal{S}$ contains all the states the agent can occupy and the action space $\mathcal{A}$ describes the allowable actions in each state. The transition probabilities $\mathcal{P}$ then define the probability $p(s'|s,a) \dot = \text{Pr}(S_t = s' | S_{t-1} = s, A_{t-1} = a)$ of transitioning from state $s$ to state $s'$ having taken action $a$, with $\mathcal{R}$ being the rewards $r(s,a) \dot = \mathbb{E}\left[R_t | S_{t-1} = s, A_{t-1} = a \right]$ that the agent receives after doing so. The agent is tasked with learning a policy $\pi (a | s)$ which specifies the probability of choosing each action given the current state. Specifically, for a given finite episodic MDP, the agent's task is to learn an optimal policy $\pi^{*}(a | s)$ that maximises expected cumulative discounted reward across an episode.

Conventionally, these actions are single-step primitive actions - the agent resamples its policy at $s'$ to determine the next action. Recent work has extended this action selection policy with a \emph{holding-time} (or \emph{skip}) for the action \cite{DBLP:conf/iclr/SharmaLR17,biedenkapp-icml21}. This is the number of times the chosen action will be repeated before the action selection policy is queried again. Agents are therefore equipped with both an action selection policy $\pi^a : \mathcal{S} \rightarrow \mathcal{A}$, and an action repetition policy $\pi^j : \mathcal{S} \rightarrow \mathcal{J}$. Here we follow the notation of \cite{biedenkapp-icml21} and let $j$ be an action repetition number, with $\mathcal{J}$ being the set of allowable action repetitions.

The action repetition policy $\pi^j$ can be learnt using the same methods used to learn a standard policy. Here, we use the $Q$-learning style update rule for learnt action repetitions derived in \cite{biedenkapp-icml21} due to its relevance to constructing SRs \cite{watkins1992q,dayan1993improving}, but methods for learning repetition policies through policy gradients also exist \cite{DBLP:conf/iclr/SharmaLR17}. An optimal $\pi^{j*}$ can be obtained by greedily choosing from the optimal $Q^{\pi j*}(s, j |a)$, with the action-repetition values drawn from transitions made in the environment according to:
\begin{equation}
    Q^{\pi j}(s, j | a) = \mathbb{E}^\pi \left[ \sum_{k=0}^{j-1} \gamma^k r_{t+k} + \gamma^j Q^\pi (s_{t+j}, a*_{t_j}) | s_t = s, a, j) \right]
\end{equation}
where $a*$ = $\text{argmax}_a \left[Q^\pi(s_{t+j}, a) \right]$ and $\gamma \in [0,1]$ represents a temporal discount factor that exponentially downweights distal rewards.

\textbf{Successor Representations} The SR was originally introduced as a method to improve generalisation in model-free RL \cite{dayan1993improving}, which it achieves by decomposing the value function into expected state occupancies and state-rewards. The SR itself, is therefore defined as the expected discounted future occupancy of state $s'$ along a trajectory initiated in state $s$ by taking action $a$:
\begin{equation}
    M^\pi(s,a,s') = \mathbb{E}^\pi \left[ \sum_{t=0}^\infty  \gamma^t\mathbb{1}(s = s') | s_0 = s, a_0 = a \right]
\end{equation}
where $\mathbb{1}$ is the indicator function. State-action values for a given policy can then be obtained by linearly combining $M^\pi$ with the rewards associated with each state, which can be learnt separately via supervised methods. Thus, the SR allows for rapid reward revaluation by reusing $M^\pi$ - the learnt expected state dynamics for a given policy.

\section{Temporally Extended Successor Representations}
Expected dynamics can be temporally extended in the same fashion as value functions. Consider the matrix formulation of the SR: $\mathbf{M^\pi} = \sum^\infty_{t=0}\left(\gamma \mathbf{T}^\pi \right)^t$, where $\mathbf{T}^\pi$ is the expected one-step transition function between states following policy $\pi$. To specify a SR that considers action repetition, let $\mathbf{T}_a$ be the expected transition conditioned on action $a$ such that $\mathbf{T}^\pi = \mathbb{E}^\pi(\mathbf{T}_a) $. The temporally extended successor representation (t-SR) for an action repetition of $j$ can then be obtained by:
\begin{equation}
    \mathbf{M}^{\pi j} = \mathbb{E}^\pi \left[\sum_{k=0}^{j} \left( \gamma \mathbf{T}_a \right)^k \right] + {\mathbb{E}^{\pi}\left[(\gamma \mathbf{T}_a)^j \right]}  \left(\mathbb{I} - \gamma \mathbf{T}^{\pi} \right)^{-1}
\end{equation}
where the right most expression is derived from the converging infinite series (Neumann series) of the standard SR. Compared to the SR, these representations skew according to the action space, reflecting the extended transitions instigated by repetition of the same action - this leads to a markedly different representation of distal state than altering the discount factor $\gamma$ \cite{Momennejad2018PredictingTF} (Fig \ref{fig:place_fields}).

To use the SR for determining action repetition, an action is first sampled from $\pi^a$ such that $M(s,a,s',j)$ now has two of its arguments specified: $s$ and $a$. The action repeat is then the $\max_j$ of $M(s, a, s', j)W$, where $W$ is the learnt reward vector. The baseline SR is learnt through temporal difference learning with state occupancies acting as cummulant rewards, while the reward weights $W$ are learnt through online supervised learning based on the expected rewards and actual rewards obtained. Thus, the t-SR can be learnt using using a typical temporal difference learning rule:
\begin{gather}
        M^{\pi j}(s,a,s', j) \xleftarrow{\alpha} M^{\pi j}(s,a,s', j) + \mathbb{E}^{\pi}\left[ \sum_{k=0}^{j-1} \gamma ^k\mathbb{1}(s_k = s) + \gamma^j M^{\pi}(s_{t+j}, a*, s') - M^{\pi j}(s,a,s',j)\right]
\end{gather}

\begin{figure}
    \centering
    \makebox[\linewidth][c]{
    \begin{subfigure}{0.22\textwidth}
    \centering
    \includegraphics[width=0.95\textwidth]{figures/gamma_0.99_j_1.png}

    \label{fig:gamma099}
    \end{subfigure}

    \begin{subfigure}{0.22\textwidth}
    \centering
    \includegraphics[width=0.95\textwidth]{figures/gamma_0.8_j_1.png}

    \label{fig:gamma08}
    \end{subfigure}

    \begin{subfigure}{0.22\textwidth}
    \centering
    \includegraphics[width=0.95\textwidth]{figures/gamma_0.99_j_3.png}

    \label{fig:j2}
    \end{subfigure}
    \begin{subfigure}{0.22\textwidth}
    \centering
    \includegraphics[width=0.95\textwidth]{figures/gamma_0.99_j_7.png}

    \label{fig:j7}
    \end{subfigure}
    }
    \caption{Comparison of SRs for different $\gamma$ and action repetition $j$. These SRs are taken under a random walk policy in a 10x10 gridworld with an action space consisting of the cardinal directions. Each figure consists of a single column of the SRs, projected over the gridworld. Projecting the column over the grid therefore shows the expected likelihood of having started in a state and moving to the state $i$ under the random policy. These figures show the different temporal effects of changing $\gamma$ and varying the action repeat $j$. Increasing action repetition leads to representations that are heavily skewed in the directions of actions. }
    \label{fig:place_fields}
\end{figure}
\section{Experiments}
    To assess the reward revaluation capability of the t-SR, we used three tabular gridworld environments which each had two possible goal locations  (Fig \ref{fig:env_diagrams}). Agents learn a policy over 10000 episodes with the reward fixed in one of these locations. The reward is then moved to the other possible location, and the agents must relearn their policy over an additional 10000 episodes. The environments are sparsely rewarded, with $+1$ reward for reaching the terminal goal location, $-1$ for entering one of black \emph{lava} squares (which terminate the episode), and $0$ reward otherwise.

\begin{figure}
    \centering
    \makebox[\linewidth][c]{
    \begin{subfigure}{0.14\textwidth}
    \centering
    \includegraphics[width=0.95\textwidth]{figures/hairpin.png}
    \caption{}
    \label{fig:hairpinenv}
    \end{subfigure}

    \begin{subfigure}{0.14\textwidth}
    \centering
    \includegraphics[width=0.95\textwidth]{figures/hairpinhard.png}
    \caption{}
    \label{fig:hairpinhardenv}
    \end{subfigure}

    \begin{subfigure}{0.14\textwidth}
    \centering
    \includegraphics[width=0.95\textwidth]{figures/hairpinharder.png}
    \caption{}
    \label{fig:hairpinharderenv}
    \end{subfigure}
    }
    \caption{Three gridworld environments were used in the reward revaluation task \textbf{(a)} Junction \textbf{(b)} Junction Hard \textbf{(c)} Junction Very Hard. Red kites indicate starting locations. White squares are occupiable states and black squared are terminal lava states yielding a $-1$ reward. Green hexagons are potential terminal goal states and yield a $+1$ reward. A single goal is present in each environment at a time. }
    \label{fig:env_diagrams}
\end{figure}
Five agents were used. A basic Q-learning agent \cite{watkins1992q}, an SR agent\cite{dayan1993improving}, a temporally extended \emph{skip Q-learning} agent \cite{biedenkapp-icml21}, a control SR agent with random temporally extended exploration similar to \cite{dabney2020temporally}, and t-SR. In all cases $\alpha = 1$ was used for value and SR updates, $\gamma = 0.99$ , and $\alpha_r = 1$ was used for updating SR reward vectors. All agents followed an $\epsilon-$greedy exploration policy with $\epsilon = 0.05$. The skip Q and t-SR agents selected both actions and action repetitions in an $\epsilon$-greedy fashion, and used a $j_{max}$ of 7. The SR agent with random temporally extended exploration sampled action repetition randomly with probability $\epsilon$ during training. All results were evaluated under the greedy policy using the same 50 seeds.

\section{Results}
Reporting the average episode return for each of the five agents, we see that t-SR relearns an optimal policy to the new goal in fewer episodes than any other agent used (Fig \ref{fig:rewards}). While initially learning at the same rate as skip Q-learning before the reward changes location, t-SR is subsequently able to alter its policy much more rapidly after the change. Note that for the hardest environment, the agents without temporal action repetition policies did not on average find the optimal policy in $10000$ episodes.

To assess the rate of policy change after the reward changes, we calculated the average total variation between policies after each episode of learning (Fig \ref{fig:TV}). That is, $TV_\pi(t) =  \frac{1}{|\mathcal{S}|}\sum_{s \in \mathcal{S}}\frac{1}{2}||\pi(s)_{t-1} - \pi(s)_{t} ||_1$. We see that for the SR-based agents, the policy changes sharply and over a much shorter number of episodes compared to the purely model-free agents - which continue to update their policies over a protracted amount of time.

Finally, to assess the optimality of the trajectories taken we report statistics on the average path length to reach to goal (Fig \ref{fig:stepssndlengths} top). From the time to convergence to the minimum number of steps needed to solve the environment, we see that t-SR performs reward revaluation approximately 3.5 times faster in the hardest environment than the next fastest agent. Further, t-SR consistently queries its policy a fewer number of times throughout learning (Fig \ref{fig:stepssndlengths} bottom).
\begin{figure}[h]
    \centering
    \makebox[\linewidth][c]{
    \begin{subfigure}{0.27\textwidth}
    \centering
    \includegraphics[width=0.95\textwidth]{figures/Test_Episode_Reward_hairpin_skip_7.png}

    \label{fig:reward_hairpin}
    \end{subfigure}

    \begin{subfigure}{0.27\textwidth}
    \centering
    \includegraphics[width=0.95\textwidth]{figures/Test_Episode_Reward_hairpinhard_skip_7.png}

    \label{fig:reward_hairpinhard}
    \end{subfigure}

    \begin{subfigure}{0.27\textwidth}
    \centering
    \includegraphics[width=0.95\textwidth]{figures/Test_Episode_Reward_hairpinharder_skip_7.png}

    \label{fig:reward_hairpinharder}
    \end{subfigure}
    }

    \caption{Average episode return under the greedy policy for the three environments measured against the number of episodes trained under the $\epsilon$-greedy policy. Results are plotted as means and standard errors over 50 seeds.}
    \label{fig:rewards}
\end{figure}

\begin{figure}[h]
    \centering
    \makebox[\linewidth][c]{
    \begin{subfigure}{0.27\textwidth}
    \centering
    \includegraphics[width=0.95\textwidth]{figures/Avg_Total_Var_hairpin_skip_7.png}

    \label{fig:tv_hairpin}
    \end{subfigure}

    \begin{subfigure}{0.27\textwidth}
    \centering
    \includegraphics[width=0.95\textwidth]{figures/Avg_Total_Var_hairpinhard_skip_7.png}

    \label{fig:tv_hairpinhard}
    \end{subfigure}

    \begin{subfigure}{0.27\textwidth}
    \centering
    \includegraphics[width=0.95\textwidth]{figures/Avg_Total_Var_hairpinharder_skip_7.png}

    \label{fig:tv_hairpinharder}
    \end{subfigure}
    }
    \caption{Average total variation between policies at successive episodes for the three environments measured against the number of episodes trained under the $\epsilon$-greedy policy. Results are plotted as means and standard errors over 50 seeds.}
    \label{fig:TV}
\end{figure}

\begin{figure}[h]
    \centering
    \makebox[\linewidth][c]{
    \begin{subfigure}{0.27\textwidth}
    \centering
    \includegraphics[width=0.95\textwidth]{figures/Avg_steps_hairpin_skip_7.png}

    \label{fig:steps_hairpin}
    \end{subfigure}

    \begin{subfigure}{0.27\textwidth}
    \centering
    \includegraphics[width=0.95\textwidth]{figures/Avg_steps_hairpinhard_skip_7.png}

    \label{fig:steps_hairpinhard}
    \end{subfigure}

    \begin{subfigure}{0.27\textwidth}
    \centering
    \includegraphics[width=0.95\textwidth]{figures/Avg_steps_hairpinharder_skip_7.png}

    \label{fig:stpes_hairpinharder}
    \end{subfigure}
    }

    \centering
    \makebox[\linewidth][c]{
    \begin{subfigure}{0.27\textwidth}
    \centering
    \includegraphics[width=0.95\textwidth]{figures/Avg_lengths_hairpin_skip_7.png}

    \label{fig:steps_hairpin}
    \end{subfigure}

    \begin{subfigure}{0.27\textwidth}
    \centering
    \includegraphics[width=0.95\textwidth]{figures/Avg_lengths_hairpinhard_skip_7.png}

    \label{fig:steps_hairpinhard}
    \end{subfigure}

    \begin{subfigure}{0.27\textwidth}
    \centering
    \includegraphics[width=0.95\textwidth]{figures/Avg_lengths_hairpinharder_skip_7.png}

    \label{fig:steps_hairpinharder}
    \end{subfigure}
    }

    \caption{Average transitions (top row) and decisions (bottom row) performed by agents the three environments measured against the number of episodes trained under the $\epsilon$-greedy policy. \emph{Decisions} refers to how many times the agent queries its action policy during an episode. Results are plotted as means and standard errors over 50 seeds.}
        \label{fig:stepssndlengths}
\end{figure}

\begin{comment}
\begin{figure}
    \centering
    \makebox[\linewidth][c]{
    \begin{subfigure}{0.27\textwidth}
    \centering
    \includegraphics[width=0.95\textwidth]{figures/Avg_Total_Temporal_Var_hairpin_skip_7.png}
    \caption{}
    \label{fig:ttv_hairpin}
    \end{subfigure}

    \begin{subfigure}{0.27\textwidth}
    \centering
    \includegraphics[width=0.95\textwidth]{figures/Avg_Total_Temporal_Var_hairpinhard_skip_7.png}
    \caption{}
    \label{fig:ttv_hairpinhard}
    \end{subfigure}

    \begin{subfigure}{0.27\textwidth}
    \centering
    \includegraphics[width=0.95\textwidth]{figures/Avg_Total_Temporal_Var_hairpinharder_skip_7.png}
    \caption{}
    \label{fig:ttv_hairpinharder}
    \end{subfigure}
    }
\end{figure}
\end{comment}

\section{Discussion and Future Work}
The success of an agent in performing reward revaluation lies in the ability to discover the new reward. This requires effectively unlearning the current policy being followed. The total variation graphs in Fig \ref{fig:TV} show that, as expected, policies used by SR-based agents change significantly in a short number of episodes compared to purely model-free value based approaches. The improved performance of t-SR over SR or SR with random temporally extended exploration is likely due to the coherent policies used in the episodes immediately after the reward switch. The temporally extended dynamics learnt by t-SR will favour moving significant distances in the state space when compared to SR. This greatly increases the likelihood of following a sequence of actions leading to the new goal. This is also true of skip Q-learning, but these model-free agents are marred by policies that will consistently lead to the old reward location for an extended number of episodes following the reward switch, greatly decreasing the likelihood of finding the new reward.
We have proposed a new variation of the SR, the t-SR, which shows improved reward revaluation performance compared to SR or other comparable temporally extended methods Our next step is to  extend this framework to the function approximation domain using successor features \cite{barreto2017successor}. We also are currently developing a form of macro action discovery using the t-SR.
\printbibliography[heading=none]

\end{document}


\section*{Methods}

\subsection{Femtosecond electron pulse generation in a transmission electron microscope}
The experimental work was carried out in a commercially available transmission electron microscope (JEOL JEM 2100F) that has been modified to allow for the investigation of ultrafast dynamics in a stroboscopic laser-pump/electron-probe measurement scheme~\cite{Feist2017}. As our electron source, we employ a W/ZrO Schottky emitter ($r=\SI{490}{\nm}$ radius-of-curvature) operated at an extraction voltage of $U_\mathrm{ext} =\SI{2}{\kV}$ and a bias voltage of $U_\mathrm{bias} =-\SI{0.3}{\kV}$. Cooling the emitter just below the continuous Schottky-emission threshold of \SI{1150}{\K} (filament current \SI{1.6}{\A}), we generate ultrashort electron pulses via linear photoemission by focusing laser pulses (\SI{160}{\fs} pulse duration, \SI{515}{\nm} central wavelength, \SI{600}{\kHz} repetition rate) onto the apex of the nanotip. While we estimate that every electron pulse initially consists of up to a few hundred electrons~\cite{Bach2019}, apertures in the electro-optical beam path limit the transmitted beam to electrons that were generated close to the optical axis, resulting in average transmitted bunch charges of below one electron per pulse. Subsequent acceleration to \SI{200}{\keV} energy and coupling into the microscope column enable a pulse characterisation in real- and reciprocal space, spectral pulse properties are determined by an imaging energy filter (CEFID, CEOS GmbH).

\subsection{Event-driven photoelectron detection}
The photoelectron correlations experiments are imaged with a hybrid pixel electron detector that is based on the Timepix3 ASIC (EM CheeTah T3, Amsterdam Scientific Instruments B.V.) and mounted behind the energy filter. The camera generates a stream of data packages containing the position of electron-activated detector pixels, their time-of-arrival (ToA), which are digitized with \SI{1.56}{\ns} time bins, and the energy (time-over-threshold, ToT) associated with incident electron events. At a beam voltage of \SI{200}{\kV} every individual electron activates a cluster of pixels with variable size ($N_\mathrm{pixels, avg} \approx$ 8 pixels), shape and energy ($\mathrm{ToT}_\mathrm{avg} \approx$ 280 $\,$ a.u.).

Single-electron event localisation of the ToT-corrected raw data stream is achieved using the Division of Nanoscopy, M4I, Maastricht University event clustering code~\cite{vanSchayckJ.Paul2021}, which is based on a Hierarchical Density-Based Spatial Clustering (HDBSCAN) in Python3. The algorithm reconstructs the timing and position of individual electrons incident on the detector from the activated pixels. Thereby, individual electrons are distinguished in terms of their ToA, attributing between three and nine neighbouring pixels activated within a time window of \SI{100}{\ns} and a summed ToT ranging from from 200 $\,$ a.u. to 400 $\,$ a.u. to the same cluster (see Ref.~\cite{vanSchayck2020}).

In a second step, the photoelectrons are clustered according to the femtosecond laser pulse that generated them. The temporal resolution of the detector (\SI{1.56}{\ns}) is much faster than the temporal pulse separation given by the laser repetition rate ($\approx \SI{1}{\us}$), but much slower than the temporal splitting of the correlated electrons at the detector ($\approx \SI{1}{\ps}$). Hereby, the electrons arriving at the detector within $ \Delta t_\mathrm{n} =$\SI{50}{\ns} are assigned to a number-class electron state $n=1,2,3,...$ determined by the number of electrons in one laser pulse.

\subsection{Double Laser-pulse electron generation}

For the two-laser-pulse generation described in Fig.~\ref{fig_2}f, a Michelson interferometer splits the incoming laser pulse into two separate pulses. One of the interference arms has a variable optical path length, implemented by a retroreflector mounted on a delay stage. The delay time of the two optical pulses is much lower than the repetition rate of the laser ($\approx \SI{2}{us}$). As a consequence, two photoelectrons generated by two separate laser pulses and two photoelectrons generated by the same pulse are both detected as two-electron events.

As the optical power on the laser tip oscillates for small delay times due to constructive and destructive interference of the laser, the number of generated electrons will strongly vary in this delay regime. Therefore, we selected delays with approximately the same one-electron-state rate ($\pm \sigma / 2$) over the integration time of five seconds.

\section*{Acknowledgements}
We thank the members of the G\"{o}ttingen UTEM team for constant support and useful discussions.

\noindent\textbf{Funding Information:}
The work at the Göttingen UTEM Lab was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) through  432680300/SFB\,1456 (project C01) and the Gottfried Wilhelm Leibniz program.

\noindent\textbf{Author contribution:}
R.H.: Data curation, Formal Analysis, Investigation, Resources, Software, Visualization, Writing - original draft, Writing – review $\&$ editing.
A.F.: Conceptualization, Formal Analysis, Investigation, Resources, Software, Supervision, Writing - original draft, Writing – review $\&$ editing.
T.D.: Investigation (supportive), Resources, Software, Visualization, Writing – review $\&$ editing.
M.M.: Investigation (supportive) Resources, Software,  Writing – review $\&$ editing.
S.V.Y: Methodology, Software, Writing – review $\&$ editing.
C.R.: Conceptualization, Formal Analysis, Funding acquisition, Methodology, Project administration, Software, Supervision, Visualization, Writing original draft, Writing – review $\&$ editing.

\bibliographystyle{apsrev4-2}