\section{Implement details}
We take two group of hyper-parameters: learning rate=5e-5, batch size=32 and learning rate=1e-5, batch size=16. We use $\mathcal{D}_{dev}$ to chose the best one. warm-up steps are set as 10. For each model, we train it for 30 epochs in total. We evaluate it for each epoch starting from epoch 10.

\section{Analysis of the choice of retrievers}

\begin{table}[!h]
	\centering
	\small
	\begin{tabular}{l|cccc}
		\toprule
		\multirow{2}{*} & \multicolumn{2}{c}{\textbf{SNLI(acc)}} & \multicolumn{2}{c}{\textbf{MNLI(acc)}} \\
		& \textbf{top} & \textbf{mm} & \textbf{top} & \textbf{mm}\\
		\midrule
		random &  68.49 & 69.35 & 59.02 &57.37 \\
		SBERT + static &  66.79 & 71.81 &59.39 &59.78 \\
		BERT-base + dynamic & 72.43 & 72.80 &  61.96& \textbf{62.88}\\
		SBERT + dynamic & \textbf{73.69} & \textbf{74.44} & \textbf{62.57}& 62.57\\
		\bottomrule
	\end{tabular}
	\caption{Impact of the choice of the retriever. $\textbf{random}$: randomly sample demonstrations.}
	\label{table:bert}
\end{table}

We also investigate the effect of the choice of the retriever on the performance. As showed in Table \ref{table:bert}, random retriever performs worst and dynamic SBERT retriever performs best. We also find dynamic pre-trained BERT-base outperforms static BERT, where BERT-base has similar model size with SBERT but it is not fine-tuned with extra dataset to learn a better sentence embedding. This indicates that with dynamic demonstration, the retriever can learn a better sentence embedding with training samples, so a small size of training set is enough to train a retriever well, which makes dynamic demonstration more useful in a low-resource domain.

\section{Generated prompts}

We show the prompt selected by top method and our max-margin method on SNLI and MNLI dataset in Table \ref{table:template1}, Table \ref{table:template2}, Table \ref{table:template3} and Table \ref{table:template4}

\begin{table*}[!h]
	\centering
	\small
	\begin{tabular}{l|ccc}
		\toprule
		\textbf{seed} & \textbf{contradiction} & \textbf{neutral} & \textbf{entailment}\\
		\midrule
		13  & Now &  Here & Close up of \\
		21   & In the background &  Then, & A photo of \\
		42   & At the same time, & At the end of the day, & A black and white photograph of \\
		87   & The back of  & In this video, & Description: \\
		100   & In the distance, &  And & A photo of \\
		\bottomrule
	\end{tabular}
	\caption{Generated templates of top method in SNLI dataset}
	\label{table:template1}
\end{table*}

\begin{table*}[!h]
	\centering
	\small
	\begin{tabular}{l|ccc}
		\toprule
		\textbf{seed} & \textbf{contradiction} & \textbf{neutral} & \textbf{entailment}\\
		\midrule
		13  & As &  Later, & Close up of \\
		21   & At the same time  &  In this photograph, & One of \\
		42   & At the same time, & On the right & A black and white photograph of \\
		87   & A close up of  & A woman and & A black and white photo of \\
		100   & Then, &   Here & The photo shows \\
		\bottomrule
	\end{tabular}
	\caption{Generated templates of max-margin method in SNLI dataset}
	\label{table:template2}
\end{table*}

\begin{table*}[!h]
	\centering
	\small
	\begin{tabular}{l|ccc}
		\toprule
		\textbf{seed} & \textbf{contradiction} & \textbf{neutral} & \textbf{entailment}\\
		\midrule
		13  & Though &   Although & The  \\
		21   & At the end of the day  &  For the most part, &  At the same time \\
		42   & Now & At the end of the day, & That \\
		87   & Although  & Even & For example, \\
		100   & Yet &   For the most part & Again, \\
		\bottomrule
	\end{tabular}
	\caption{Generated templates of top method in MNLI dataset}
	\label{table:template3}
\end{table*}

\begin{table*}[!h]
	\centering
	\small
	\begin{tabular}{l|ccc}
		\toprule
		\textbf{seed} & \textbf{contradiction} & \textbf{neutral} & \textbf{entailment}\\
		\midrule
		13  & For some reason &   So & In the meantime,  \\
		21   & For some reason  & However, &  As you can see\\
		42   & Even though & The & At least \\
		87   & It's not that  & Even & Finally, \\
		100   & But in the end, &  In fact & , \\
		\bottomrule
	\end{tabular}
	\caption{Generated templates of max-margin method in MNLI dataset}
	\label{table:template4}
\end{table*}

\section{Conclusion}

In this paper, we investigate how to solve the natural language inference generation task in a few-shot setting. We propose LM-PDD to combine a PLM with prompts and demonstrations, including a novel template selection method, which leverages the development set for conditional generation tasks, and a dynamic demonstration method. Our methods outperform previous prompt selection and demonstration methods, achieving average 8\% accuracy over the vanilla fine-tuning method.
\section{Experiments}
In this section we first present our experiments setup and baseline models in \secref{sec:exp} and \secref{sec:baseline} respectively. Then we show the different aspects of evaluation results and give a detailed analysis from \secref{sec:ana} to \secref{sec:ga}. According to our experiments, we want to determine these questions: First, does the maximum margin function help to automatically generate templates and does dynamic demonstration work in the few-shot setting? Second, are the sentences produced by our model reasonable? Third, how is the generalization ability of our dynamic demonstration method?

\subsection{Experiments Setup}
\noindent
\textbf{Dataset.} We evaluate our model on two English datasets: SNLI and MNLI. Because the MNLI test set doesn't have ground truth labels, we use the mismatched development set as the test set and matched development set as the development set. Different from~\citet{DBLP:conf/acl/GaoFC20} use $K=16$ for classification tasks, we set $K=200$ considering the generation task is more challenging. Detailed statistics are listed in \tabref{table:dataset}.
\label{sec:exp}
\begin{table}[!h]
	\centering
	\small
	\begin{tabular}{l|ccc}
		\toprule
		\textbf{Dataset} & \textbf{\#Training} & \textbf{\#Development} & \textbf{\#Test}\\
		\midrule
		SNLI  & 550,152 &  10,000 & 10,000 \\
		MNLI   & 392,302 &  10,000 & 10,000 \\
		\bottomrule
	\end{tabular}
	\caption{Statistics of SNLI and MNLI dataset.}
	\label{table:dataset}
\end{table}

\begin{table*}[t!]

	\setlength\tabcolsep{4pt}
	\centering
	\small
	\begin{tabular}{l|cccccc|cccccc}
		\toprule
		\multirow{2}{*}{\textbf{Methods}} & \multicolumn{6}{c|}{\textbf{SNLI}} &\multicolumn{6}{c}{\textbf{MNLI}}  \\

		&\textbf{acc~(\%)}   &\textbf{B-4}  &\textbf{R-2}  &\textbf{PPL$\downarrow$}    &\textbf{Div-2}  &\textbf{Div-3}  &\textbf{acc~(\%)}   &\textbf{B-4}  &\textbf{R-2}  &\textbf{PPL$\downarrow$}     &\textbf{Div-2}  &\textbf{Div-3}   \\
		\midrule

		BART-large$^\spadesuit$  & 94.17 & 8.75 & 17.63&41.91 &0.214 &0.407 &89.42 &10.55 &19.25 & 39.74& 0.365& 0.570\\
		\midrule

		Rule-base$^\clubsuit$  & 75.30 & 5.13 & 12.83 & 114.34 & 0.213& 0.402&69.07 &6.44 &16.45 &107.18 &0.289 &0.467 \\
		\midrule

		AMD  & 34.75(1.91) & 0.81 & 2.19 & 678.45 & 0.097& 0.265& 34.66(3.16)&0.00 &0.25 &981.49 &0.198 & 0.547\\
		BART-large  & 66.45(8.00) & 8.63 & 15.27 &\textbf{61.93} &0.182 &0.356 &55.45(4.33) &\textbf{10.05} &16.69 &\textbf{45.35} &0.415 &0.640 \\
		\midrule
		$Prompt_{man}$ & 67.41(5.63) & 8.63 &15.58 & 67.49& 0.176& 0.343& 58.62(3.01)&9.66 &16.57 &47.46 &0.424 &0.649 \\
		$\ +static$ & 71.13(9.44) & \textbf{9.10} &\textbf{15.93} &63.95 &0.176 &0.342 &59.82(5.66) &9.90 &16.58 &47.47 &0.426 &0.650 \\
		$\ +dynamic$ & 74.01(7.89) & 8.11 &14.58 &71.36 &0.188 &0.362 &\textbf{63.33}(1.97) &9.35 &16.10 &48.87 &\textbf{0.442} &\textbf{0.674}  \\
		\midrule
		$Prompt_{top}$ & 68.04(10.01)& 8.50& 15.38& 67.79&0.176 &0.344 &59.20(9.81) &9.64 &16.45 &47.64 &0.425 &0.653 \\
		$\ +static$ & 66.79(12.23) &8.08 & 14.71& 63.57& 0.174& 0.336& 59.39(5.34)& 9.96& \textbf{16.71}& 46.39 & 0.426& 0.649\\
		$\ +dynamic$ & 73.69(4.16) & 8.10& 14.49& 68.72& 0.191& 0.367& 62.57(4.30)& 9.54& 16.29& 47.42& 0.434& 0.662\\
		\midrule
		$Prompt_{mm}$ & 69.33(6.82) & 8.51 &15.53 & 67.79& 0.176& 0.345& 59.53(4.20)&9.75 &\textbf{16.71} &47.64 &0.423 &0.647 \\
		$\ +static$ & 71.81(6.17) &8.71 & 15.49& 65.39& 0.177& 0.342& 59.78(4.33)& 9.69& 16.36& 46.60& 0.424& 0.647\\
		$\ +dynamic$ & \textbf{74.44}(4.74) & 7.97&14.42 &70.28 &\textbf{0.192} &\textbf{0.370} &62.57(2.13) &9.38 &16.04 &47.45 &0.440 &0.670 \\
		\bottomrule
	\end{tabular}
	\caption{Our main results using BART-large generator. Best results in few-shot settings are marked with \textbf{bold} font. $\spadesuit$ stands for \textbf{\textit{Full-shot}}: fine-tune with full data; $\clubsuit$  stands for \textbf{\textit{Zero-shot}}: no training data; other methods are in \textbf{\textit{Few-shot}} setting: we use $K$ = 200 (per class) for few-shot experiments. We report mean performance over 5 different split. For accuracy metric, we also report standard deviation; $man$: use manually defined prompt; $top$: auto prompt selected by maximum score; $mm$: auto prompt selected by max margin; $static$: use static SBERT as the retriever; $dynamic$: train SBERT retriever with generator; $\downarrow$:lower is better.}
	\label{table:linkprediction}
\end{table*}

\noindent
\textbf{Evaluation Metrics.} We adopt the standard text generation metrics perplexity (PPL), BLEU-4 score~\citep{papineni2002bleu} and ROUGE-2 score~\citep{lin-2004-rouge} to automatically compare generated results and reference hypotheses. Since these metrics do not consider the semantic meaning of sentences, we further use a state-of-the-art NLI classifier trained on 4 datasets~\citep{nie-etal-2020-adversarial} to calculate the accuracy of predictions, which achieves 92.6\% and 90.6\% accuracy in SNLI and MNLI mismatched datasets respectively. Good hypotheses are supposed to be not only accurate but also diverse, and therefore we use distinct n-grams normalized by the total number of generated n-grams Div-n~\citep{li-etal-2016-diversity} to measure diversity.
We also develop human evaluation in~\secref{sec:he}.

\noindent
\textbf{Implement Details.} We use BART-large as the generator. For the retriever, we deploy our experiment with the BERT-base and SBERT models. We optimize models using Adam~\citep{DBLP:journals/corr/KingmaB14}. The hyper-parameters are chosen by $\mathcal{D}_{dev}$. The max length is set to 128 for non-demonstration methods and 170 for others. We conduct all experiments on single NIVIDIA V100 16GB GPUs without half-precision. We use ROUGE-2 on the development set to choose the best models instead of the accuracy because we find the NLI classifier tends to give high accuracy even before the model is converged. More details can be found in Appendix A and B, including the generality of dynamic demonstration and generated prompts.

\subsection{Baselines}
\label{sec:baseline}
We compare our model with both unsupervised and supervised methods:

\noindent
\textbf{Rule-base:} A simple rule-based method. First, we use Stanford Stanza~\citep{qi2020stanza} to tokenize premises and get POS tags. For entailment, we simplify the premises by removing \textbf{\textit{adj}} and \textbf{\textit{adv}} words as hypothesis. For contradiction, we randomly replace one word in each premise with its antonym. If there are no candidate words, we add negative words into the sentence (e.g. ``not'' and ``don't''). For neutral, we randomly replace one word in each premise with its hyponym.

\noindent
\textbf{AMD:} A RNN-based model proposed by ~\citet{DBLP:journals/csl/StarcM17}. It learns a latent representation $\mathbf{z}$ that represents the mapping between the premise and the
label on one side, and the hypothesis on the other side.

\noindent
\textbf{BART-large:} A pre-trained language model which performances well on a range of abstractive dialogue, question answering, and summarization tasks.

\subsection{Main Results \& Analysis}
\label{sec:ana}
Our main results is shown in \tabref{table:linkprediction}. Overall, the full-shot finetuning of the BART-large model performs best except for diversity with overwhelming training consumption as a sacrifice, and the rule-base method outperforms methods in few-shot settings in terms of accuracy, while fails to perform well in other metrics. Despite the fair accuracy, the rigid rule leads the generated hypotheses less comprehensible, and the trivial edition would not provide much gain for data augmentation. In the few-shot setting, AMD is poor due to lack of pre-trained knowledge, and prompt-based methods take advantage over the most of metrics because the training and the predicting share the same LM objective.
Generally, the B-4, R-2 and PPL are not always consistent with the accuracy result but the oscillation is slight. The increase of diversity is usually associated with the decline of B-4 and R-2, which is explainable because these metrics only consider the overlap of n-grams with one fixed reference sentence and neglect their semantic meaning. Our dynamic demonstration yields the best accuracy score in few-shot settings, achieving 8\% improvement over BART-large model in both SNLI and MNLI datasets. The diversity of our methods even exceeds BART-large trained in the full-shot scenario on MNLI, which is meaningful for the data augmentation application.

\noindent
\textbf{Prompt Template Selection.}
When comparing different prompt generation strategies $man$, $top$ and $mm$ without demonstrations, we can find the manually designed prompt performs not as good as prompts generated by T5 model in accuracy and diversity metric, and for the accuracy of automatic generation prompts, those selected by max-margin function beat those selected by maximum score on most of metrics. This verifies our assumption that good templates of one condition get high scores in the corresponding condition while low in other conditions, and good templates narrow down the standard deviation of the accuracy.

\noindent
\textbf{Demonstration Selection.}
Under each prompt generation setting, we observe that the static demonstration improves the most of results over the original prompt except the $top$ setting, mainly due to the instability of templates generated by the maximum score. Our proposed dynamic demonstration method boosts the accuracy and diversity result with considerable gain, including the $top$ setting, which indicates that our dynamic demonstration is able to distinguish the more proper demonstration example. On the MNLI dataset, the dynamic demonstration achieves the best accuracy on manually defined templates, reflecting that in some cases the manual prompt template is simple but effective when the number of conditions is limited. In general, the dynamic demonstration shows its competitiveness no matter which prompt template selection strategy is.

\subsection{Human evaluation}
\label{sec:he}
In order to check the quality of the generated hypotheses, we carry out human evaluation. We recruit 5 students who are proficient in English to score the generation result of six models, with 50 samples in each condition and each dataset, and each sample is labeled by at least 3 evaluators. We examine two aspects of the quality: (1) Is the hypothesis in the right condition with the premise? (0 or 1) (2) Is the sentence reasonable grammatically? (0, 1 or 2, the higher the better). The main result is illustrated in the first two subfigures in \figref{fig:human}. For accuracy, the max-margin strategy is helpful in static demonstration, but in SNLI's dynamic demonstration things are different, maybe the power of dynamic demonstration hides the advantage of max-margin. The improvement of dynamic demonstration is always eye catching, which is consistent with auto-metric in \tabref{table:linkprediction}. For grammaticality, the rule-based method is inferior to other methods, indicating that the rule-based method owns poor readability despite the accuracy.

\begin{figure}[th]
	\centering
	\scalebox{1.0}{\includegraphics[width=1\columnwidth]{figure/human.pdf}}
	\caption{First column: the human evaluation of different datasets. The same color stands for the same prompt selection. Second column: accuracy of entailment, neutral and contradiction respectively, and here we merge the results of static and dynamic demonstration of different prompt selection methods as the ensemble results.}
	\label{fig:human}
\end{figure}

To further investigate the power of dynamic demonstration in different conditions, we can find that the advantage is not obvious under the scenario where these methods already obtain high accuracy, like entailment. Instead, it is more facilitative under those difficult conditions.

\subsection{Case study}
The case study result is shown in \tabref{table:case}. For the static method, it finds the simple connection between ``family'' and ``child'' while fails to recognize the semantic relation between the demonstration and the premise. For the dynamic method, in addition to finding the connection between ``ocean'' and ``beach'', it also excavates the deeper pattern, which helps the PLM to generate the correct corresponding hypothesis.
The case shows our dynamic demonstration method can find a more similar example as a demonstration to guide the generation than the static method.

\begin{table}[!h]
	\centering
	\small
	\begin{tabular}{l}
		\toprule
		SBERT + static \\
	   \textbf{Input:} \underline{A child playing with a sword at the bottom of the} \\
	   \underline{stairs. \textcolor{blue}{As} a child playing with a spoon.} [SEP] A young \\
	   family enjoys feeling ocean waves lap at their feet. \textcolor{blue}{As}\_\_\\
	   \textbf{condition:} Contradiction \\
	   \textbf{Output:}  The waves are crashing onto the beach..\\
	   \textbf{Classifier output:} Neutral (Bad generation)\\
		\\
		SBERT + dynamic \\
		\textbf{Input:} \underline{A middle-aged woman in a dark bathing suit and} \\\underline{her middle-aged husband in an orange hat walk cozily} \\ \underline{along the beach. \textcolor{blue}{As} there is no water.} [SEP] A young \\ family enjoys feeling ocean waves lap at their feet. \textcolor{blue}{As}\_\_\\
		\textbf{condition:} Contradiction \\
		\textbf{Output:}  There is no sand.\\
		\textbf{Classifier output:} Contradiction (Good generation)\\
		\bottomrule
	\end{tabular}
	\caption{Case study for the dynamic method. The underline text is demonstrations. Template words are marked as blue.}
	\label{table:case}
\end{table}

\subsection{Generality test}
\label{sec:ga}
In order to evaluate the generality of our dynamic demonstration
method, we implement it based on state-of-the-art few-shot approach: LM-BFF
and develop experiments on 13 NLP classification tasks.
We use $K=10$ for all the tasks.
The LM-BFF implemented by ourselves uses the same setting in the original paper. In few-shot settings, the performance of model suffers from instability, and our implementing results are quite different from theirs\footnote{We directly run their released code from: \url{https://github.com/princeton-nlp/LM-BFF}}, so we only compare with our implementing results. As shown in table \ref{table:generazation}, our dynamic demonstration method outperforms the static method on 10 out of 13 tasks, which suggests its strong generality.

Besides, during the test, LM-BFF samples demonstration sets from $D_{train}^{'}$ and ensemble the predictions across all sets, while for our dynamic demonstration, we abandon it because our dynamic demonstration method has the ability to tune hidden vectors on $D_{train}^{'}$ during training, and the model is expected to have the best result with the top-1 example, so there is no need to ensemble model to make the results more stable.

\begin{table}[!h]
	\centering
	\small
	\begin{tabular}{l|cc}
		\toprule
		 & \textbf{Training} & \textbf{Inference}\\
		\midrule
		LM-BFF  & 10x &  1x \\
		LM-BFF+dynamic   & 1x &  16x \\
		\bottomrule
	\end{tabular}
	\caption{Training and inference speed comparison}
	\label{table:time}
\end{table}

Table \ref{table:time} is a comparison of training and inference speed. We can see our dynamic method takes more time to train dynamic demonstration while it takes less time to do inference. The training speed for dynamic demonstration directly depends on the hyper parameter $K$, and under the few-shot setting, a small $K$ guarantees the training speed would not be too slow.

\begin{table*}[th]
	\centering
	\small
	\begin{tabular}{l|ccccccc}
		\toprule
		 & \textbf{SST-2} & \textbf{SST-5} & \textbf{MR} & \textbf{CR} & \textbf{MPQA} & \textbf{Subj} & \textbf{TREC} \\
		\midrule
		\textbf{LM-BFF} (In paper)  & 93.0(0.6) & 49.5(1.7) & 87.7(1.4) & 91.0(0.9) & 86.5(2.6) & 91.4(1.8) & 89.4(1.7) \\
		\midrule
		\textbf{LM-BFF} (Our implement)   & \textbf{93.0}(0.5) & 50.4(1.2) & 87.3(1.9) & 90.7(1.1) & 86.2(1.8) & 90.4(2.3) & 86.2(3.9)\\
		\textbf{LM-BFF}$+dynamic$ & 91.8(0.9) & \textbf{50.6}(1.3)& \textbf{88.0}(1.1) & \textbf{91.2}(0.8) & \textbf{86.7}(1.7) & \textbf{91.2}(1.7) & \textbf{87.3}(3.0) \\
		\midrule
		 & \textbf{CoLA} & \textbf{SNLI} & \textbf{QNLI} & \textbf{RTE} & \textbf{MRPC} & \textbf{QQP} & \textbf{-}\\
		\midrule
		\textbf{LM-BFF} (In paper)  & 21.8(15.9) & 77.5(3.5) & 68.5(5.4) & 71.1(5.3) & 78.1(3.4) & 67.7(5.8) &\textbf{-}\\
		\midrule
		\textbf{LM-BFF} (Our implement)   & \textbf{20.4}(14.4)& \textbf{80.5}(1.7)& 70.5(4.8) & 69.0(3.3) & 75.0(4.5) &58.6(6.4)&\textbf{-}\\
		\textbf{LM-BFF} $+ dynamic$ & 17.7(17.5) & 78.9(2.4) & \textbf{71.9}(3.7) & \textbf{72.5}(3.7) & \textbf{77.4}(3.8) & \textbf{66.4}(7.4) &\textbf{-}\\
		\bottomrule
	\end{tabular}
	\caption{Test dynamic demonstration method on 13 NLP classification tasks. }
	\label{table:generazation}
\end{table*}

\section{Introduction}
Natural Language Inference (NLI), also known as the Recognizing Textual
Entailment (RTE) is a task of determining whether a text premise $p$
can entail, contradict or be neutral with a given hypothesis $h$~\citep{DBLP:conf/mlcw/DaganGM05}.
This task servers as an important benchmark for testing model's reasoning
ability.

Although NLI is proposed as a classification problem,
it's also worth considering it as a generation task. Recently,
several works reformulate NLI task as a generation task~\citep{DBLP:journals/corr/KolesnykRR16,DBLP:journals/csl/StarcM17,DBLP:journals/corr/abs-1803-02710,DBLP:journals/corr/abs-1909-09788}.
They explore how to generate the hypothesis $h$ given
a premise $p$ and a logic condition $c$ or reversely generate $p$ given $h$ and $c$.
\citet{DBLP:journals/corr/KolesnykRR16} show that the task of generating
entailed sentences can be served as a benchmark to measure the reasoning
ability of sequence to sequence models.
\citet{DBLP:journals/csl/StarcM17} first use it for data augmentation. They use generated hypotheses to construct a new NLI style dataset, on which a better classifier can
be trained.

\begin{figure}[t]
	\centering
	\scalebox{1.0}{\includegraphics[width=1.0\columnwidth]{figure/intro_prompt.pdf}}
	\caption{The difference between how standard fine-tuning methods and prompt-based methods treat the NLI generation task. The underlined text is a demonstration. The Green words are condition-specific templates.} \label{fig:intro_prompt}
\end{figure}

However, the success of these works heavily depends on the
SNLI dataset~\citep{DBLP:conf/emnlp/BowmanAPM15}, which is a manually created
dataset. Thus it's a challenge to use the proposed approaches in specific domains or real-world applications which are different from the SNLI domain,
and where there is not enough annotated data. In light of this,
we want to explore how to improve the performance of NLI generation task
in a few-shot setting.

Recently prompt-based learning has led to large improvements in NLP tasks under zero-shot or few-shot settings~\citep{liu2021pre,DBLP:conf/emnlp/PetroniRRLBWM19,DBLP:conf/eacl/SchickS21,DBLP:conf/emnlp/ShinRLWS20}. The main idea is to reformulate the downstream tasks as the pre-training task of pre-trained Language models (PLMs) using a prompt template. \citet{DBLP:conf/nips/BrownMRSKDNSSAA20} show that given a few demonstrations of inputs along with the prompt, GPT-3 achieves near state-of-the-art results in some SuperGLUE tasks.
Inspired by their success, we investigate how to adapt prompts and demonstrations to NLI generation task, which is reformulated as shown in \figref{fig:intro_prompt}(b).

Finding the right prompt is the first stage.
The problem of generating candidate templates and choosing the best one on the development set method proposed by \citet{DBLP:conf/acl/GaoFC20} is that in NLI generation task, we need to induce different templates for different conditions which makes the result on the development set less reliable,
since we only use a subset of the data to measure each template.
To ameliorate this, we propose a max-margin strategy to pick the template that
receives a high score in its corresponding condition but low scores
in all other conditions, considering the conditions in NLI task are conflicting with each other.

The second stage is to sample demonstrations. \citet{DBLP:conf/acl/GaoFC20} and \citet{DBLP:journals/corr/abs-2101-06804} simply sample them by the static cosine similarity based on models trained on related datasets (this model is regarded as a \textit{retriever}). This approach is suboptimal as:
(1) it doesn't make use of the training data;
(2) in cases of a low-resource language, we
do not have sufficient labeled data to train such a retriever.
To address these issues,
we propose a dynamic method that combines
the probability from the generator and the probability from the retriever to help the retriever fit target tasks.

In summary, our contributions are:
(1) To the best of our knowledge, we are the first to investigate NLI generation task in a few-shot setting.
(2) We propose a max-margin template selection method and a dynamic
demonstration selection method. Combining these two methods with PLMs,
our LM-PDD model gains 8\% absolute improvement over the standard fine-tuning models.
(3) We test our demonstration selection method on
13 natural language classification tasks in few-shot settings,
the results show our method has strong generality.

\section{Methodology}

The architecture of our model is depicted in Figure \ref{fig:model}. In this section, firstly we give out the hand-crafted templates. Then we introduce our prompt selection method and dynamic demonstration strategy.

\subsection{Hand-crafted Templates}

\begin{table}[!h]
	\centering
	\small
	\begin{tabular}{l|l}
		\toprule
		\textbf{Condition} & \textbf{Template}\\
		\midrule
		Entailment  & [$p$] Then [$h$]  \\
		Neutral   & [$p$] Maybe [$h$]  \\
		Contradiction   & [$p$] But [$h$]  \\
		\bottomrule
	\end{tabular}
	\caption{Hand-crafted templates used in our experiments}
	\label{table:manua}
\end{table}
Manually defining the templates for each condition requires domain expertise knowledge. Table \ref{table:manua} shows our manually defined templates, which is different from the templates defined for NLI classification task~\citep{DBLP:conf/eacl/SchickS21}. We design them based on our knowledge, intuition, and some simple tests on the $\mathcal{L}$ we used.

\subsection{Max-margin Template Selection}

First we adapt the method in section \ref{sec:agt} so that it works on conditional generation problem in our task. For an input sample $s_i = (p^i, c^i, h^i) \in \mathcal{D}_{train}$, we build the filled prompt as:
\begin{align*}
	\mathcal{T}_g (s_i)\ =\ [p^i]\ \mbox{[MASK]}\ [h^i]
\end{align*}\label{eq:t5}
Since we need to define a different template for each condition $m \in \mathcal{C}$, equation \ref{eq:decodet_old} is modified as:
\begin{align}
	\sum_{s_i\in\mathcal{D}_{train}, c^i=m}log P_{T5}(\mathcal{T}_m | \mathcal{T}_g (s_i))
	\label{eq:decodet}
\end{align}
where $\mathcal{T}_m$ is the template used for samples whose condition is $m$. To pick the best template $\mathcal{T}_m^{'}$, we use:
\begin{align}
	\mathcal{T}_m^{'}=\mathop{\arg\max}\limits_{\mathcal{T}_m}\ S_{\mathcal{L}}(\mathcal{D}_{train}^m, \mathcal{D}_{dev}^m, \mathcal{T}_m)
	\label{eq:best}
\end{align}
where $\mathcal{D}^m=\{s_i|s_i\in\mathcal{D}, c^i=m\}$. We call this method as $top$ template selection.

Experiments in~\citet{DBLP:conf/acl/GaoFC20} show the size of $\mathcal{D}_{dev}$ used in equation \ref{eq:best_old} will significantly influence the quality of chosen templates, while we only use $1/|\mathcal{C}|$ samples in $\mathcal{D}_{dev}$ to measure each template in equation \ref{eq:best}.

To address this issue, we propose our max-margin template selection method. Considering the conditions in NLI generation task conflicting with each other, we can give such assumption: A good template should not get high scores in other conditions. For example, the template $[p]\ But\ [h]$ designed for ``contradiction'' are supposed to achieve a bad performance in ``entailment'' and ``neutral''. Based on this, we refine function \ref{eq:best} as:
\begin{align}
	\mathcal{T}_m^{'}=&\mathop{\arg\max}\limits_{\mathcal{T}_m}\ \sum_{k\in\mathcal{C}} d_{m,k}\cdot S_{\mathcal{L}}(\mathcal{D}_{train}^k, \mathcal{D}_{dev}^k, \mathcal{T}_m) \notag\\
	&\mbox{where,}\ d_{m,k} =
	\begin{cases}
		1,  & \mbox{if }m=k \\
		-1, & otherwise
	\end{cases}
	\label{eq:mm}
\end{align}

\subsection{Dynamic Demonstration}

For classification problem, \citet{DBLP:conf/acl/GaoFC20} sample one example for each class. In this task, we only sample one example with the same condition as the demonstration, because templates for each condition are totally different from each other, and mixing templates as an input will mislead $\mathcal{L}$.

We call the demonstration method in section \ref{sec:staticd} as static demonstration since the retriever is not changed in the whole experiments and the embedding for each sample is fixed. As we have mentioned in section \ref{sec:staticd}, a model fine-tuned on task-related datasets performs better, we wonder can we train the retriever based on $\mathcal{D}_{train}$ which has no similarity labels?

One solution is to annotate the similarity score for each pair. But it's unrealistic as:
(1) It's hard for a human to design the similarity boundary;
(2) the number of desired annotations is quite large even in a few-shot setting since it's square to $\left| \mathcal{D}_{train}\right|$.

Inspired by the document retrieve method from RAG~\citep{DBLP:conf/nips/LewisPPPKGKLYR020} and prompt ensemble method from~\citet{DBLP:journals/tacl/JiangXAN20}, we propose $dynamic$ demonstration method by modifying the model objective $P(h^i | p^i, c^i)$ as:
\begin{align}
	&\prod_{j}^{N} \sum_{k}P_{\theta_{r}}(s_k | s_i) P_{\theta_{lm}}(h_j^i | x_{prompt}^i, s_k, h_{1:j-1}^i)\label{eq:finalobj} \notag\\
	&\mbox{where,}\ P_{\theta_{r}}(s_k | s_i) = \frac{exp(Sim(s_i, s_k)}{\sum_{t} exp(Sim(s_i, s_t))}
\end{align}
Here $s_k, s_t\in \mathcal{D}_{train}^{c^i}$. Thus the retriever's parameters $\theta_r$ are optimized to maximize the similarity score of the input sample $s_i$ and the sample which leads to the largest generator probability of the ground truth.

During training, calculating $P_{\theta_{r}}(s_k | s_i)$ over the whole $\mathcal{D}_{train}^{c^i}$ is costly, so we do a top-k approximation. For each sample $s_i$:
(1) First we calculate $P_{\theta_{r}}(s_k | s_i)$ over $\mathcal{D}_{train}^{c^i}$ and choose the samples with top-k probability;
(2) these chosen samples make up $\mathcal{D}_{i}$, then we calculate equation \ref{eq:finalobj}, where $s_k, s_t\in \mathcal{D}_{i}$.

During the test, we only use the retriever to sample the top-1 example as the demonstration, which is different from the decoding strategy used in RAG. Because RAG retrieves context from a corpus of 21M documents, where a lot of documents with relevant information can be found, while in a few-shot setting, the size of $\mathcal{D}_{train}$ is quite small.

\section{Preliminaries}

In this section, we first show how to fine-tune models with prompts and demonstrations. Then we introduce the automatic template generation method and demonstration strategy proposed in LM-BFF~\citep{DBLP:conf/acl/GaoFC20}.

\subsection{Prompt-based Fine-tuning with Demonstration}

Starting from the GPT model~\citep{radford2018improving}, pre-train and fine-tune paradigm becomes the mainstream method. To solve this task, we can use an autoregressive language model $\mathcal{L}$ (eg., GPT-2~\citep{radford2019language}, BART~\citep{DBLP:conf/acl/LewisLGGMLSZ20}). For each sample $s_i=(p^i, c^i, h^i)$, we take input $x^i$ as:
\begin{align*}
	x^i\ =\ [c^i]\ \mbox{[SEP]}\ [p^i]
\end{align*}\label{eq:fine}
$\mathcal{L}$ is fine-tuned to maximize the objective:
\begin{align}
	P(h^i | p^i, c^i) = \prod_{j}^{N} P_{\theta_{lm}} (h_{j}^i | x^i, h_{1:j-1}^i)
	\label{eq:obj}
\end{align}
However, this paradigm is hard to learn from a small amount of data.

Prompt is a possible solution for this problem. We can reformulate the task with a template so that the input sentences are closer to the inputs during pretraining. For a sample $s_i=(p^i, contradiction, h^i)$, we can create a prompt as:
\begin{align*}
	x_{prompt}^i\ =\ [p^i]\ But\ \_\_\_
\end{align*}\label{eq:exp}
$\mathcal{L}$ will fill in the blank based on the prefix, which is the same as the pretraining task. We use the template word ``But'' here, because we hope it can direct $\mathcal{L}$ to generate the rest part with opposite meanings.
As showed in \ref{fig:intro_prompt}(b), we can also add some demonstrations in the prompt $x_{prompt}^i$ as:
\begin{align*}
	[p^{(1)}]\ But\ [h^{(1)}]\ \mbox{[SEP]}\ ...\ \mbox{[SEP]}\ [r^i]\ But\ \_\_\_
\end{align*}\label{eq:expdemo}
These demonstrations are selected from $\mathcal{D}_{train}$.

\subsection{Automatic generation of template}\label{sec:agt}

\citet{DBLP:conf/acl/GaoFC20} generate templates automatically with a T5 model~\citep{DBLP:journals/jmlr/RaffelSRLNMZLL20} for classification problems.

For an input sample $s_i=(x_i, y_i) \in \mathcal{D}_{train}^{'}$\footnote{ $\mathcal{D}_{train}^{'}$ represents the training dataset for classification problems in their paper. We use symbol $'$ to distinguish it from the previously defined $\mathcal{D}_{train}$}, they build three different kinds of filled prompts $\mathcal{T}_g (s_i)$ whose template words are replaced by a [MASK] token.
Then a T5 model is used to fill in the [MASK] part. Because T5 is pre-trained to fill in the missing spans, there is no need to specific the number of [MASK] tokens here, which is more convenient than previous gradient-based prompt search methods~\citep{DBLP:conf/emnlp/WallaceFKGS19,DBLP:conf/emnlp/ShinRLWS20}. The goal is to find the template $\mathcal{T}$ which maximizes:
\begin{align}
	\sum_{s_i\in\mathcal{D}_{train}^{'}}log P_{T5}(\mathcal{T} | \mathcal{T}_g (s_i))
	\label{eq:decodet_old}
\end{align}

During the decoding, a wide beam search (e.g., 100) is used to obtain a large set of diverse templates. The reason why they need so many templates here is that the templates are affected by the architecture and pre-training tasks of the PLMs. Theses templates generated by T5 may not work on the $\mathcal{L}$ finally used. They fine-tuned each generated template on $\mathcal{D}_{train}^{'}$ and use $\mathcal{D}_{dev}^{'}$ to pick the best template $\mathcal{T}^{'}$:
\begin{align}
	\mathcal{T}^{'}=\mathop{\arg\max}\limits_{\mathcal{T}}\ S_{\mathcal{L}}(\mathcal{D}_{train}^{'}, \mathcal{D}_{dev}^{'}, \mathcal{T})
	\label{eq:best_old}
\end{align}
where $S_{\mathcal{L}}$ is a score function to measure performance using $\mathcal{L}$.

\subsection{Demonstration with Similarity}\label{sec:staticd}
To get the demonstrations, GPT-3 uses a random strategy to drawn from $\mathcal{D}_{train}$.  ~\citet{DBLP:conf/acl/GaoFC20} and ~\citet{DBLP:journals/corr/abs-2101-06804} use a BERT~\citep{DBLP:conf/naacl/DevlinCLT19}/RoBERTa~\citep{DBLP:journals/corr/abs-1907-11692} based retriever to sample examples with similarity. Their experiment results show (1) controlling the examples used in prompt is crucial for the model's performance; (2) using a model fine-tuned on task-related datasets is better than the original pre-trained
model.

They use SBERT~\citep{reimers-2019-sentence-bert}, which is fine-tuned on a large and diverse dataset, to build the retriever. The same SBERT model is used to encode both input samples and candidate demonstrations:
\begin{align}
	\textbf{emb}(s_i) = \mbox{SBERT}(p^i)
	\label{eq:emb}
\end{align}
The similarity is judged by cosine similarity:
\begin{align}
	Sim(s_i, s_j) = \frac{\textbf{emb}(s_i)\cdot \textbf{emb}(s_j)}{\left \| \textbf{emb}(s_i)\right \|_2\cdot \left \| \textbf{emb}(s_j)\right \|_2}
	\label{eq:sim}
\end{align}
The demonstrations examples for $s_i$ are sampled from the top 50\% instances.

\section{Related Work}

\textbf{NLI Generation}
~\citet{DBLP:journals/corr/KolesnykRR16} first proposed to serve NLI task as a generation task, while they only concentrated on entailment condition. Later ~\citet{DBLP:journals/csl/StarcM17} added two other conditions into this task to make it a conditional generation task. Our task definition follows many of their concepts except that we pay attention to few-shot settings. The previous works are all based on RNN architecture with attention mechanism, while in our work we use a stronger pre-trained language model as a baseline. There are some works exploring the inversion style of NLI generation task, which is to predict the premise based on the hypothesis and the condition. \citet{DBLP:journals/corr/KolesnykRR16} focused on entailment inversion style. They found that the model is learned to add more detailed information to the premise. \citet{DBLP:journals/corr/abs-1803-02710} used all three conditions to learn the conditional latent space over the representations of a logical antecedent of the given statement.

\noindent
\textbf{Prompt in Natural Language Generation}
Prompt-based methods is usually applied to NLG tasks by using prefix prompts together with autoregressive pre-trained LMs. \citet{radford2019language} demonstrated with prompts such as ``translate to french, [x], [z]'' or ``TL;DR'', and found that pre-trained LMs have impressive ability on generation tasks such as machine translation and summarization in the zero-shot setting. Further, \citet{DBLP:conf/nips/BrownMRSKDNSSAA20} showed that prompts also performed well in few-shot settings. \citet{DBLP:journals/corr/abs-2012-11926, DBLP:journals/corr/abs-2106-10715, DBLP:conf/acl/LiL20, DBLP:journals/corr/abs-2107-03374} explored how to adapt prompts for few-shot text summarization, machine translation, data-to-text and code generation tasks.

\noindent
\textbf{Automatic prompt search}
The automatically selected prompts consist of discrete prompts and continuous prompts (also called soft prompts). For the discrete prompts: \citet{DBLP:journals/tacl/JiangXAN20} used a text corpus to mine templates based on given training samples. Given a seed prompt, various paraphrasing methods can be exploited for generating more candidate prompts~\citep{DBLP:journals/corr/abs-2106-11520, DBLP:conf/eacl/HavivBG21}. \citet{DBLP:conf/emnlp/WallaceFKGS19} first proposed to search prompts automatically based on gradient. \citet{DBLP:conf/acl/GaoFC20} and \citet{DBLP:journals/corr/abs-2102-12206} explored how to use pre-trained T5 model to generate templates, which is more convenient compared with previous methods since it doesn't need extra corpus or modification of the model. For the continuous prompts: \citet{DBLP:conf/acl/LiL20} first proposed to use trainable continuous task-specific hidden representation vectors as prompts. There are some works making use of discrete prompts, like initializing continuous prompts with discrete prompts and hybrid prompts~\citep{DBLP:conf/naacl/ZhongFC21, DBLP:conf/naacl/QinE21, DBLP:journals/corr/abs-2103-10385}.

\noindent
\textbf{Demonstration learning}
Demonstration learning is one of the methods to combine multi prompts. This method is first used by GPT series~\citep{radford2019language, DBLP:conf/nips/BrownMRSKDNSSAA20}. While in GPT models the demonstrations are selected randomly, researchers found that the selection with similarity would significantly improve the final performance~\citep{DBLP:conf/acl/GaoFC20, DBLP:journals/corr/abs-2101-06804}. \citet{DBLP:journals/corr/abs-2101-06804} and ~\citet{DBLP:conf/acl/KumarT21} also discovered that the order of prompts provided to the model has a great influence on the performance of the model. However, the above methods fail to select demonstrations dynamically.
\section{Problem Setup}

\subsection{Task Formulation}

The dataset $\mathcal{D}$ consists of premises $\mathcal{P}$, conditions $\mathcal{C}=\{entailment, neutral, contradiction\}$ and hypotheses $\mathcal{H}$. Each sample $s_i$ is a tuple $(p^i, c^i, h^i)$, where $p^i \in \mathcal{P}$, $c^i \in \mathcal{C}$ and $h^i \in \mathcal{H}$. The NLI generation task is defined as: Given premise $p$ and condition $c$, predict hypothesis $h$. This formulation is the same as ~\citet{DBLP:journals/csl/StarcM17} except we assume there are only $K$
samples for per condition in training set $\mathcal{D}_{train}$ and development set $\mathcal{D}_{dev}$. This is following the few-shot settings in ~\citet{DBLP:conf/acl/GaoFC20}.

\subsection{Evaluation protocol}

Previous works show the performance of models fine-tuned on the small dataset is largely affected by the data split and hyper-parameters~\citep{DBLP:journals/corr/abs-2002-06305,DBLP:conf/iclr/0007WKWA21}.
To address this issue, we use the same evaluation protocol in
\citet{DBLP:conf/acl/GaoFC20}. We measure the average performance on
5 different randomly sampled $\mathcal{D}_{train}$ and $\mathcal{D}_{dev}$
splits. The hyper-parameters are chosen by the performance in
$\mathcal{D}_{dev}$ for each split.

\begin{figure*}
	\centering
	\scalebox{1.0}{\includegraphics[width=1.7\columnwidth]{figure/model.pdf}}
	\caption{An illustration of our LM-PDD. The blue dash line is the first stage: prompt template auto-generation, while the dark line stands for the dynamic demonstration training in stage two.}
	\label{fig:model}
\end{figure*}


\section{Introduction}

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{figures/setup_v4.pdf}
    \caption{Experimental Setup. From a large original text dataset (left), we draw 90\% of our desired training dataset in a non-repeated fashion, and 10\% as repeats of a tiny portion of the original dataset (right). We hold constant that 10\% of total training tokens will come from repeats, but we vary the repeated fraction in our runs. In other words, the sample to be repeated might be very small, like 0.01\% of the total training tokens repeated 1000x, or relatively large, like 1\% of the total training tokens repeated 10x. A small, held-back portion of the original dataset (yellow in left figure), not including any repeated data, is used as a test set and is the test loss reported in all subsequent figures.
    }
    \label{fig:setup}
\end{figure}
Large, high-quality text datasets are crucial for training large language models \cite{https://doi.org/10.48550/arxiv.2005.14165, https://doi.org/10.48550/arxiv.2112.11446}. Such datasets often contain many copies of substantially overlapping documents, which greatly impairs the performance of language models on downstream tasks \cite{lee2021deduplicating}. However, it is not well understood why data repetition impacts performance to such a large extent.

In this paper we study data repetition in language models through two lenses: the macroscopic lens of scaling laws, and the microscopic lens of mechanistic interpretability \cite{elhage2021mathematical, olsson2022context}. For the first lens, we trained transformer \cite{https://doi.org/10.48550/arxiv.1706.03762} language models on mostly unique data plus a small fraction of repeated data (Figure \ref{fig:setup}), varying the repeated dataset size, model size, and fraction of tokens trained on repeated data. We find a strong double-descent phenomenon \cite{https://doi.org/10.48550/arxiv.1710.03667, https://doi.org/10.48550/arxiv.1812.11118, https://doi.org/10.48550/arxiv.1912.02292}, such that there is a defined range of repetition frequency for which performance is harmed to a surprisingly large extent. We suspect there is a range in the middle where the data can be memorized \it and\rm\space doing so consumes a large fraction of the model's capacity, and this may be where the peak of degradation occurs. The location of the region suggests that large models like GPT-3, Gopher, and PALM \cite{https://doi.org/10.48550/arxiv.2005.14165, https://doi.org/10.48550/arxiv.2112.11446, https://doi.org/10.48550/arxiv.2004.07159} need to be careful about overfitting their high quality distributions like Wikipedia and books.

For the second lens, mechanistic interpretability (attempting to reverse engineer the detailed computations performed by the model) we show that repeated data disproportionately damages induction heads. Induction heads use a circuit of 2 attention heads to "complete the pattern by copying and completing sequences" \cite{olsson2022context}. The damage to induction heads is observed through degradation in copying, prefix matching, and through inspection.

Together, the two lenses provide an integrated picture of how repeated data might be causing the network (or part of it) to shift from generalization to memorization, and mechanistically how this could be harming performance of the overall language model.

\subsection{Summary of Results}

\begin{figure}
    \centering
    \includegraphics[width=0.49\textwidth]{figures/double_10_v2.jpg}
    \includegraphics[width=0.49\textwidth]{figures/overfitting_coincides.pdf}
    \caption{Models of different sizes show a degradation in performance at a specific range of repeats that shrinks with model size (left panel). At its peak the degradation sometimes reaches the equivalent of a 2x decrease in model size. The right panel shows that divergence (blue line) from a healthy, straight scaling law (red) lines up with when the models start to dramatically overfit the repeated subset (green curve). The blue line on the right corresponds to a vertical slice of models in the left diagram trained on the repeated subset for 120 epochs. All these models were trained on 90\% unique data and 10\% repeated tokens.}
    \label{fig:double_descent}
\end{figure}

To systematically study repeated data, we trained transformer \cite{https://doi.org/10.48550/arxiv.1706.03762} language models on mostly unique data plus a small fraction of repeated data (Figure \ref{fig:setup}), varying the repeated dataset size, model size, and fraction of tokens trained on repeated data over 2-3 orders of magnitude. All models were trained for 100B tokens. We examined the resulting models using both scaling laws and mechanistic interpretability tools. Our main findings were as follows:

\begin{itemize}
\item \textbf{Repeated data induces a strong double-descent phenomenon} \cite{https://doi.org/10.48550/arxiv.1710.03667, https://doi.org/10.48550/arxiv.1812.11118, https://doi.org/10.48550/arxiv.1912.02292}, in which data repeated a few times does not cause much damage to language model performance, data repeated very many times also does not cause much damage, but there is a peak in the middle where damage is surprisingly large. For instance, when we train an 800M parameter transformer with 10\% of training tokens drawn from the repeated subset (yellow curve in Figure \ref{fig:double_descent}) we find the loss can be nearly as high as for the 340M parameter transformer (light green curve). We see an epoch-wise \cite{https://doi.org/10.48550/arxiv.1912.02292} double descent learning curve in Figure \ref{fig:dd_learning_curve} is driving this performance degradation. We suspect there is a range in the middle where the data can be memorized \it and\rm\space doing so consumes a large fraction of the model's capacity, and this may be where the peak of degradation occurs. Figure \ref{fig:double_descent} on the right shows that the peak performance hit coincides with where the train loss on the repeated data approaches zero, similar to previously observed double-descent phenomena. This also provides a practical diagnostic for when repeated data is likely to be harming the model.
\item \textbf{Repeated data can cause a divergence from power-law scaling.} For the blue curve in Figure \ref{fig:double_descent} right (122 repeated epochs), we see only a moderate impact to performance (line on log-log graph) until the model is scaled up to 100M parameters, after which we see a large divergence from power law scaling of cross entropy loss. Extrapolating the region of large degradation in Figure \ref{fig:poor_scaling} predicts meaningful degradation of repeating data only 2 times for large (GPT-3 size) models, though the region would be shifted if the models were trained to the compute optimal frontier \cite{https://doi.org/10.48550/arxiv.2203.15556}.
\item \textbf{Repeated data causes a disproportionately large performance hit to copying, a mechanism for in-context learning.} We constructed a simple copying eval, the loss on the first paragraph of Harry Potter copied 11 times. We observe that using 3\% repeated data at the worst number of repeated epochs caused up to a 3x reduction in effective model size (performance equal to model with 3x fewer parameters) on this task whereas it only caused at most a 15\% reduction in effective model size on test loss.
\item \textbf{The disproportionate performance hit to copying coincides with a disproportionate degradation of induction heads}. In line with \cite{olsson2022context} we evaluated the models on their prefix matching score, repeated sequences of random tokens and observed the degree to which attention heads attend to earlier tokens that are preceded by a token that matches the present token. We observe that using 3\% repeated data at the worst number of repeated epochs caused on average a 32\% reduction in effective model size on this task whereas it only caused at most a 15\% reduction in effective model size on test loss.
\item \textbf{Repeated text data causes a small but still disproportionate performance drop out of distribution, as measured by cross entropy loss on Python code.} Unlike our the Harry Potter copying and prefix matching evals we mostly see the performance drop with higher levels of repetition, 50-90\%.
\item \textbf{One and two-layer attention only models trained on repeated data are worse at exactly copying and fuzzily copying (for instance correctly predicting Dursleys given that Dursley has appeared previously) proper names on inspection.} When we inspect per tokens losses of smaller models we can see this degradation in a simple, understandable form of copying in a paragraph of text.
\item \textbf{Training on repeated Python code creates a similar behavior.} When training on Python we also observe a double descent phenomenon and a predictable poor performance region in terms of model size and repeated epochs, though the shape of both curves are somewhat different.
\item \textbf{Pre-training on repeated data damages models.} Pre-training with repeated data leads to worse performance than both training from scratch and fine-tuning from a control model pre-trained on the original text dataset. During fine-tuning, the repeated data model forgets the repeated dataset, so we consider the model pre-trained with repeated data to be strictly worse than the model fine-tuned from the unique dataset.
\end{itemize}

\section{Results}

\begin{figure}
    \centering
    \includegraphics[width=0.49\textwidth]{figures/dd_learning_curve.pdf}
    \includegraphics[width=0.49\textwidth]{figures/dd_plateau_curve.pdf}
    \caption{ Learning curves for test loss on 800M models with 90\% repeated data (left) and 50\% repeated data (right), each with varying numbers of repeats/sizes of the repeated fraction. The graph on the left shows characteristic double descent curves. Repeated epochs corresponds to the number of epochs on the repeated tokens, the rest of the data is seen only once. For several models, test loss drops as normal during the beginning of training, but then starts to rise during the middle of training before dropping again. In the graph on the right with only 50\% repeated data, we see that the double descent bumps have turned into long plateaus for highly affected models.}
    \label{fig:dd_learning_curve}
\end{figure}

\textbf{Repeated data induces a strong double descent phenomenon.}
The results from training models on different sizes, fractions of repeated data, and frequency of repeats are shown in Figures 2 and 3.  Figure 2 (left) shows that when we train on 10\% repeated data and vary the frequency of repetition (or equivalently the number of epochs of repeated data), there is a specific range of repetition frequency for which damage to model performance is maximized.  The range depends on the model size but for a 800M parameter model it occurs at roughly 100x repeats of 0.1\% of the data, and degrades performance nearly to that of a 340M parameter model.  This is a large degradation given that only 10\% of the data is repeated. The peak coincides with the advent of memorization on the repeated data (Figure 2 right) -- a possible indicator of a double descent phenomenon.

Figure \ref{fig:dd_learning_curve} shows learning curves for different repetition frequencies and for 50\% and 90\% of the data being repeated.  In the extreme case of 90\% repeated data and the correct frequency of repetition (100x-10,000x), we confirm the presence of a literal double descent curve in which the loss decreases, increases, and then decreases again (Figure \ref{fig:dd_learning_curve} left).  As we lower the fraction of repeated data to 50\%, the curve becomes a long plateau rather than double descent, but it appears to be fundamentally an epoch-wise double descent phenomenon \cite{https://doi.org/10.48550/arxiv.1912.02292}. These peaks and plateaus again coincide with the training loss on the repeated data approaching zero as shown in Figure \ref{fig:double_descent}. As in \cite{https://doi.org/10.48550/arxiv.1912.02292} we see double descent effects caused by both increasing model size and epochs. We suspect there is a range in the middle where the data can be memorized \it and\rm\space doing so consumes a large fraction of the model's capacity, and this may be where the peak of degradation occurs, for a more thorough discussion of this question see the discussion (section \ref{discussion}).

\textbf{Repeated data can cause a divergence from power-law scaling.} Figure \ref{fig:poor_scaling} zooms in on the degradation of performance, measured as a function of model size for different repetition frequencies of the repeated data.  For example, models trained for 1,220 repeats and 10\% repeated data show a dip in performance to the equivalent of a model 0.55x as large, when the model size is 10M to 100M parameters.  As the model size continues to increase, performance recovers to 0.8x model-size equivalent for a 1B parameter model.  For a smaller number of repeats (122 repeats), the dip occurs later, centered around 1B parameters.

\begin{figure}
    \centering
    \includegraphics[width=0.49\textwidth]{figures/poor_scaling_emerge.pdf}
    \includegraphics[width=0.49\textwidth]{figures/poor_region.png}
    \caption{On the left we plot the same results as in Figure \ref{fig:double_descent}, re-parameterized in terms of the effective model size multiplier implied by the test loss (performance equal to a model with x times as many parameters). For a given number of repetitions, degradation occurs only for a specific range of model sizes. For example, for the blue curve (122 repeated epochs), we see almost no performance deviation from a power law scaling law (line on log-log graph) until the model is scaled up to 100M parameters, after which we see a divergence. We see the same divergence around 400M parameters for 12,200 repeated epochs. The right graph shows a large, predictable region over which the degradation occurs, and suggests that large models like GPT-3, Gopher, and PALM \cite{https://doi.org/10.48550/arxiv.2005.14165, https://doi.org/10.48550/arxiv.2112.11446, https://doi.org/10.48550/arxiv.2004.07159} need to be careful about overfitting their high quality distributions like Wikipedia and books -- although note that this holds constant the number of total training tokens. The blue and green curves correspond to the right and left sides of the double descent region where we observe 50\% of the maximum effect. They are an aggregation of that curve for the scans where we trained on 3\%, 10\%, 20\%, 50\%, and 90\% repeated data. The details of both fits are in Appendix \ref{app:region_fits}. A large number of runs needed to be aggregated to produce a clean fit for region of reduced performance.}
    \label{fig:poor_scaling}
\end{figure}

The right panel of Figure \ref{fig:poor_scaling} shows the range over which we observe at least 50\% of the maximum degradation; this corresponds to a ``band'' or region in the (model size, repetition frequency) plane.  Both boundaries of the region are a good fit to a power law relating frequency of repetition to the number of parameters of the model, namely:

\[E = k * N ^ \alpha \]

where $E$ corresponds to epochs of repetition and $N$ corresponds to the parameters in the model. it is notable that the lines in figure 2b are relatively parallel. The fits for the above lines are given in the table below:

\begin{center}
\begin{tabular}{|c | c | c |}
 \hline
  & $k$ & $\alpha$ \\
 \hline
 right boundary & 5.1e7 & -.50 \\

 \hline
 left boundary & 4.2e6 & -.56 \\ [1ex]
 \hline
\end{tabular}
\end{center}

Note that extrapolating these boundaries leads to a prediction of significant degradation from repeating data as little as 2x on state-of-the-art language models with hundreds of billions of parameters, although this applies for a constant number of training tokens (100B). In practice large models are trained for more than this\cite{https://doi.org/10.48550/arxiv.2203.15556}, and as shown in Figure \ref{fig:dd_learning_curve}, training past the double descent peak is helpful, so the degradation would likely not be quite as bad. When looking at Figure \ref{fig:dd_learning_curve} we see that the the poor performance region would be shifted left for large models trained on the compute efficient frontier (the pareto frontier of compute and performance) \cite{https://doi.org/10.48550/arxiv.2001.08361}.

Overall it seems that in addition to being robust to task, model size, and architecture as shown in previous work \cite{https://doi.org/10.48550/arxiv.1710.03667, https://doi.org/10.48550/arxiv.1812.11118, https://doi.org/10.48550/arxiv.1912.02292} double descent as a general phenomenon appears to be robust to occurring in a sub-distribution and that it can have a large effect on overall performance even while being a modest fraction of training tokens.

\textbf{Repeated data causes a disproportionately large performance hit to copying, a mechanism for in-context learning}. The ability of a language model to copy text (in the sense of being provided with a context consisting of a passage repeated several times, and testing whether the model can repeat it once more) is a potential measure of \textit{generalization}, as copying is independent of the content of the text.  Also, recent interpretability work has suggested that copying may be implemented by crisp internal algorithmic structures (\cite{olsson2022context}), again suggesting generalization.  It thus seems valuable to investigate what happens to copying during a memorization-related degradation in performance, which we have shown above occurs in our experiments.

To do this constructed a simple evaluation in which copying is heavily emphasized: we measure the loss on the first paragraph of Harry Potter copied 11 times. The models trained on repeated data performed much worse on this evaluation (Figure \ref{fig:hp_copying_1}), \textit{substantially out of proportion to the degradation on the loss itself}.  In other words, copying is \textit{preferentially} harmed by training on repeated data.  For example, a 3\% fraction of repeated data leads to a 1.15x reduction in effective model size (performance equal to model with 1.15 fewer parameters) on the general loss, but a much larger 3x effective model size reduction in terms of copying ability.  As can be seen in Figure 5, the damage to copying is greater than the damage to overall loss across the entire range of repeated data fractions.  This suggests that the shift to memorization caused by repeated data is selectively harming at some behaviors associated with generalization.

To get another view on the same phenomenon, we measured the loss of various models on the $X$th consecutive copy of the Harry Potter paragraph, where $X$ runs from 1 to 12. As shown in Figure \ref{fig:hp_copying_2} (left), for most models the loss gradually decreases with increasing numbers of copies of the paragraph (i.e. the model has an easier time predicting an additional copies after seeing more consecutive copies), but at the peak of the double descent phenomenon, the loss is much higher and, strikingly, does not decrease at all with additional copies of the paragraph.  This large aberration shows how strong the selective effect of the double descent phenomenon on copying is.  General in-context learning is also harmed at the pessimal number of repeated epochs (Figure \ref{fig:hp_copying_2} right), though to a lesser extent than copying.

\begin{figure}
    \centering
    \includegraphics[width=0.49\textwidth]{figures/model_size_multipler_hp.pdf}
    \includegraphics[width=0.49\textwidth]{figures/loss_zoomed_multiplier.pdf}
    \caption{We constructed a simple measure of the model's copying ability, consisting of the loss on the first paragraph of Harry Potter repeated 11 times. We measured the double descent peak performance for a given model size and fraction of repeated data and compared that to a fit of these evaluations on the control model (trained on unique text) scan to generate an effective model size. We observe that 3\% repeated data at the pessimal number of repeated epochs caused a 3x reduction in effective model size on this task for a for several model sizes, whereas it only caused at most a 1.15x reduction in effective model size on test loss. We see much larger effects on the copying evaluation than on overall performance for repeated data fractions between 3\% and 20\%. The model size multiplier for copying is based on interpolation and the model size multiplier for test loss is based on a power law fit (see Appendix \ref{app:copy_prefix} for more details). }
    \label{fig:hp_copying_1}
\end{figure}

\textbf{The disproportionate performance hit to copying coincides with a disproportionate degradation of induction heads}.
Having connected the damage associated with repeated data with a measure of generalization (in-context copying of text), we next took the connection one step further, by trying to also probe the potential mechanistic basis of copying.  \cite{olsson2022context} identifies ``induction heads'' as a possible basis for copying and in-context learning behavior in general, so we decided to measure these and try to connect them back to the repeated data double descent phenomenon.

\cite{olsson2022context} defines induction heads by their ability to facilitate simple copying given a repeated random sequence of tokens (though in practice this definition ends up including heads with more complex behaviors too). Induction heads use a circuit of 2 attention heads to "complete the pattern by copying and completing sequences." This can be split up into attending to the relevant token (prefix matching) and increasing the logit corresponding to the attended-to token.

\begin{figure}
    \centering
    \includegraphics[width=0.49\textwidth]{figures/prefix_matching_score_multiplier.pdf}
    \includegraphics[width=0.49\textwidth]{figures/loss_full_multiplier.pdf}
    \caption{Comparison of degradation of prefix matching score with repeated data, compared to general degradation of the test loss.  We measured the double descent peak performance for a given model size and fraction of repeated data and compared that to a fit of the prefix matching score on the control model scan to generate an effective model size. We observe that 3\% repeated data causes on average \ref{fig:appendix_c_prefix} a 1.47 model size multiplier on prefix matching score while causing less than a 1.15x model size reduction in effective model size on test loss. Again we see much larger effects on the prefix matching score than on overall performance for repeated data fractions between 3\% and 20\%. The model size multiplier for prefix matching is based on a linear fit (see Appendix \ref{app:copy_prefix} for more details of fit). The test loss shown on the right is the same graph as in Figure 5, but with differently scaled axes for ease of comparison.}
    \label{fig:prefix_matching}
\end{figure}

We decided to probe the prefix matching score as measure of mechanistic structure that is distinct from the behavior of copying itself.  Figure \ref{fig:prefix_matching} shows the same setup as Figure \ref{fig:hp_copying_1} except for prefix matching score instead of copying loss.  As can be seen in the figure, preferential damage to prefix matching score is not present across the whole range of repeated data fraction as it is for copying, but at low fractions of data repeated, there is still preferential damage.  For example, at 3\% repeated tokens, there is a 2x effective parameter decrease in prefix matching score, but only a 1.15x effective parameter decrease in general (test) loss.

As another example, we find it interesting that the sharp drop in prefix matching score for a 1.5M parameter model with 50\% repetition corresponded to a complete breakdown of paragraph level copying.  This complete breakdown of paragraph level copying corresponds to a 1.5M parameter model having the effective overall performance of a 30,000 parameter model, while having an equivalent prefix matching score to a model with effectively 2,000 parameters.

Although not as conclusive as the previous results, these clearly show that prefix matching is preferentially degraded in some cases.

\begin{figure}
    \centering
    \includegraphics[width=0.49\textwidth]{figures/broken_copying.pdf}
    \includegraphics[width=0.49\textwidth]{figures/2_layer_per_token.pdf}
    \caption{Degradation of copying and in-context learning at the peak of the double descent curve. On the left we show the 2-layer models trained on 50\% repeated data from Figure \ref{fig:hp_copying_1}, evaluated on the first paragraph of Harry Potter copied X times where X runs from 1 to 11. In Appendix \ref{app:hp_fewer_characters}, we explore shortening the length of the paragraph to verify the problem is with copying rather than long contexts. The right shows per token losses on the test set. Both graphs show dramatically reduced performance (higher copying loss, lower benefit to in-context learning) at the peak of the double descent.}
    \label{fig:hp_copying_2}
\end{figure}

\textbf{One and two-layer attention only models are worse at copying and fuzzily copying proper names on inspection}. To examine the effect on induction heads and in-context learning even more closely, we looked at more granular copying in one and two layer attention-only transformers, for which interpreting the internal structure (and especially induction heads) is known to be particularly straightforward \cite{elhage2021mathematical, olsson2022context}. That is, we can reverse engineer a large portion of attention-only-transformers (no MLP's) with a circuits-level understanding (understanding how individual neurons act together to produce useful behavior) \cite{cammarata2020thread:}. These small models also exhibit the same double-descent phenomenon as larger models (Appendix \ref{app:logit_attribution}).

For 1-layer attention only models, where copying takes the form of skip-trigrams, we can easily see that the repeated data model is worse at a form of copying associated with these skip trigrams.  Namely, we compare the probabilities that the repeated data and control models assign to each token in a paragraph, and focus especially on proper names which occur repeatedly in the paragraph (Figure \ref{fig:1l_attn_hp}).  The most obvious way to correctly predict these re-occurring names is by copying, and we see that in most cases the control model (trained on unique text) performs much better than the one with repeated data (yellow underlines).

\begin{figure}
    \centering
    \includegraphics[width=1.0\textwidth]{figures/1l_attn_hp.jpg}
    \caption{Visualization of the difference in loss on the first paragraph of Harry Potter for control and 10\%-repeated-data runs of a 1-layer attention-only model. Orange highlights correspond to the control model performing better, purple corresponds to the repeated data performing, and the intensity corresponds to the magnitude of the difference in per token losses.  Proper names (which are a good target for copying when they occur more than once) are underlined in yellow on second or later occurance; it is clear that the control model performs better on these. Often the difference is dramatic: for the last three appearances of ``Potters'' the control model puts a >97\% chance on ``ters'' given ``Pot'', whereas the repeated data model puts <4\% chance on that token.}
    \label{fig:1l_attn_hp}
\end{figure}

Very specifically, predicting repeated names requires exactly a skip-trigram pattern \cite{elhage2021mathematical} which is the algorithmic operation 1-layer attention-only models are known to perform. For example, the following skip-trigrams are useful in the Harry Potter paragraph in Figure \ref{fig:1l_attn_hp}:

\[[a][b] \ldots [a] => [b] \;\;\;\;\;\;\;\; [\;Pot][ter] \ldots [\;Pot] => [ter] \]
\[[a][b] \ldots [a] => [b^\prime] \;\;\;\;\;\; 	[\;Pot][ter] \ldots [\;Pot] => [ters] \]
\newline

We also plotted the same visualization for a 2-layer attention-only model (which is known to contain simple induction heads), and find the control model is better at fuzzy copying (Figure \ref{fig:2l_attn_hp}).

\begin{figure}
    \centering
    \includegraphics[width=1.0\textwidth]{figures/2l_attn_per_token.jpg}
    \caption{Same as Figure 9, but for 2-layer attention-only models. Proper names (which are a good target for copying when they occur more than once) are underlined in yellow on second or later occurance. Here the repeated-data model sometimes does better on repeated proper names, but there are still clear examples of the control performing much better. These examples are highlighted in green and discussed. On the token [ley] in the second appearance of [D][urs][ley] the control model places a 92\% likelihood on [ley] whereas the repeated data model places a 10\% likelihood. On the token [leys] in the second appearance of [D][urs][leys] the control model places a 44\% likelihood on [leys] whereas the repeated data model places a 4.9\% likelihood. On the [ley] in [ un][D][urs][ley][ish] the control model places a 68\% likelihood on [ley] whereas the repeated data model places a 0.4\% likelihood.
}
    \label{fig:2l_attn_hp}
\end{figure}

Visually, it is less obvious (compared to the 1-layer case) that the 2-layer repeated model is worse at names, and there are a few examples where it puts 1.1x higher odds on the correct token. But on the other hand there are dramatic cases of the control model doing 500x times better (odds ratio on correct token) for fuzzy copying, like unDursleyish, which is exactly the kind of degradation we'd expect to see from disrupting induction heads.

We attempted to leverage logit attribution (which earlier tokens contributed to the prediction of the current token through a "direct path" with this attention head) to see if the difference was primarily due to the induction head being less active or other heads interfering with it \cite{olsson2022context}. We were unable to find clear evidence of either, but we include our exploration of a 2 layer attention only model in Appendix \ref{app:logit_attribution}.

\textbf{Repeated data causes a smaller, disproportionate performance drop on our out-of-distribution evaluations.}

\begin{figure}
    \centering
    \includegraphics[width=0.49\textwidth]{figures/py_text_multiplier.pdf}
    \includegraphics[width=0.49\textwidth]{figures/py_text_multiplier_2.pdf}
    \caption{We observe that training on high levels of repeated data causes a small disproportionate drop on out-of-distribution performance (Python loss). The effect is noisy, but since we do not see a model size effect we take the average in the figure on the right (harmonic mean of multipliers). For large repeated fractions of 50\% and 90\% we see model size multipliers of .84 and .75.}
    \label{fig:ratio_py_text}
\end{figure}

Given that we overfit the model, we expected it to perform worse off distribution, which we do observe (Figure \ref{fig:ratio_py_text}). We notice almost an opposite pattern to what we observed in the induction head results. We see most of the disproportionate drop at 50\% and 90\% rather than 1-10\%.

\textbf{We observe a double descent phenomenon in sparse sweep of models trained on python, but we the Python scans exhibit a somewhat different overall shape}. To add more generality to our results, we repeated the same experiments on a Python dataset instead of natural language (Figure \ref{fig:py_double_descent}).  If we use the same method to fit the poor performance region, we see a broadly similar fit and a second epoch for today's large models (approximately 200B parameters) is still robustly in the reduced performance region for python. However the fit is noisier than the fit for text and the two lines are no longer parallel.

\begin{figure}
    \centering
    \includegraphics[width=.49\textwidth]{py_double_descent.pdf}
    \includegraphics[width=.49\textwidth]{poor_region_py.png}
    \caption{Double descent phenomenon for models trained on python. Training on Python gives similar results to what Figure 2 and Figure 4 show for language models.  Here 50\% of the dataset consists of repeats and 50\% is unique. On the left side is degradation in performance, occurring over a specific range of repetition that varies with model size.  On the right, we again see a large region of poor performance as we did in Figure \ref{fig:poor_scaling}, although the fit is noisier. Again the blue and green curves correspond to the right and left sides of the double descent curve where we observe 50\% of the maximum effect.}
    \label{fig:py_double_descent}
\end{figure}

The noise may partially be explained by the Python fits being averaged over half as many settings for the fraction of tokens that are repeated data. It could also be that we need a higher resolution Python scan to get a cleaner estimate for the poor performance region. Finally, the Python data was trained on approximately 2 epochs as described in the methods section (so it included some repetition on the main dataset as well, not just the repeated subset). Python also may have more unintentional repetition than text, from copying and pasting of example code and forking of codebases. Such repetition could change the shape of the region of poor performance. More analysis of the Python experiments is shown in Appendix \ref{app:region_fits}.

\textbf{Pre-training on repeated data hurts fine-tuned performance} We find that the negative impact of repeated data persists after fine-tuning natural-language models on Python (Figure \ref{fig:ossification}).

\begin{figure}
    \centering
    \includegraphics[width=0.49\textwidth]{ossification.pdf}
    \includegraphics[width=0.49\textwidth]{dominant_2.pdf}
    \caption{Effect of repeated data during pre-training on fine-tuning.  Models were pre-trained on 90\% repeated data (red lines) or on totally unique data (blue lines), and then fine-tuned on Python (always unique data).  The repetition frequency was chosen to maximize the performance hit.  The model pre-trained on repeated data encounters a sizable performance hit during fine-tuning (left panel), causing it to not only perform worse than the model pre-trained on unique data, but also worse than a model trained from scratch (green line).  The right panel shows fine-tuning curves of the two models.  The model pretrained on repeated data performs much worse for several billion tokens (red line), but eventually catches up to the model pretrained on unique data (blue line).}
    \label{fig:ossification}
\end{figure}

It is noteworthy that the performance hit once fine-tuned is much smaller. An 800M model pre-trained on 50\% repeated data from the double descent peak had its effective parameters reduced by 10x in Figure \ref{fig:1_noisy} in Appendix \ref{app:region_fits}. When we fine-tune from the repeated model we see a 1.6x reduction in effective parameters compared to training from scratch. This is still meaningful damage to the model, but it is recovered substantially. Since the repeated model forgets the repeated dataset after a modest amount of fine-tuning (Figure \ref{fig:ossification}, we consider the fine-tuned model with repeated data pre-training to be dominated by the fine-tuned model from the unique dataset.

\section{Methods}
The decoder-only transformer models were trained on an 8192 token context with the same settings as described in \cite{https://doi.org/10.48550/arxiv.2112.00861} for 100B tokens. Our language experiments utilized a 400B token dataset with 55\% heavily filtered common crawl data (220B tokens), 32\% internet books (128B tokens), and some smaller distributions including OpenWebText, Wikipedia, and Stack Exchange; most of which we sourced from The Pile \cite{https://doi.org/10.48550/arxiv.2101.00027}, and leveraged the 50,304 vocabulary GPT-2 encoding \cite{radford2019language, https://doi.org/10.48550/arxiv.1910.03771}.

Code models were trained or fine-tuned on 45B tokens of Python for 2.2 epochs. Fine-tuning experiments had the same hyperparameters as pre-training experiments, but with learning rates reduced by a factor of 2 and reduced warmups.

We varied model size, repeated dataset size, and the fraction of tokens trained on repeated data by 3, 2.5, and 2 orders of magnitude respectively.

\section{Related Work}
\textbf{Scaling Laws}
\newline
A scaling law lens consists of finding a small set of hyperparameters that have large, predictable impacts on model performance, and was present throughout this work (at least one of the hyperparameters is generally model size, compute, or dataset size). The predictive nature of scaling laws makes them useful in a broad number of research and engineering settings. The implications of scaling laws are sufficiently broad and understandable that understanding them is relevant to policy makers \cite{https://doi.org/10.48550/arxiv.2202.07785}. Predictable scaling trends in neural networks
were first studied with \cite{https://doi.org/10.48550/arxiv.1712.00409}. \cite{https://doi.org/10.48550/arxiv.2001.08361} demonstrated that test loss performance on language
modeling tasks scales as a predictable function of model size, dataset size, and compute. The scaling law lens has become more popular over time. For instance scaling
laws have been shown in many modalities (e.g., images, video, math, etc.) \cite{https://doi.org/10.48550/arxiv.2010.14701}, acoustics \cite{https://doi.org/10.48550/arxiv.2106.09488}, transfer to code,
\cite{hernandez2021scaling}, and few-shot adaptation of vision models \cite{https://doi.org/10.48550/arxiv.2110.06990}. Existing scaling laws have been revisited as training setups change; for instance, \cite{https://doi.org/10.48550/arxiv.2203.15556} found that many recent large models have been under-trained. Our work uses the scaling law lens on an aspect of dataset quality and supplements the lens with an interpretability lens, and we believe our work is novel in both these respects.

\textbf{Mechanistic Interpretability}
\newline
A mechanistic interpretability lens was used in this work. Mechanistic interpretability refers to attempting to reverse engineer the detailed computations performed by the model. The mechanistic interpretability lens is useful for pure scientific understanding and has the potential to anticipate safety issues from future more powerful models. There is a relatively detailed understanding of mechanistic interpretability for convolutional image models \cite{cammarata2020thread:}, some understanding for multimodal models \cite{goh2021multimodal, https://doi.org/10.48550/arxiv.2103.00020}, and such an understanding is starting to be built up for Transformers trained on language \cite{elhage2021mathematical, olsson2022context}. For a more thorough background on interpretability progress see the related work section of \cite{elhage2021mathematical}. These results are an example of a ``bridge'' between microscopic phenomena inside the network and macroscopic trends in the loss, and we're only aware of one other example of such a bridge \cite{olsson2022context}.

\textbf{Double Descent}
\newline
Double descent was first shown in generality by Belkin et al. \cite{https://doi.org/10.48550/arxiv.1812.11118} where it was observed for decision trees, random features, and 2-layer neural networks. Similar behavior has been observed in \cite{article, NIPS2001_26f5bd4a, https://doi.org/10.48550/arxiv.1710.03667, geiger2019jamming, https://doi.org/10.48550/arxiv.1912.02292}. For a more thorough background on double descent see Nakkiran et al. \cite{https://doi.org/10.48550/arxiv.1912.02292}. We extend the double descent phenomenon to a setting we see as more practical since data repetition in various forms appears to be a universal, long-term issue; whereas modern large language models are generally outside of the parameters and data regime of previously observed double descent phenomenon.

\textbf{Rise of Engineering Large, Diverse Language Datasets}
\newline
Algorithmic innovation \cite{DBLP:journals/corr/abs-2005-04305}, compute \cite{amodei2018ai}, and data are three of the major factors that drive the advance of AI. The engineering and science of large, diverse language datasets is relatively new. Pre-2017 many language models were trained on a single distribution of text, such as news articles \cite{https://doi.org/10.48550/arxiv.1602.02410}, Wikipedia \cite{https://doi.org/10.48550/arxiv.1609.07843}, or fiction books \cite{https://doi.org/10.48550/arxiv.1506.06726}. GPT-2 \cite{radford2019language} leveraged webtext, outbound Reddit links with at least 3 upvotes in order to use human curation/filtration to ensure quality in addition to a broad distribution. GPT-2's capabilities are largely attributed to its scaled-up size and dataset (10x the parameters and 10x the data of GPT) \cite{radford2019language}. The next generation of language models, \cite{https://doi.org/10.48550/arxiv.2005.14165, https://doi.org/10.48550/arxiv.2112.11446, https://doi.org/10.48550/arxiv.2203.15556}, leveraged large, diverse datasets that consist of many sub-distributions. Constructing such datasets includes a large number of decisions: choosing sampling weights, quality filtering, de-duplication, fuzzy de-duplication, epochs per dataset, and more. There has not yet been substantial public work that quantitatively shows the impact of such decisions, but the dataset ablations in Appendix A of the Gopher \cite{https://doi.org/10.48550/arxiv.2112.11446} paper are notable. They clearly show the benefit of their dataset mixture, quality filter, exact de-duplication, and fuzzy de-duplication for 1.4B parameter models. Our work aims to provide some insights and potential diagnostics for researchers and engineers designing large datasets for language models.

\section{Discussion}
\label{discussion}
\subsection{Why does repeating a small fraction of data damage performance so much?}
We showed that a dataset with only 10\% repeated tokens can reduce model performance by an effective 2x in parameter count, much more than if that 10\% of the data had simply never been trained on. The repeated data thus degrades model performance out of proportion to its share in the dataset. Why does this occur, and why only for a specific amount of repetition?  One plausible hypothesis comes from looking at the model's ``incentives'' to memorize vs generalize. To informally explore this hypothesis consider the following rough numbers, a 800M parameter model typically has a loss of roughly 2.0 nats/token, a 400M parameter model has a loss of roughly 2.2 nats/token, and fully memorized data will have a loss of 0 nats/token. Now suppose a 800M model is trained on 90\% unique data and 10\% tokens consisting of repeated data. We can ask whether it is a ``good tradeoff'' for the model to memorize the repeated data (leading to 0 loss on 10\% of the dataset), at the cost of degrading performance by the equivalent of a 2x multiple in model size (which raises loss on the other 90\% from 2 to 2.2). Some simple arithmetic suggests that it is: $0.9 * 2.2 + 0.1 * 0 = 1.98 < 2.0$. Another way to say this is that zero loss is such a huge drop compared to the differences in entropy between model sizes that driving the loss to zero on even a tiny subset can incentivize enormous degradation in quality.

This however leaves open the question of when this tradeoff is necessary or possible -- and here is where the double descent phenomenon comes in. If a lot of data is repeated only a few times (say 5\% of the data repeated 2x) then the model may not have the capacity to memorize it, and also does not see it enough times during training to do so. If a tiny amount of data is repeated very many times (say 0.01\% of the data repeated 1000x), then the model will memorize it, but because it is so small the model need not use much capacity to do so, so the degradation in quality will likely be small. There is a range in the middle where the data can be memorized \it and\rm\space doing so consumes a large fraction of the model's capacity, and this may be where the peak of degradation occurs.

\subsection{Generalization, memorization, and induction heads}
Our results show that overfitting on the repeated data results in worse test loss, and this co-occurs with a disproportionate degradation in the model's induction heads (prefix matching score) and its ability to copy text. Copying sequences can be seen as a form of generalization, as it requires algorithmic operations that are independent of the content of the data. \cite{elhage2021mathematical, olsson2022context} provided evidence for induction heads as the mechanism implementing copying and other pattern-matching. For the 2 layer model shown in Figure \ref{fig:hp_copying_2} it seems as if the pressure to memorize the repeated dataset has led a skip tri-gram head to replace the induction head entirely. Thus our results tell a story where a type of generalization and its internal implementation are disrupted when the model memorizes repeated data -- a vivid illustration of the memorization-generalization trade-off. Future work could take this even further, by measuring the number of parameters devoted to memorization and trying to observe them competing for space with induction heads. Finally, it is worth noting that the co-occurence of copying degradation and induction head degradation is itself some additional evidence for induction heads as the source of in-context learning; Olsson et al. \cite{olsson2022context} was not fully conclusive and our results further bolster the case.

\subsection{Bridging mechanistic interpretability and scaling laws}
The results connecting memorization to the degradation of mechanistic interpretability structures \cite{olsson2022context} are an example of a ``bridge'' between microscopic phenomena inside the network and macroscopic trends in the loss. We view such connections as very fruitful tools for research, because they allow us to see the same thing through different lenses: the macroscopic behavior demonstrates the significance of the microscopic mechanisms, and the microscopic mechanisms help explain how and why the macroscopic phenomena occur. Switching back and forth between the two allows for a deeper understanding of both, as well as more robust diagnostics if something goes wrong. We are aware of at least one other instance of such a bridge -- the correspondence between the formation of induction heads and the boost in in-context learning near the beginning of training \cite{elhage2021mathematical, olsson2022context} -- but such connections remain rare so far, and we believe that finding more of them is a promising route to more deeply understanding neural nets.

\subsection{Repeated data and fine-tuning} We hypothesized repetition might help explain why models trained from scratch sometimes outperformed models that were pre-trained and then fine-tuned \cite{hernandez2021scaling}. For our purposes, we define ossification as any pre-training that leads a fine-tuned model to perform worse than a model trained from scratch (given a fixed compute and data budget). It required relatively extreme repetition in pre-training (90\% training on repeated tokens at  peak of double descent curve, 73x reduction in effective model size) to see a large ossification effect (1.6x reduction in effective model size) within our fine-tuning setup. We still think repetition might explain a large fraction of ossification when we consider training on various types of repetition we did not study here (sentence level, paragraph level, similar documents, distribution, etc). Overall, our finding that repetition can induce ossification provides medium causal evidence to this hypothesis. We think ossification is an interesting phenomenon that merits further study.

\subsection{Limitations}
We attempt to discuss limitations throughout the text where appropriate, but for the reader's convenience, we enumerate them here. We attempt to list them in a loosely descending order of importance.
\begin{enumerate}
    \item We used a fixed number of tokens for all models (similar to the GPT-3 model sweep), because these models were trained prior to the release of Chinchilla, which showed the compute frontier (pareto frontier of performance and compute) is quite different than previously understood \cite{https://doi.org/10.48550/arxiv.2005.14165, https://doi.org/10.48550/arxiv.2203.15556}.
    \item Our fits for region of poor performance were relatively noisy, and we only observed a clean trend by aggregating them. This is discussed in the Results section and further explored in Appendix \ref{app:region_fits}.
    \item The data we repeated was a random subset of the original dataset, and is thus not directly applicable to the situation where higher quality data (such as Wikipedia) is intentionally repeated to improve quality. Nevertheless, it seems plausible that the results would carry over.
    \item We measured loss, rather than downstream NLP evaluations. Overfitting does not always entail worse performance on downstream tasks \cite{https://doi.org/10.48550/arxiv.2203.02155}, so it is possible that the degradation we observe does not carry over to these tasks.
    \item We did not explore the effects of early stopping, dropout, weight decay, or other regularization.
    \item We did not investigate simpler systems than 1L attention-only models, which might contain more complete mechanistic insights.
\end{enumerate}

\subsection{Future Directions}
Below are some future directions we think are promising:
\begin{enumerate}
    \item A compute efficient frontier scan to predict the poor performance region.
    \item Varying the type of repetition. We could inject repeated sentences or paragraphs at the beginning or end of some fraction of contexts, or repeat chunks of documents in a different order. We could also explore cases where the repeated data has a different distribution than the unique data.
    \item Further interpretability work. Are there neurons that tell the model what distribution it is in: unique or repeated? Are there neurons through which we can observe and edit the repeated sequences?
    \item Drill down on memorization and generalization. Could we measure the number of model parameters taken up by memorization vs generalization, either behaviorally or by using mechanistic interpretability to identify parameters that are storing memorized data? Can we measure how this varies across the double descent, and thus watch the competition between memorized data and induction heads for model capacity?
    \item Could repetition and double descent help explain loss spikes during training? If a model can largely memorize a particularly easy batch in a single gradient step then a very skinny double descent could present as a loss spike.
\end{enumerate}

\section{Conclusion}
We've shown that small fractions of repeated data, if repeated at the right frequency, can cause surprisingly severe degradation to model performance. We show that this degradation scales predictably, occurs across datasets, and is associated with disproprotionate damage to internal mechanisms associated with generalization, such as induction heads. In practical terms, these results provide a tool for predicting and diagnosing data-repetition-related problems in language models. In more conceptual terms, they are an example of a bridge between the macroscopic domain of scaling laws and the microscopic domain of mechanistic interpretability, as well as a lens for gaining a more detailed understanding of how generalization and memorization work. We believe these conceptual themes are promising ones, and hope to see more work that employs them.

\section*{Acknowledgments}

We thank Ethan Perez, Jan Leike, and Martin Wattenberg for helpful feedback on the draft. We thank Daniela Amodei, Jamie Kerr, Jia Yuan Loke,  Rebecca Raible, and Tim Telleen-Lawton for support with the project.

\section*{Author Contributions}

\textbf{Danny Hernandez} led the project performed the majority of experiments, analysis, and writing.

\textbf{Tom Brown} led engineering efforts for the scaling team, including efficient pre-training and gave helpful feedback on the paper.

\textbf{Tom Conerly} made engineering contributions on the scaling team.

\textbf{Nova DasSarma} managed the underlying cluster infrastructure.

\textbf{Dawn Drain} helped with pre-training research and infrastructure.

\textbf{Sheer El-Showk} helped with pretraining research and dataset construction.

\textbf{Nelson Elhage} contributed significantly to interpretability tooling, provided support on that tooling, and gave helpful feedback.

\textbf{Zac Hatfield-Dodds} helped with codebase maintenance and with engineering

\textbf{Tom Henighan} helped with pretraining the underlying language models, with dataset creation, with managing the cluster during some phases of the project, and gave helpful feedback on the paper.

\textbf{Tristan Hume} contributed to interpretability tooling that was leveraged in this work.

\textbf{Scott Johnston} helped with pretraining research.

\textbf{Ben Mann} contributed to pretraining and cluster management.

\textbf{Chris Olah} lead the interpretability team, which provided tooling and support for this work.

\textbf{Catherine Olsson} contributed to interpretability tooling, provided support on that tooling, and provided interpretability research advice.

\textbf{Dario Amodei} contributed greatly to the framing and writing of the work and advised the project.

\textbf{Nicholas Joseph} helped design and build a framework for efficient training of large language models, gave helpful feedback on the paper, and advised the project.

\textbf{Jared Kaplan} led pre-training efforts initially and advised the project.

\textbf{Sam McCandlish} led pre-training efforts and advised the project.