text:
\section{Introduction}
\vspace{-.75em}
Recent work has shown that an adversary can extract training data from Pretrained Language Models (LMs) including Personally Identifiable Information (PII) such as names, phone numbers, and email addresses, and other information such as licensed code, private clinical notes, and 128-bit UUIDs~\citep{carlini2021extracting, lee-etal-2022-deduplicating, huang2022large, LehmanJPGW21}. In 2021, an AI chatbot \textit{Iruda} became the first AI system to be sued for violating the Personal Information Protection Act after generating the exact home addresses and bank account numbers of actual individuals unintentionally~\citep{park_2021}. ~\citet{heikkil_2022} has also shown that GPT-3~\citep{brown2020language}, one of the most well known LM currently in commercial use, offered detailed private information about the Editor-in-Chief of MIT Technology Review including his family members, work address, and phone number. Considering findings that show extracting training data gets easier as LMs scale to larger sizes~\citep{carlini2022quantifying} and that it is common practice for practitioners to release billion parameter pretrained LMs for public use~\citep{gao2020pile, black2021gpt, zhang2022opt}, it has become important to provide privacy guarantees for large LMs.

\begin{figure}[ht!]
    \centering
    \includegraphics[width=1\linewidth]{figures/Figure1.pdf}
    \caption{Comparison of previous approaches and \textit{knowledge unlearning} when an individual practices his/her Right-To-Be-Forgotten (RTBF).}
    \label{fig:overview}
\end{figure}

Practitioners are required to \textit{delete} personal information from the LMs by individuals' request because each individual has the ``Right To Be Forgotten (RTBF)"~\citep{mantelero2013eu, graves2021amnesiac} and can limit the direct and indirect commercial use of their personal information~\citep{villaronga2018humans}. Previous methods addressing privacy risks for language models attempt to remove all private information from the training data (data preprocessing)~\citep{aura2006scanning, dernoncourt2017identification, lison-etal-2021-anonymisation, kandpal2022deduplicating} or attempt to design algorithms that ensure differential privacy (DP)~\citep{dwork2008differential, dwork2006calibrating, abadi2016deep, anil2021large, li2022large, yu2022differentially}. Both approaches require \textit{retraining} the underlying LM every time individuals want to practice their RTBF, which makess them inadequate for large LMs that are extremely costly to retrain. Furthermore, as pointed out by \citet{brown2022does}, data preprocessing methods assume private information to be easily identifiable, specified, and removed and DP algorithms can only guarantee protection for information that has clear privacy borders, which make them inadequate in the real-world scenarios where the standard of privacy might differ by each individuals.

To this end, we propose \textit{knowledge unlearning} (Figure \ref{fig:overview}) as an efficient solution that can be applied with just a few parameter updates instead of pretraining the underlying LM again. We perform experiments on GPT-Neo LMs (125M, 1.3B, 2.7B)~\citep{black2021gpt} and show that simply changing the gradient descent to the opposite direction during language modeling (which can also be seen as \textit{maximizing} instead of \textit{minimizing} the loss function) is effective at protecting target sequences from extraction attacks with little to no performance degradation on the initial LM capabilities measured via 9 common NLP benchmarks: Hellaswag~\citep{zellers2019hellaswag}, Lambada~\citep{paperno-etal-2016-lambada}, Winogrande~\citep{sakaguchi2021winogrande}, COPA~\citep{gordon-etal-2012-semeval}, ARC-Easy~\citep{Clark2018ThinkYH}, ARC-Challenge~\citep{Clark2018ThinkYH}, Piqa~\citep{bisk2020piqa}, MathQA~\citep{amini-etal-2019-mathqa}, and PubmedQA~\citep{jin2019pubmedqa}. For some cases, \textit{knowledge unlearning} unexpectedly shows significant improvements in LM performance for some of the benchmarks.

We compare our approach with data deduplication method~\citep{kandpal2022deduplicating} which is known to mitigate privacy risks, and show the effectiveness of knowledge unlearning by providing a stronger privacy guarantee while being much more efficient. We also provide a general guideline that can be used to quantify the \textit{memorization} and \textit{extraction likelihood} of target token sequences and suggest when we can empirically consider them to have been ``forgotten''. Specifically, we introduce a novel metric that measures the extraction likelihood by varying the prefix length of the target token sequence and quantifying how much of the suffix is actually extracted from the LM.

Surprisingly, for \textit{knowledge unlearning}, we find that it is easier to forget a chunk of instances \textit{sequentially} rather than trying to forget them all at once. We provide further analysis and show that the difficulty of \textit{knowledge unlearning} depends heavily on the target data being forgotten, especially the domain of the target data. We also provide empirical examples of performing extraction attacks and how exactly \textit{knowledge unlearning} provides a privacy guarantee for the LM.

To summarize, our main contributions are fourfold:
\begin{itemize}
  \item We compare \textit{knowledge unlearning} with a data preprocessing approach and show that our approach results in little to no performance degradation of general language modeling performance (sometimes resulting in improvement) while providing stronger empirical privacy guarantees in situations individuals practice their RTBF and being 3,500,000x more computationally efficient than the compared approach.

  \item We perform additional experiments to determine which factors contribute to the difficulty of knowledge unlearning and find that (1) trying to forget many samples at once results in substantial LM performance degradation which can be mitigated by \textit{sequentially} forgetting chunks of data and that (2) the domain of the target data (Code, License, Wikipedia, etc.) plays a critical role in determining how hard they are to forget.

  \item We provide a novel metric and a general guideline for quantifying the privacy risks for LMs and determine when they should be considered to have ``forgotten" a given target sequence.

  \item \textit{Knowledge unlearning} surprisingly seems to make LMs stronger where the extreme cases bring \textit{+8.0\%} (37.6\% $\rightarrow$ 45.6\%), \textit{+10.1\%} (57.4\% $\rightarrow$ 67.5\%), and \textit{+7.9\%} (62.2\% $\rightarrow$ 70.1\%) improvements on Lambada for \textsc{GPT-Neo} 125M, 1.3B, and 2.7B, respectively.
\end{itemize}
\vspace{-.5em}
\section{Related Work}
\vspace{-.5em}
\label{sec:related_work}
\subsection{Privacy Methods for Language Models}
Prior work that tries to mitigate privacy risks for LMs can be divided mainly into data pre/post-processing methods and differential privacy methods.

\paragraph{(Data) Pre/Post-Processing} Data preprocessing aims to sanitize the training data; it aims to get rid of all data that might violate any kind of privacy from the training data prior to training. These methods mostly utilize measures such as parsers and classification models that try to identify and predict patterns that constitute private information. This is effective at identifying well-formatted private information such as social security numbers or special forms of medical notes~\citep{aura2006scanning, dernoncourt2017identification, lison-etal-2021-anonymisation, kandpal2022deduplicating}. However, as pointed out by \citet{brown2022does}, considering that private information is mostly context-dependent and sometimes in a non-specific format, data preprocessing methods cannot fully claim that they provide privacy guarantees, especially guarantees that match each individual's standards. Methods that attempt to utilize \textit{post-processing} methods such as applying censorship to the LM outputs still face the same limitations.

In this work, we compare our proposed method with a  data preprocessing approach proposed by \citet{kandpal2022deduplicating} which shows that deduplicating the training corpora before pretraining helps pretrain LMs that show stronger robustness against extraction attacks than an LM pretrained under the same circumstances without deduplicating the pretraining corpora. However, we highlight that this approach, which may still be effective at mitigating the overall privacy risks, is not the most suitable approach when considering a realistic scenario of individuals requesting the removal of their information from the implicit parameters of the LMs.

\paragraph{Differential Privacy}
Differential Privacy (DP) aims to guarantee that the effect of an individual input on the output of a specific function is bounded~\citep{dwork2008differential, dwork2006calibrating}. In the context of deep neural networks, DP, which needs to be applied during the training phase, aims to construct models that can ensure that the individual information within the training data cannot be inferred~\citep{abadi2016deep}. While DP has shown to be surprisingly effective at fine-tuning LMs~\citep{li2022large, yu2022differentially}, pretraining LMs with DP still suffers from substantial performance gap, expensive computation, and slow convergence~\citep{anil2021large}. Furthermore, as pointed out by \citet{brown2022does}, DP can only provide limited guarantees for LMs because DP requires a unified definition for privacy boundaries, which is inherently impossible for natural language data. Furthermore, in a realistic scenario where individuals may practice their Right-To-Be-Forgotten (RTBF) dynamically after model deployment, it is extremely difficult to define a notion of privacy that matches the requirements of each individual beforehand, which is required for training an LM with DP.

\subsection{Machine Unlearning}
Machine unlearning has received attention as an alternative approach to overcome data privacy issues in machine learning~\citep{cao2015towards, ginart2019making, bourtoule2021machine, graves2021amnesiac}. Several studies attempt to explore machine unlearning for deep neural networks~\citep{golatkar2020eternal, mehta2022deep}. However, they mostly focus on proposing algorithms for image classification models where they aim to forget a whole class; that is, achieve random performance for specific image classes such as ``cats'' or ``ships''.
We are the first, to the best of our knowledge, to explore unlearning a specific sequence of tokens for LMs which is a quite different set-up from traditional image classification models ($\sim$tens of image classes vs. a sequence of tokens that can each be classified into $V \in \mathbb{R}^{\sim 50,000}$). In this work, we coin this approach as \textit{knowledge unlearning} since we are more focused on forgetting specific \textit{knowledge} represented by sequences of tokens.

\citet{zhou2022fortuitous} focus on how \textit{forgetting} can be leveraged to improve the performance of the underlying model. They propose ``forget-and-relearn'' that unifies existing iterative training algorithms by selectively removing undesirable information and re-learning good features, helping boost performance for the task of image classification and multi-agent emergence communication. The underlying assumption is that it is often easier to define and stop unwanted behavior than to teach good behavior. We also show this phenomenon in Section \ref{sec:experiments} where we  unintentionally find unlearning just a few sequences of tokens sometimes boosts general LM capabilities.
\vspace{-0.5em}
\subsection{Memorization in Language Models}
Previous work that explores to which extent LMs have memorized their training data approach the phenomenon with two different viewpoints. Some work view memorization of LMs simply as a threat to individual privacy~\citep{carlini2021extracting, carlini2022quantifying, jagielski2022measuring}  and utilize metrics that quantify how much the LMs are susceptible to adversarial attacks. These metrics are mostly dependent on the specific types of attacks such as the membership inference attack~\citep{shokri2017membership} and measure the privacy risks of LMs by quantifying the success rate of these attacks.

Another line of work simply quantifies how much \textit{knowledge} is accumulated and forgotten during pretraining by extracting relational knowledge about the world~\citep{petroni2019language, lazaridou2021mind, jang2022towards, jang2022temporalwiki}. This line of work does not view memorization as a negative trait, but as a positive one that can be leveraged to extract world knowledge from its implicit parameters and perform knowledge-intensive tasks such as question answering or training knowledgeable conversation agents.

Our work is highly related to \citet{jagielski2022measuring}'s work where they also assert that forgetting can be a relaxed version of differential privacy. However, there are two main differences between our work and theirs. First, they only analyze forgetting as a \textit{passive} form of mitigating privacy, asserting that data seen early in large-scale training obtain privacy benefits, whereas we suggest a more \textit{active} form of forgetting. Second, they only show analysis results with image classification and audio generation models while we specifically focus on large LMs.
\vspace{-.75em}
\section{Knowledge Unlearning for Language Models}
\label{sec:method}
\vspace{-.75em}
\subsection{Methodology}
We propose simply \textit{negating} the original training objective of minimizing the negative log-likelihood of the token sequences as our main method of knowledge unlearning in LMs. Specifically, given a sequence of tokens $\boldsymbol{x}=(x_1,...,x_T)$, our unlearning training objective is simply minimizing the following loss function:

\begin{equation}\label{eq1}
\mathcal{L}_{UL}(f_{\theta}, \boldsymbol{x}) = - \sum_{t=1}^{T}\text{log}(1 - p_{\theta}(x_{t}|x_{<t}))
\end{equation}

where $x_{<t}$ denotes the token sequence $x=(x_1,...,x_{t-1})$ and $p_{\theta}(x_{t}|x_{<t})$ denotes the conditional probability of predicting the next token to be $x_t$ when given $x_{<t}$ to an LM $f$ with parameters $\theta$.

Prior work refer to this training objective as \textit{unlikelihood} training and combines it together with the original loss of minimizing the negative log-likelihood for the final objective of enhancing language modeling quality~\citep{Welleck2020Neural} and few-shot learning for downstream NLP tasks~\citep{tam-etal-2021-improving}. In contrast, we simply optimize the unlikelihood training objective since we are only concerned with forgetting. While this method seems simple, it is highly effective at forgetting specific token sequences without affecting the overall LM capabilities as shown in Section \ref{sec:experiments}.

\subsection{Quantifying Privacy Risks of Language Models}
\label{subsec:metrics}
In this subsection, we introduce two metrics we use to quantify the privacy risks given a specific token sequence and how we empirically define the token sequence to be forgotten.

\paragraph{Extraction Likelihood (EL)} We first introduce a new metric, EL. Given a sequence of tokens $\boldsymbol{x}=(x_1,...,x_T)$ and an LM $f$ with pre-trained parameters $\theta$, we define EL to be as follows:

\begin{equation}\label{eq2}
\textsc{EL}_n(\boldsymbol{x}) = \dfrac{\sum_{t=1}^{T-n} \textsc{Overlap}_n(f_{\theta}(x_{<t}),x_{\geq t})}{T-n}
\end{equation}
\begin{equation}\label{eq3}
 \textsc{Overlap}_n(\boldsymbol{a}, \boldsymbol{b}) = \dfrac{\sum_{c\in n\mbox{-}grams(\boldsymbol{a})} \mathbbm{1}\{c \in n\mbox{-}grams(\boldsymbol{b})\}}{| n\mbox{-}grams(\boldsymbol{a})|}
\end{equation}

where $ n\mbox{-}grams()$ denotes the list of $n$-grams in the given token sequence and $f_{\theta}(x_{<t})$ denotes the output token sequences from the LM $f_{\theta}$ when given $x_{<t}$ as input that can have max lengths $|x_{\geq t}|$ but may be shorter when the \textsc{EOS} (end-of-sequence) token is generated beforehand.

The process of varying the prefix length $|x_{<t}|$ can be seen as varying the \textit{strength} of adversarial attacks. This is based on the assumption that the more prior information is provided about the target token sequence, the easier the LM will be able to extract it. Overall, EL can be seen as estimating the general \textit{extraction likelihood} since we are measuring the average success rate of varying extraction attacks quantified via getting the n-gram overlap of generated and target token sequences. While previous metrics quantifying the privacy risks of LMs are dependent on specific adversarial attacks, this characteristic of EL allows it to quantify the general likelihood of extraction without any dependency on specific extraction attacks.

We regard $n$ to be a hyper-parameter that can be varied depending on the stringency of privacy standards. The higher $n$ is set, the stricter we set the standard for a successful extraction attack.

\paragraph{Memorization Accuracy (MA)} We define Memorization Accuracy (MA) as follows:

\begin{equation}
\textsc{MA}(\boldsymbol{x}) = \dfrac{\sum_{t=1}^{T-1} \mathbbm{1}\{\text{
argmax}(p_{\theta}(\cdot|x_{<t})) = x_t\}}{T-1}
\label{equation:ft}
\end{equation}

MA quantifies how much $f_{\theta}$ has memorized the given token sequences and was proposed by \citet{tirumala2022memorization} to analyze the training dynamics of large LMs.

\paragraph{Empirical Definition of Forgetting} By utilizing both $\textsc{EL}_{n}$ and MA, we empirically define a specific token sequence $\boldsymbol{x}$ to be forgotten and is no longer susceptible to extraction attacks when the following conditions are met:

\begin{equation}\label{eq:ft}
\textsc{EL}_n(\boldsymbol{x}) \le \dfrac{1}{|D^{\prime}|}\sum_{\boldsymbol{x}^{\prime}\in D^{\prime}}\textsc{EL}_n(\boldsymbol{x}^{\prime}) \text{ and } \textsc{MA}(\boldsymbol{x}) \le \dfrac{1}{|D^{\prime}|}\sum_{\boldsymbol{x}^{\prime}\in D^{\prime}}\textsc{MA}(\boldsymbol{x}^{\prime})
\end{equation}

where $D^{\prime}$ represents a validation corpora not seen during training. In other words, we define $\boldsymbol{x}$ to be forgotten when the $\textsc{EL}_{n}$($\boldsymbol{x}$) and MA($\boldsymbol{x}$) reach a value that is lower than the average $\textsc{EL}_{n}$ and MA on token sequences that were not seen during training.
\vspace{-.75em}
\section{Experiments}
\label{sec:experiments}
\vspace{-.75em}
\subsection{Models, Datasets, and Configurations}
\paragraph{Baseline Models}
For the experiments, we use the \textsc{GPT-Neo} (125M, 1.3B, 2.7B) LMs~\citep{black2021gpt} initially pretrained on all of the Pile corpora (825GB)~\citep{gao2020pile}, and the \textsc{OPT} (125M, 1.3B, 2.7B) LMs~\citep{zhang2022opt}, pretrained on a subset of the \textit{deduplicated} version of the Pile as well as other corpora from different domains. For the experiments, we perform unlearning the \textsc{GPT-Neo} LMs and quantify the privacy risks of the target data compared to the \textsc{OPT} LMs to measure how effective our proposed approach is in contrast to deduplicating the training corpora before pretraining the underlying LM~\citet{kandpal2022deduplicating}. We do not use the exact LMs from \citet{kandpal2022deduplicating} because the LMs were not open-sourced, and thus use the OPT LMs instead.

\vspace{-0.5em}
\paragraph{Target Data} For the actual target data used to quantify the privacy risks of the LMs, we sample instances from the Training Data Extraction Challenge~\footnote{https://github.com/google-research/lm-extraction-benchmark} where 15,000 examples (each are 200 token sequences long) from 16 different domains of the Pile corpora that are identified to be somewhat easy-to-extract are provided. For our experiments, we randomly sample \textit{s} samples from the 15,000 examples and make the underlying LM forget the \textit{s} samples at once. As a default, we show the average results of 5 random samplings of \textit{s} samples for all of our experimental settings. We only provide the average of the 5 samplings and do not separately report the standard deviation. Instead, we provide the results of each individual run in Appendix \ref{appen:full_results}.
\vspace{-0.5em}
\paragraph{Evaluation Datasets} Guaranteeing stronger privacy for LMs may become meaningless if it requires sacrificing their original capabilities.
Thus, while quantifying the privacy risks of LMs, we also quantify the original LM capabilities by evaluating the LMs on 9 different tasks: Hellaswag~\citep{zellers2019hellaswag} and Lambada~\citep{paperno-etal-2016-lambada} benchmarks to measure linguistic reasoning abilities, Winogrande~\citep{sakaguchi2021winogrande} and COPA~\citep{gordon-etal-2012-semeval} to measure commonsense reasoning abilities, and ARC-Easy~\citep{Clark2018ThinkYH}, ARC-Challenge~\citep{Clark2018ThinkYH}, Piqa~\citep{bisk2020piqa}, MathQA~\citep{amini-etal-2019-mathqa}, PubmedQA~\citep{jin2019pubmedqa} benchmarks to measure the scientific reasoning abilities. We use the test set for Lambada and the validation set for the rest of the benchmarks. We also show the results of measuring the perplexity on the validation corpora of Pile and Wikitext in Appendix \ref{appen:perplexity}. We do not include measuring perplexity as one of the main evaluations because perplexity might not be the most suitable metric for quantifying general LM performance, especially in the case of unlearning~\footnote{We give further explanation of why we think perplexity is not a good proxy for quantifying the general LM capability in our scenario in Appendix \ref{appen:perplexity}.}.

\vspace{-0.5em}
\paragraph{Configurations}
For the learning rate, we set it to 5e-5.
We show the effect of varying learning rates in Appendix \ref{appen:varying_lr}. We use a constant learning rate scheduling throughout the run. We fix the global batch size to be the same as \textit{s} (how many samples are forgotten at once) because having global batch sizes smaller than $s$ proved to degrade general LM capabilities~\footnote{In Section \ref{subsec:analysis}, We show that $s$ plays a critical role in determining how much the unlearning will degrade in general capabilities of the LM since $s=128$ shows to result in much degradation. Method to mitigate this is proposed in Section \ref{subsec:analysis} as well.}. For $\textsc{EL}_n$, we set \textit{n}=10 which means EL measures the extraction likelihood of extracting \textit{n} consecutive tokens of varying extraction attack. For calculating $\textsc{EL}_{10}$ and MA, we use a naïve greedy decoding strategy. We set both the dropout and weight decay rates to 0. Lastly, while we provide a guideline of empirically deciding a single token sequence to be forgotten in Section \ref{subsec:metrics}, for considering a \textit{chunk} of $s$ token sequences to be forgotten, we use the average $\textsc{EL}_{10}$ and MA as an approximation of the individual $\textsc{EL}_{10}$ and MA.

\subsection{Main Experiments}
\begin{table*}[t!]
\centering
\fontsize{8}{10}\selectfont
\caption{\small Forgetting Threshold for \textsc{GPT-Neo} LMs}
\begin{tabular*}{0.4\columnwidth}{l|cc}
    \toprule
    \multirowcell{2}{\textbf{Model (Size)}} & $\textbf{\textsc{EL}}_{10}$ (\%)  & \textbf{MA(\%)} \\
    & \textbf{Threshold} & \textbf{Threshold} \\
    \midrule
    \textsc{GPT-Neo} (125M) & 29.94 & 4.99\\
    \textsc{GPT-Neo} (1.3B) & 33.27 & 5.68\\
    \textsc{GPT-Neo} (2.7B) & 34.02 & 5.53\\
    \bottomrule
\end{tabular*}
\label{table:threshold}
\end{table*}
\vspace{-0.5em}
\paragraph{Forgetting Threshold}
First, we show how we get the Forgetting Threshold for $\textsc{EL}_{10}$ and MA, the values where we consider the token sequence to be forgotten and unsusceptible from extraction attacks, for all model sizes of \textsc{GPT-Neo} LMs in Table \ref{table:threshold}. For $D^\prime$, we perform weighted sampling (same domain distribution as the Pile training corpora) of 10,000 instances each with token lengths 200 from the Pile validation corpora and measure the average $\textsc{EL}_{10}$ and MA (Equation \ref{eq:ft}), which are empirically set as the Forgetting Threshold values.

\begin{table*}[t!]
\centering
\small\addtolength{\tabcolsep}{-4pt}
\fontsize{7.5}{9}\selectfont
\caption{\small Main Results showing the average of 5 random sampling of $s=32$ (forgetting 32 samples at once). \textsc{Neo} denotes the initial \textsc{GPT-Neo} LM, \textsc{Neo} + \textsc{UL} represents performing unlearning on the initial \textsc{Neo} until it provides a stronger security guarantee for the target sequences than \textsc{OPT}, \textsc{Neo} + $\textsc{UL}^{+}$ represents performing unlearning on \textsc{GPT-Neo} until target sequences are \textit{forgotten}, and \textsc{OPT} represents the LM with deduplication applied. Avg. denotes the average accuracy of the 9 evaluation benchmark datasets. Best comparable performances are \textbf{bolded} and second best \underline{underlined}.}
\begin{tabular*}{1\columnwidth}{lc|cc|ccccccccc|c|c}
\toprule
\multicolumn{1}{c}{\multirow{2}{*}{\textbf{Model}}} & \textbf{\#} & \textbf{EL$_{10}$} & \textbf{MA} & \textbf{Hella.}  & \textbf{Lamba.}  & \textbf{Wino.} & \textbf{COPA} & \textbf{ARC-E} & \textbf{ARC-C} & \textbf{Piqa} & \textbf{MathQ} & \textbf{PubQ} & \textbf{Avg.} & \multirow{2}{*}{\textbf{Epoch}} \\
& \textbf{Params} & (\%) $\downarrow$  & (\%) $\downarrow$ & (ACC) & (ACC) & (ACC) & (ACC) & (ACC) & (ACC) & (ACC) & (ACC) & (ACC)  & (ACC) & \\ \midrule
\textsc{Neo} & 125M & 30.9 & 77.4 & \textbf{28.2} & 37.6 & 51.8 & \underline{62.0} & \textbf{45.6} & \textbf{22.0} & \textbf{63.3} & \textbf{22.5} & \textbf{57.6} & \textbf{43.4} & -\\
\textsc{Neo + UL} & 125M & \underline{3.7} & \underline{50.1} & \underline{27.8} & \textbf{41.6} & \underline{52.2} & 58.6 & 41.6 & \underline{21.2} & \underline{61.0} & 21.9 & \underline{57.5} & \underline{42.6} & 11.0\\
\textsc{Neo + UL$^+$} & 125M & \textbf{1.0} & \textbf{27.4} & 27.3 & 30.4 & 52.0 & 54.4 & 37.5 & 20.7 & 59.4 & \underline{22.0} & 55.4 & 39.9 & 17.2\\
\textsc{OPT} & 125M & 8.6 & 52.9 & 27.3 & \underline{38.9} & \textbf{53.2} & \textbf{66.0} & \underline{45.3} & 20.7 & 60.7 & 21.8 & 47.4 & 42.4 & -\\
\midrule
\textsc{Neo} & 1.3B & 67.6 & 92.2 & \textbf{37.0} & 57.4 & 54.8 & 70.0 & \textbf{56.6} & \underline{25.8} & \textbf{70.4} & \underline{21.9} & \underline{53.8} & \underline{49.8} & -\\
\textsc{Neo + UL} & 1.3B & \underline{11.0} & \underline{62.2} & \underline{36.7} & \underline{59.6} & 55.0 & \underline{70.4} & \underline{55.2} & \textbf{26.2} & \underline{70.1} & \underline{21.9} & 52.2 & 49.7 & 8.0\\
\textsc{Neo + UL$^+$} & 1.3B & \textbf{1.9} & \textbf{30.4} & 36.4 & \textbf{62.1} & \underline{55.5} & 69.4 & 53.7 & \textbf{26.2} & 69.4 & \underline{21.9} & 52.8 & 49.7 & 13.8\\
\textsc{OPT} & 1.3B & 23.3 & 67.1 & 34.4 & 58.8 & \textbf{57.1} & \textbf{76.0} & 55.1 & 24.3 & 69.0 & \textbf{23.2} & \textbf{57.8} & \textbf{50.6} & -\\
\midrule
\textsc{Neo} & 2.7B & 70.4 & 93.4 & \underline{40.8} & 62.2 & 56.4 & \underline{75.0} & \textbf{59.6} & 25.4 & \textbf{73.0} & 21.4 & 57.0 & \underline{52.3} & -\\
\textsc{Neo + UL} & 2.7B & \underline{13.0} & 66.0 & 40.7 & 63.5 & 56.4 & 72.0 & \underline{59.5} & 26.7 & \underline{72.8} & 21.7 & 57.1 & \underline{52.3} & 5.4\\
\textsc{Neo + UL$^+$} & 2.7B & \textbf{1.6} & \textbf{31.0} & \textbf{40.9} & \textbf{64.5} & \underline{56.6} & 69.2 & 57.5 & \textbf{27.6} & 72.0 & \underline{21.9} & \underline{57.2} & 51.9 & 10.8\\
\textsc{OPT} & 2.7B & 25.6 & 69.2 & 37.0 & \underline{64.4} & \textbf{59.2} & \textbf{78.0} & 56.7 & \underline{27.0} & 70.7 & \textbf{22.9} & \textbf{58.2} & \textbf{52.7} & -\\
\bottomrule
\end{tabular*}\label{table:main}
\end{table*}

\paragraph{Main Results} Table \ref{table:main} shows the main results of performing unlearning on LMs of varying sizes. In the table, \textsc{Neo} denotes the \textsc{GPT-Neo} LMs. \textsc{Neo + UL} denotes applying unlearning on the \textsc{GPT-Neo} LM until it reaches lower $\textsc{EL}_{10}$ and MA than \textsc{OPT}. Lastly, $\textsc{Neo + UL}^{+}$ denotes performing unlearning on the \textsc{GPT-Neo} until the target token sequences are \textit{forgotten} ($\textsc{EL}_{10}$ and MA value reach below the Forgetting Threshold). While we provide the average performances of the 5 random samplings in Table \ref{table:main}, we provide each individual runs in Appendix \ref{appen:full_results} for reference.

We highlight four main points regarding the results. (1) \textsc{OPT} LMs show a much lower $\textsc{EL}_{10}$ and MA than \textsc{GPT-Neo} LMs, confirming that deduplicating the pretraining corpora is indeed helpful for mitigating privacy risks. (2) $\textsc{Neo + UL}^{+}$ results in degradation of average LM performance for the 125M LM while retaining most of its previous capabilities for the 1.3B and 2.7B LMs. Interestingly, for some benchmarks such as Lambada and ARC-Challenge, it actually results in a boost of performance. (3) While the LMs scale to larger sizes, it takes fewer epochs for the target sequences to be forgotten. Together with (2), this implies that larger LMs are strong \textit{un}learners. (4) While $\textsc{Neo + UL}^{+}$ provides a stronger privacy guarantee than \textsc{OPT} without sacrificing average LM performance from \textsc{Neo}, it is much more computationally efficient (3,500,000x) than re-training the underlying LM, which is required for all data preprocessing approaches~\footnote{Computational efficiency is measured via FLOPs which is calculated by (6 x Total Training Tokens x Parameter Size) as in \citet{brown2020language}. FLOPs for \textsc{OPT} LMs were estimated using information from \citet{zhang2022opt}. We provide the FLOPs for the methods in Appendix \ref{appen:flops}.}. This characteristic makes knowledge unlearning very advantageous in scenarios where people might practice their RTBF and thus \textit{dynamic} forgetting of specific token sequences are required.

Overall, results show unlearning to be an effective approach to providing a stronger empirical privacy guarantee than existing LMs measured via $\textsc{EL}_{10}$ and $\textsc{MA}$ while retaining and sometimes even improving general LM capabilities.

\paragraph{Sequential Unlearning is more Stable than Batch Unlearning.}
\begin{figure}[t]
\centering
\begin{subfigure}[b]{0.45\textwidth}
\includegraphics[width=\textwidth]{figures/fig3a.pdf}
\caption{Batch Unlearning}
\end{subfigure}\hspace{2em}
\begin{subfigure}[b]{0.45\textwidth}
\includegraphics[width=\textwidth]{figures/fig3b.pdf}
\caption{Sequential Unlearning}
\end{subfigure}
\caption{\small Average LM performance on the 9 benchmarks when varying the total number of samples forgotten at once is shown in (a) and the average LM performances when the 128 samples are divided into 4 chunks and are forgotten sequentially is shown in (b). The lines denote the average performances of 5 random samplings and the standard deviation is shown as the shaded regions. The dotted lines in (b) denotes the $s=128$ performance in (a) for comparison purposes.}
\label{fig:epochs}
\end{figure}

We show the effect of varying $s$ (the \# of data instances to be forgotten at once) in Figure \ref{fig:epochs}a across model scales. We denote this approach as \textit{batch} unlearning. As shown by the $s=128$ results, it is harder to forget more samples at once, resulting in substantial degradation of average LM performance regardless of how large the LM is. Since $s\leq32$ does not show much degradation, we explore if \textit{sequentially} unlearning can be a solution. In Figure \ref{fig:epochs}b, we show the result of dividing the 128 samples into 4 chunks of 32 and performing sequential unlearning; we unlearn each chunk at a time until the chunk reaches the forgetting threshold. Surprisingly, as shown by the performance gap at $s=128$ between the dotted lines (the $s=128$ performance of Figure \ref{fig:epochs}a) and straight lines, the end result is vastly different even though exactly the same instances were forgotten. Sequential unlearning shows almost no degradation of average LM performance. In Appendix \ref{appen:seq_unlearn}, we show that chunks once forgotten stay forgotten and that later chunks are forgotten much faster compared to the initial chunk. This result hints at the \textit{generalization} of unlearning, which we do not further explore in the scope of this work. The result also suggests that knowledge unlearning can be \textit{continually} applied to LMs when needed.

\subsection{Analysis of Knowledge Unlearning}
\label{subsec:analysis}
\vspace{-0.5em}
\paragraph{Providing Better Intuition of What Exactly Happens During Knowledge Unlearning.}
\begin{figure}[t]
\centering
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{figures/hellaswag.pdf}
\caption{Hellaswag}
\end{subfigure}
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{figures/lambada.pdf}
\caption{Lambada}
\end{subfigure}
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{figures/wino.pdf}
\caption{Winogrande}
\end{subfigure}
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{figures/copa.pdf}
\caption{COPA}
\end{subfigure}
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{figures/arc-e.pdf}
\caption{ARC-Easy}
\end{subfigure}
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{figures/arc-c.pdf}
\caption{ARC-Challenge}
\end{subfigure}
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{figures/piqa.pdf}
\caption{Piqa}
\end{subfigure}
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{figures/mathqa.pdf}
\caption{MathQA}
\end{subfigure}
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{figures/pubmedqa.pdf}
\caption{PubMedQA}
\end{subfigure}
\caption{\small Performance on the LM benchmarks as we perform 10 different unlearning runs on \textsc{GPT-Neo} 1.3B where $s=1$.
}
\label{fig:performance_change}
\end{figure}

To show exactly what happens to the LM during knowledge unlearning, we show how the performance of each of the LM benchmarks changes as we perform 10 runs of unlearning to the \textsc{GPT-Neo} (1.3B) model (each run with $s=1$) in Figure \ref{fig:performance_change}. As shown in the figure, the LM performance for each benchmark varies tremendously on which sample is chosen to be forgotten. Furthermore, the ending time of each run is different, indicating that some samples are forgotten faster than others.

To provide a better intuition of exactly how knowledge unlearning guarantees privacy, we perform an extraction attack with a token sequence sample in Table \ref{table:attack} where we show the model-generated text from the extraction attack \textit{before} and \textit{after} applying knowledge unlearning. While the extraction attack is extremely successful at extracting the rest of the suffix before unlearning (100\% of the token sequence), only a small portion ($\sim$3\% of the token sequence) of the suffix is extracted after applying unlearning.

\begin{table*}[t!]
\centering
\fontsize{5.75}{7}\selectfont
\caption{\small An example extracting the suffix of a token sequence from \textsc{books3} domain from \textsc{GPT-Neo} 1.3B showing the effect of knowledge unlearning. Model generated text given a prefix of length 100 are shown in \textcolor{blue}{Blue}.}
\begin{tabular*}{1\columnwidth}{c|cc}
    \toprule
    \textbf{Domain} & \textbf{Status} & \textbf{Text}\\
    \midrule
    \multirow{15}{*}{\textsc{books3}} & \multirow{4}{*}{\textbf{Original}} & \multirow{5}{*}{\begin{tabular}[|l|]{@{}l@{}}About the Publisher Australia HarperCollins Publishers (Australia) Pty. Ltd. 25 Ryde Road (PO Box 321) Pymble, NSW 2073, Australia \\http://www.harpercollinsebooks.com.au Canada HarperCollins Publishers Ltd. 55 Avenue Road, Suite 2900 Toronto, ON, M5R, 3L2, Canada\\ http://www.harpercollinsebooks.ca New Zealand HarperCollins Publishers (New Zealand) Limited P.O. Box 1 Auckland, New Zealand \\http://www.harpercollinsebooks.co.nz United Kingdom HarperCollins Publishers Ltd. 77-85 Fulham Palace Road London, W6 8JB, UK \\http://www.harpercollinsebooks.co.uk\end{tabular}}
    \\ \\ \\ & \textbf{Text} \\ \\ \cmidrule(lr){2-3}

    & \multirow{4}{*}{\textbf{Before}} & \multirow{5}{*}{\begin{tabular}[|l|]{@{}l@{}}About the Publisher Australia HarperCollins Publishers (Australia) Pty. Ltd. 25 Ryde Road (PO Box 321) Pymble, NSW 2073, Australia \\http://www.harpercollinsebooks.com.au Canada HarperCollins Publishers Ltd. 55 Avenue Road, Suite 2900 Toronto, ON, M5R, 3L2, Canada\\ http://\textcolor{blue}{www.harpercollinsebooks.ca New Zealand HarperCollins Publishers (New Zealand) Limited P.O. Box 1 Auckland, New Zealand} \\\textcolor{blue}{http://www.harpercollinsebooks.co.nz United Kingdom HarperCollins Publishers Ltd. 77-85 Fulham Palace Road London, W6 8JB, UK} \\\textcolor{blue}{http://www.harpercollinsebooks.co.uk}\end{tabular}}
    \\ \\ \\ & \textbf{Unlearning}\\ \\ \cmidrule(lr){2-3}

    & \multirow{4}{*}{\textbf{After}} & \multirow{5}{*}{\begin{tabular}[|l|]{@{}l@{}} About the Publisher Australia HarperCollins Publishers (Australia) Pty. Ltd. 25 Ryde Road (PO Box 321) Pymble, NSW 2073, Australia \\http://www.harpercollinsebooks.com.au Canada HarperCollins Publishers Ltd. 55 Avenue Road, Suite 2900 Toronto, ON, M5R, 3L2, Canada\\ http://\textcolor{blue}{www.harpercollins.com.au/Publishers/ Publisher: level three Level two is levels one and two together. The new face of a already great} \\\textcolor{blue}{title! Level one: Just right. Level two: Great. Level three: Awesome. The BloomsburyPublishersPublishers.com.au/PublishersPublishers} \\\textcolor{blue}{Levels are for bibliographic information or advanced level. s}\end{tabular}}
    \\ \\ \\ & \textbf{Unlearning}\\ \\
    \bottomrule
\end{tabular*}
\label{table:attack}
\end{table*}
\vspace{-1em}
\paragraph{Why Are Some Instances Harder to Forget?}
\begin{table*}[!t]
\centering
\small\addtolength{\tabcolsep}{-3.5pt}
\fontsize{8}{10}\selectfont
\caption{\small Unlearning \textsc{GPT-Neo 1.3B} on token sequences sampled from 8 different domains. We fix the epoch to 10, set $s=8$ and show the result of the average of 5 random samplings. \textit{Italicized} () denotes the $\Delta$ from \textsc{Initial}.}
\begin{tabular*}{1\columnwidth}{l|cc|ccccccccc|c}
\toprule
\multirowcell{2}{\textbf{Domains}} & \textbf{Initial}  & \textbf{Final} & \textbf{Hella.} & \textbf{Lamba.} & \textbf{Wino.} & \textbf{COPA} & \textbf{ARC-E} & \textbf{ARC-C} & \textbf{Piqa} & \textbf{MathQ} & \textbf{PubQ} & \textbf{Avg.} \\
& $\textbf{\textsc{EL}}_{10}$ & $\textbf{\textsc{EL}}_{10}$ & \textsc{(acc)} &  \textsc{(acc)}&  \textsc{(acc)} & \textsc{(acc)} & \textsc{(acc)} & \textsc{(acc)} & \textsc{(acc)} & \textsc{(acc)} & \textsc{(acc)} & \textsc{(acc)}\\
\midrule
\textsc{Initial} & - & - & 37.0 & \textbf{57.4} & \textbf{54.9} & 70.0 & \textbf{56.6} & 25.8 & \textbf{70.4} & \textbf{21.9} & \textbf{53.8} & \textbf{49.8} (\textit{0.0}) \\
\midrule
\textsc{freelaw} & 60.4 & 12.1 & \underline{37.2} & 52.2 & 53.9 & 68.4 & 55.5 & 26.2 & \underline{70.1} & 21.7 & \underline{53.5} & 48.7 (\textit{-1.1}) \\
\textsc{git. (code)} & 63.9 & 0.6 & \textbf{37.3} & \underline{53.4} & 54.4 & 69.2 & 56.3 & 26.0 & 69.9 & 21.5 & 49.8 & 48.7 (\textit{-1.1})\\
\textsc{git. (license)} & 75.8 & 0.0 & 37.1 & 52.0 & 54.2 & 69.0 & \underline{56.4} & \underline{26.4} & \underline{70.1} & \underline{21.8} & 51.8 & 48.8 (\textit{-1.0})\\
\textsc{enron emails} & 77.3 & 0.0 & 36.9 & 57.2 & \underline{54.8} & 68.4 & 55.8 & 26.3 & 69.8 & \underline{21.8} & 53.1 & \underline{49.4} (\textit{-0.4})\\
\textsc{books3} & 70.2 & 0.0 & 36.4 & 49.5 & 54.2 & \textbf{70.8} & 55.6 & 25.5 & 69.9 & 21.7 & 47.4 & 47.9 (\textit{-1.9})\\
\textsc{pile CC} & 67.8 & 0.0 & 35.7 & 45.9 & 53.8 & \underline{70.4} & 54.2 & \textbf{26.9} & 69.7 & \underline{21.8} & 52.0 & 47.8 (\textit{-2.0})\\
\textsc{uspto back.} & 59.4 & 0.0 & 33.7 & 44.7 & 53.5 & 67.0 & 45.9 & 24.0 & 67.0 & 21.5 & 50.3 & 45.3 (\textit{-4.5})\\
\textsc{pubmed cent.} & 71.8 & 0.0 & 36.5 & 44.5 & 54.1 & 69.6 & 55.6 & 24.8 & 70.0 & \textbf{21.9} & 46.4 & 47.0 (\textit{-2.8})\\
\bottomrule
\end{tabular*}
\label{table:domain}
\end{table*}

To measure why some instances are harder to forget, we perform 5 random samplings of $s=8$ from 8 different domains from the Training Data Extraction Challenge\footnote{\href{https://github.com/google-research/lm-extraction-benchmark}{https://github.com/google-research/lm-extraction-benchmark}} and perform unlearning on the \textsc{GPT-Neo 1.3B} LM. We also show the results of each individual run in Appendix \ref{appen:full_results}. As shown in Table \ref{table:domain}, despite undergoing the same number of token updates (10 epochs of unlearning), different domains result in vastly different outcomes; \textsc{enron emails} results in the average LM performance degradation of only -0.4\% while \textsc{uspto backgrounds} results in -4.5\% degradation. Furthermore, the final $\textsc{EL}_{10}$ varies depending on the domain, suggesting that some domains (e.g., \textsc{Freelaw}) are harder to forget than others. Lastly, domains that are more \textit{structured}, which means the data consists of some kind of patterns such as a list of emails (\textsc{enron emails}) or code (\textsc{github (code)}), seem to result in less degradation of LM performance in contrast to domains that are more \textit{unstructured}, which means the data consist of mostly raw English text such as a review for journal submission (\textsc{pubmed central}). We provide examples from each domain in Appendix \ref{appen:domain_samples}.
\vspace{-.75em}
\section{Closing}
\vspace{-0.5em}
In this paper, we propose \textit{knowledge unlearning} as a method for mitigating privacy risks in LMs that is not only orders of magnitude more efficient than previous methods, but also provides a stronger empirical privacy guarantee with little to no degradation of general LM capabilities measured by evaluating on 9 common LM benchmarks. As large LMs expand their use cases, potentially affecting the daily lives of people, the research community should make sure that the privacy of individuals is not violated intentionally or unintentionally by the knowledge stored in the implicit parameters of large LMs. Since it is inherently impossible to prevent and predict all future privacy concerns prior to pretraining the LM, we suggest the community consider knowledge unlearning for ensuring privacy upon individuals' requests post hoc pretraining~\footnote{We provide some limitations of our work in Appendix \ref{appen:limits}.}.
\label{sec:conclusion}

\subsubsection*{Acknowledgments}
We thank Hanseok Oh, Minsu Kim, James Thorne, and Hyunji Lee for useful discussion and feedback while preparing the paper draft. This work was supported by Institute of Information \& communications Technology Planning \& Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST)).

redpajama_set_name: RedPajamaArXiv


text:
Insecticide found in suspected pet poisoning
The Clarion-Ledger
NEWTON, Miss. - Insecticide killed at least one dog suspected of being poisoned in Newton County.
According to The Meridian Star, a toxicology report identifies two organophosphate insecticides and permethrin as the culprits in the dog's death.
The report comes after several Oakland community residents were shocked over their pets' sudden deaths in November.
Sherri Munn, whose pet was the dog tested in the report, says at least five dogs died and at least two were taken to veterinarians.
Newton County Sheriff Jackie Knight says proving the dogs were intentionally poisoned will be difficult.
According to the Centers for Disease Control, organophosphates are the most widely used insecticides. The National Pesticide Information Center says products containing the synthetic chemical permethrin may be liquids, powders, dusts, aerosol solutions, sprays, and treated clothing.

redpajama_set_name: RedPajamaCommonCrawl


B.S. Degree in Forestry from Michigan Technological University, 1990.
USDA Technical Service Provider (TSP), Certified Tree Farm Inspector, Sustainable Forestry Education Program 1999.
Timber Management Plans, Timber Marking, Timber Marketing, Sales Administration, Timber Stand Improvement, Tree Planting, Plantation Maintenance, Timber Inventory, Timber Taxation Assistance, Timber Appraisals, Expert Witness, and Invasive Species Management, inlcuding Brush Mowing.
Association of Consulting Foresters, Michigan Forest Association, Society of American Foresters (Certified Forester), and Indiana Forestry and Woodland Owners Association.
Commercial General Liability, Workmen's Compensation.


text:
Prince Street
Prince Street is the new monthly variety show by, for and about people who love food. With special guests, profiles, interviews, field reports, cooking and comedy, our team of award-winning journalists, actors and writers explore the ever-changing world of what we eat, what we cook, and the surprising ways food is life.
Named after Dean & DeLuca's current and original SoHo locations, both along Manhattan's Prince Street, our podcast goes beyond the landmark store, and its nearly 40-year history in New York City, to share compelling stories with food at their heart. We'll be touching upon contemporary culture—food, film, art, fashion, business, music, design and more—to find common threads between them and what we eat.
Each of the 12 episodes in our first season will feature stories organized around a theme we can all relate to, in cooking and in life.
Episode one is all about Preparation. Episode 2: Risk. Episode 3: Craving. Episode 4: Rule Breaking. Episode 5: Anxiety. Episode 6: Fullness. Future episodes will focus on Triumph, Failure, Secrets, Table Setting, Community and Clean Up.
Editor-in-chief Howie Kahn, a James Beard Award-winning journalist oversees the 40-minute show along with writer and producer Elisabeth Robinson. Prince Street's roster of renowned correspondents include novelist and wine columnist Jay McInerney, actor and director Griffin Dunne, TV personality Eden Grinshpan, Mile End Deli's Noah Bernamoff, Fathom co-founder and CEO Pavia Rosati, and Sierra Tishgart, a multi-platform writer and reporter, spanning print, web and television.
Upcoming guests include actors Scarlett Johansson and John Malkovich, filmmaker and wine maker Francis Ford Coppola, author Jonathan Franzen and chefs Eric Ripert, Ivan Orkin and Ruth Rogers.


Sidney High School's Rachel Blauert shot an 86 on Monday afternoon at the District B-4 tournament in Gering.
The Sidney High girls' golf team is headed back to the state tournament after finishing third on Monday in the Class B District 4 at Monument Shadows Golf Club in Gering.
The Lady Raiders shot a team score of 384. That left them behind team champion and defending Class B state champion Scottsbluff (349) and McCook (381).
Sidney will compete for the Class B state title on Oct. 12-13 at Quail Run Golf Course in Columbus.
Rachel Blauert recorded Sidney's top score, carding a round of 86 to finish fourth overall.


at 223 facilities throughout our catchment area.
Welcome to Queen's School of Rehabilitation Therapy's clinical education site. Clinical education is an exciting and integral component of both the occupational and physical therapy curriculum.
Queen's School of Rehabilitation Therapy currently depends upon over 400 clinical educators (occupational therapists and physical therapists) at 223 facilities throughout our catchment area.
The objective of clinical education is to provide students with high quality, innovative and dynamic clinical learning opportunities.
A searchable directory of current courses including course objectives, evaluation and learning outcomes.
The Queen's catchment area extends west to east from Oshawa to Cornwall and up to Orillia/Peterborough/Lindsay and Smiths Falls in the north. We welcome all clinical educators to the Queen's community.