text:
\section{Introduction}\label{sec:introduction}

The use of recommendation systems has grown significantly in recent years. Shopping websites present users with item recommendations based on their history and demographics; movie and book recommendation websites are being used every day to pick new favorites; and online music streaming services generate dynamic playlists to suit each user's preferences.

Collaborative filtering (CF) is a popular and successful approach to providing user recommendations using knowledge about the user's preferences and the preferences of other users with similar interests to predict which items the user is most likely to be interested in \cite{Adomavicius2005,Su2009}. There is a large body of literature on CF algorithms \cite{Adomavicius2005,Breese1998,Das2007,Castro2007,Castro2007a,George2005,Herlocker2004,1167344,4633869,Resnick1994,Su2009,Zhu2009}, which can be categorized into memory-based and model-based techniques \cite{Adomavicius2005,Su2009}.

Since the key point in CF systems relies on users' preferences and past actions to make predictions, many users may feel uneasy because of privacy concerns \cite{6686270,Calandrino2011231}. Additionally, the need may arise for two or more CF systems to leverage their combined data to provide their users with more accurate recommendations. This type of computation that relies on data from more than one party is referred to as multi-party computation \cite{goldreich1998secure}. To leverage this type of computation, the parties need to be able to do it securely, without allowing the other parties to read and store the user data. This is when privacy-preserving data mining techniques \cite{Agrawal2000} become necessary, leading to many different algorithms for privacy-preserving CF (PPCF); e.g: \cite{Ahmad2007,1004361,5066586,4403128,6570605,6133147,7033684,6785787}.

This survey examines a large number of contributions to PPCF recommendation systems in published literature. It divides the papers into broad categories multiple times based on different factors, with the goal of providing a comprehensive overview of the recent literature in the field. This is a broad perspective that the authors believe to be missing from the recent PPCF literature. The remainder of this survey is organized as follows: Section~\ref{sec:background} provides some background information on privacy-preserving data mining (Section~\ref{ssec:privacy-preserving}) and CF recommendation systems (Section~\ref{ssec:collaborative-filtering}). Section~\ref{sec:survey} contains a survey of the recent PPCF literature, organized into subsections corresponding to the different classifications. Finally, Section~\ref{sec:conclusion} concludes the survey with a summary of the researchers' observations after examining the recent literature.

\section{Background}\label{sec:background}

\subsection{Common Privacy-Preserving Techniques}
\label{ssec:privacy-preserving}

\begin{itemize}
	\item \emph{Secure Multiparty Computation}\\
	To maximize
	privacy or minimize information disclosure, Secure Multiparty Computation (SMC) is the goto technique which
	was first introduced by Yao's Millionaire problem \cite{Yao86}.
	This was extended to multiparty computations by
	Goldreich et al. \cite{Goldreich87}. SMC can be categorized as either information theoretic
	or computational \cite{BGW88}. In the computational model,
	the adversary is assumed to be bounded by polynomial-time. In the information theoretic model,
	the adversary is assumed to be unbounded. Much work exists to address various aspects (e.g.,
	complexity, adversarial behaviors, the number of corrupted parties) of SMC.
	There are generally two types of adversaries related to the SMC definitions: semi-honest and malicious \cite{Goldreich}.
	The semi-honest adversarial model often leads to more efficient privacy-preserving protocols, but the malicious model is less restrictive and thus more realistic.
	\item \emph{Randomization and Perturbation} \\
	The main idea in the perturbation approach is that data are modified (e.g., adding noise but preserving
	the underlying statistics) before being disclosed and analyzed.
	The key is that the original distribution can be
	roughly reconstructed from the perturbed data.
	The paper by Agrawal and Srikant \cite{Agrawal00} introduced this notion.
	In addition to additive noise, multiplicative noise can also be used \cite{LiuTKDE06}.

	\item \emph{$k$-Anonymity} \\
	The technique was developed to prevent external linking attacks \cite{kanonymity,kanon_defn}. The basic idea is that
	a dataset is $k$-anonymous if each record appears at least $k$ times according to
	a pre-defined set of quasi-identifier attributes.
	The main approach to achieve $k$-anonymity is information generalization and suppression.

\end{itemize}

\subsection{Collaborative Filtering}\label{ssec:collaborative-filtering}

CF addresses the problem of finding the degree to which a given user would like an item based on the knowledge of what similar users thought of that item and other similar items \cite{Su2009}. It is primarily based on the assumption that users who share similar interests for a subset of items will share the same opinion for other items \cite{Ahmad2007,Adomavicius2005}.

In formal terms, the system is given a set of $N$ items $$\mathbb{X} = \{x_i: i \in 1,2,...,N\},$$ a set of $M$ users $$\mathbb{U} = \{u_j: j \in 1,2,...,M\},$$ and a set of ratings $r_{i,j}$, where $r_{i,j}$ corresponds with the rating that item $x_i$ received from user $u_j$. Given a set of known ratings, CF attempts to find the predicted rating $\tilde{r}_{t,k}$ that user $u_k$ is most likely to assign to item $x_t$.

The range of values for the ratings varies by domain. Some domains only consider the explicit ratings that the users make, either by assigning a rating to an item or by making a purchase through the system \cite{1167344}. Other domains that lack access to this type of information often rely on other metrics, such as the number of user clicks on web pages and the relative time the user spends looking at a certain web page \cite{Das2007}.

CF techniques face many challenges, such as dealing with sparse rating datasets and large volumes of user and item data. They are often required to generate recommendations in real-time or with near real-time requirements, and incorporate new ratings from users while the system is in operation. Additionally, rating data is often noisy and subjective, which may lead to inaccurate recommendations \cite{Su2009}.

One of the main challenges for CF systems is maintaining the privacy of user information. Various attacks can be used on recommendation systems to expose the preferences and behavioral patterns of a specific user or identify a user within a particular dataset \cite{MacAonghusa2015}\note[IE]{This paragraph is too short and may be out of place.}.

CF approaches can be classified into two categories; memory-based and model-based. There are also hybrid techniques that incorporate both the memory-based and model-based approaches \cite{Adomavicius2005,Su2009}\note[IE]{This paragraph is too short.}.

In memory-based CF techniques, a heuristic is used to make rating predictions based on some or all of the ratings that are already known \cite{Adomavicius2005,Su2009}. They often rely on identifying users that share similar interests with a given user, and use the known ratings of those users to make predictions about the preferences of the user.

Different memory-based CF approaches employ different measures to compute the similarity between users \annote[IE]{and how they make their predictions}{awkward and unclear...}. For example, GroupLens, one of the early generation CF systems \cite{Resnick1994} used the Pearson correlation measure to find similar users, then made its prediction using an average of the ratings of similar users for the same item, weighted by the absolute value of the correlation between each of those users and the user in question.

Model-based CF techniques learn a model based on the known ratings, and use that model to predict how a user would rate a previously-unrated item \cite{Adomavicius2005,Su2009}. The learning system uses the training data to build a  model that represents the patterns and relationships in the training data, and then it uses that model to make predictions about the preferences of the users in the system.

Model-based CF systems vary based on the type of data they are analyzing. For categorical ratings, classification-based algorithms are often used to build the CF model. Conversely, regression models are often used when the ratings are numerical or continuous. An example of some early work on model-based CF \cite{Breese1998} used Bayesian models, where a probabilistic Bayesian network is built using the known ratings, and  used later to predict the value for an unknown rating.
\note[IE]{Add a sentence or two about Matrix Factorization, and cite our paper.}

Hybrid CF techniques join multiple approaches to build a more flexible and robust recommendation system. They often rely on domain knowledge and heuristics to take advantage of multiple aspects of the problem and provide more accurate and personalized recommendations. They may rely on item content, user demographic information, or combine both memory-based and model-based approaches to build the CF system \cite{Su2009}. Some work in the literature, such as \cite{Das2007}, joins both memory-based and model-based techniques into hybrid techniques. These aim at leveraging some details about the problem domain to build systems that bridge the gap between memory-based and model-based techniques, thus providing an added advantage in cases where a new user or item is being introduced to the system.

The interested reader is referred to \cite{Su2009} and \cite{Karydi2016}, which provide surveys of the key developments to the different CF approaches as well as an evaluation of the advantages and disadvantages of each approach.

\subsection{Related Studies}\label{ssec:related-work}

There have been some recent studies with similar goals that examined recent PPCF literature. In one of the more recent studies \cite{Ozturk2015276}, the authors looked at the current trends in PPCF and considered possible future trends for the field. The study looks at other recent survey papers, examines their strengths and shortcomings and then classifies the recent papers in the field with respect to the trends common between them and the different aspects of the CF algorithms used. The authors provided an analysis of the number of publications in the field over the past two decades, and identified three main goals that they think future research should address simultaneously: privacy, accuracy and online performance.

The authors of \cite{BILGE2013Survey} provided a comprehensive survey of the PPCF literature until 2013. They used several different attributes to classify the PPCF literature, including data partitioning cases and the techniques they used to preserve privacy, and presented guidelines and potential future directions for research in the PPCF field. It is one of the most comprehensive survey papers in recent years.

The study by \cite{Bilge2013a} provides a thorough comparison between different clustering-based PPCF approaches. It evaluates the capabilities of recent literature in clustering-based CF in terms of privacy, and provides ways to apply different families of clustering algorithms to the CF problem while keeping the user information private.

In \cite{6686270}, the authors classify the recent PPCF literature into two categories based on the centrality of data. They consider centralized methods to be ones in which the data is stored in one central location and decentralized methods to be ones where the data is distributed between multiple parties, or where users maintain control of their own rating profile. They discuss the current trends based on their survey of the literature and list a number of current PPCF challenges as well as new ones that they anticipate.

Other recent studies include \cite{Jeckmans2013263} and \cite{Batmaz2017}, which provide a more general examination of privacy in recommendation systems, as well as \cite{Gunes2012} and \cite{Lam2004}, which focus on one specific vulnerability known as shilling attacks.

\section{Survey of Recent Work}\label{sec:survey}

Due to the popularity of CF systems, and the prevalence of privacy concerns among users and developers of CF recommendation systems, there have been many contributions to this area of the literature in recent years. The study presented here takes a different approach that was not found in recent surveys on PPCF. It examines and classifies the papers in the field multiple times into broad categories, based on different factors. First, it highlights the way authors of different papers defined privacy and identifies the problem each paper was trying to solve. Then, the papers were classified with respect to how they solved the general PPCF problem, breaking them up into multiple categories corresponding to the most commonly used approaches in the literature.  This classification is meant to provide a broad perspective of the field and help researchers and practitioners interested in PPCF identify the approach that is best suited to their needs.

\subsection{Vulnerability}

Online CF recommendation systems can be subject to different vulnerabilities that compromise the privacy of the users' data and the overall integrity of the system. In this section, the recent PPCF literature is examined and the type of vulnerability that each study attempts to address is identified.

\subsubsection{User Profile Exposure}

Most of the recent PPCF literature attempts to address the general problem of user profile exposure by providing guarantees that the users' preferences remain private and are not exposed to other system users or system admins or being sold to marketing agencies. Table~\ref{tab:vulnerability} cites examples from recent PPCF literature where the studies focused on solving this problem.

The definition of the profile exposure vulnerability often varies, but the common factor used to identify papers addressing this vulnerability is that they focus on protecting the user profile data before it is stored or transmitted, using obfuscation, microaggregation or cryptographic approaches. Sometimes the objective is to protect the data from semi-honest third parties, while other times it is providing the user with guarantees that their profile data is safe from prying eyes, even within the domain of the service providing the CF recommendations.

\subsubsection{Inference Attacks}

The second most popular problem addressed by PPCF literature is that of inference attacks. In this category, the main challenge is protecting against malicious system users who try to inject fake profiles into the system or collect a lot of recommendation data, with the goal of inferring or identifying profiles of other users in the system. This can apply to either a semi-honest or a malicious attack model, depending on the level of security guarantee that the system provides. In contrast with user profile exposure, the system can have access to the complete user profile data, and the objective is to prevent other parties or system users from inferring information about that data. Examples of this category are cited in Table~\ref{tab:vulnerability}.

Studies that address vulnerability to inference attacks often rely on providing $k$-anonymity guarantees to user profiles such that a user profile cannot be identified within a group of $k$ profiles. They may rely on data obfuscation or clustering approaches, as well as microaggregation, to provide these guarantees. This is a more challenging problem in the area of PPCF, where a trade-off may be necessary between the accuracy of the provided recommendations and the degree of anonymity that the system provides.

\subsubsection{Shilling Attacks}

Another popular vulnerability that PPCF studies attempt to address is shilling attacks. This is an attack in which a malicious entity can create a large number of fake profiles that have false ratings with the goal of influencing the system towards or away from recommending certain items. The last row of Table~\ref{tab:vulnerability} cites examples of this type of study.

The study by \cite{Bilge20143671} discusses the sensitivity of different CF schemes to shilling attacks, and suggests that clustering-based CF methods are more robust and can detect and exclude fake profiles. Although this vulnerability may not influence the privacy of user data directly, it can greatly bias a CF recommendation system and decrease its accuracy and usability significantly.

\begin{table}[h]
  \centering
  \caption{Papers divided by vulnerability}
  \label{tab:vulnerability}
  \begin{tabular}{ll}
    \toprule
    \textbf{Type of Vulnerability} & \textbf{Papers} \\
    \midrule
    User Profile Exposure & \cite{Ahmad2007, 5616200, Aimeur2008, Amatriain2009, Armknecht201165, 6483317, baraglia2010preserving, Baraglia2006559, Basu20141706, Basu2012447, 6133147, 6676721, Basu2012, BILGE2013Survey, Bilge2010, Bilge2013a, Bilge2012, Bilge2013, Pathak2013, Borole20141364, Boutet2014169, 1004361, Castagnos2006, Castagnos2008247, 6406488, Erkin201361, Erkin201277, Jeckmans2014, Jeckmans2013400, 6261088, Kaleli2012, Kaleli2010, Kaleli201547, 6332047, 5066586, Kikuchi2013617, Li2016311, Li2011, Liu2015781, Luo2009, 6615356, Memis2013166, 6570605, 7217100, Montaner2002, 6680862, Okkalioglu2016199, 4403128, Polat2005a, 1250993, Polat2005b, Polatidis201562, Scipioni20112011, Shang2013, Smirnov2015455, 5474755, 7044660, Boutet2016827, Troiano201456, 6890141, Wang2013231, Wu2016144, Yakut2011, Zhang2014, 6682707, 7095527}       \\
    \midrule
    Inference Attacks     & \cite{MacAonghusa2015, Calandrino2011231, Canny2002, Casino2013490, Casino20151000, Chen2014218, Clemente2015725, Frey20155, Gao2014717, Honda201539, Honda201243, 7033684, Ramakrishnan2013, Shao2014244, Shokri2009, Wang2014224, Weinsberg:2012:BIO:2365952.2365989, Zhang2014320, Zhang2006, Zhao2015, 7195575, 6785787, Zhu2014}       \\
    \midrule
    Shilling Attacks      & \cite{Bilge20143671, bilge2013robust, Chirita2005, Gunes201513, Gunes201354, Gunes2012, Gunes20161, Lam2004, Okkalioglu2015901}      \\
    \bottomrule
  \end{tabular}
\end{table}

\subsection{Approaches to Privacy}

This survey's authors used their observations of the recent literature to divide the contributions into several categories below based on their approach to preserving privacy: cryptography-based, obfuscation-based, clustering-based, and heuristic-based techniques.

Cryptography-based techniques (\ref{ssec:cryptography}) rely on using cryptography to secure the user data and often use homomorphic encryption to allow computations to be applied to the data securely. Obfuscation-based techniques (\ref{ssec:obfuscation}), on the other hand, rely on applying transformations to the data to obfuscate or anonymize it, such that the individual users cannot be identified within the data or results. Clustering-based techniques (\ref{ssec:clustering}) use clustering to group users together into small communities, then extract features that are representative of this community as a whole, and use those to generate the recommendations. Therefore, this guarantees that no individual user's data will be identified. Finally, other techniques in the literature (\ref{ssec:heuristic}) rely on alternate methods, such as generating recommendations based on items instead of users, to generate the recommendations without exposing the user data.

In the following subsections, some of the recent work is categorized in terms of the approach to preserving privacy, and some potential advantages and disadvantages to each type of approach are examined.

\subsubsection{Cryptography-based techniques}\label{ssec:cryptography}

Cryptography-based PPCF appears to be the most popular category in recent literature. Techniques in this group rely on cryptographic measures to carry out the calculations needed for providing recommendations securely, without compromising the privacy of the users' data. They are often used to prevent the user profile exposure vulnerability discussed earlier, which most commonly occurs under a semi-honest attack model.

One of the earlier contributions to this area was by Canny \cite{1004361,Canny2002}. In \cite{1004361}, a cryptographic algorithm for computing a public aggregate of the data was presented that can then be used to securely generate personalized recommendations for individuals. It uses homomorphic encryption to apply the CF calculations on the data and decrypt the results without exposing the individual users' data. The work in \cite{Canny2002} extends the idea further by creating a method based on an Expectation Maximization probabilistic factor analysis model and using a privacy-preserving peer-to-peer homomorphic encryption protocol to apply the CF calculations. The interested reader is referred to \cite{1004361} and \cite{Canny2002} for a complete exposition of the approach summarized here.

In \cite{Ahmad2007}, the authors present an architecture for PPCF using the notion of distributed trust, which relies on a coalition of trusted servers instead of a single server. This distribution of trust among multiple servers makes the system more resilient against faults and attacks while providing an element of privacy for the users' data. The implementation of this architecture relies on a threshold homomorphic encryption protocol and was implemented and evaluated in an experimental setting.

The work presented in \cite{5066586} attempts to address one of the drawbacks of cryptography-based techniques, which is the high computational overhead of encrypting and decrypting large amounts of data. In this contribution, the authors rely on clustering the items and sampling the user data to reduce the computational burden of the cryptographic protocol. The reduced data is then used in a homomorphic cryptography scheme to securely generate CF recommendations with a significant reduction in computational time. In \cite{6332047}, the authors continue to address the goal of improving the time performance of cryptographic PPCF by introducing a quasi-homomorphic similarity measure that allows the use of local similarities to approximate the global similarity and analyze the accuracy of this approximation approach in addition to its running time improvement.

The contributions in \cite{6133147} and \cite{6676721} discuss some important practical considerations for implementing CF on cloud platforms, privacy and security being among the chief concerns of implementing such systems. In \cite{6133147}, they present a practical implementation of a PPCF system, based on the Google App Engine for Java (GAE/J) cloud platform. They designed algorithms that rely on a homomorphic encryption scheme to preserve the privacy of user data in the cloud. This work is further analyzed and extended in \cite{6676721} to address real world Software-as-a-service and Platform-as-a-service cloud settings.

Some of the more recent contributions addressed more specific concerns, such as horizontally-partitioned datasets \cite{6261088}, overlapped ratings \cite{6570605}, and updating the user preferences in real time \cite{7217100}. Others, such as \cite{6890141} and \cite{7044660}, designed cryptography-based PPCF algorithms for other tasks \cite{6890141} or incorporated cryptographic methods with other approaches to create efficient PPCF systems \cite{7044660}.

While cryptography-based techniques have the advantage of providing reliable security without sacrificing the accuracy of their results, one of the main concerns for this family of techniques is the scalability to systems that require the processing of millions of items and users, especially in online settings where the response time needs to be minimal while providing accurate recommendations.

\subsubsection{Obfuscation-based techniques}\label{ssec:obfuscation}

In obfuscation-based techniques, user profile data is transformed in some way that prevents individual users from being identified by using the data or the system's output, while maintaining the same or a close level of accuracy in the generated recommendations. This is often used to guard against inference attacks by other system users or external entities, following either a semi-honest or a malicious attack model.

The work introduced in \cite{1250993} relies on randomized perturbation techniques to introduce randomness in the data such that a user could not be individually identified with any certainty. Similarly, \cite{Kikuchi2013617} applies some randomness to the response provided by the system to prevent the preferences of individual users from being inferred from the system's responses while allowing the users to calculate the exact response using the randomized one.

Another popular approach in this category is based on the obfuscation of user data, and was examined by \cite{4403128, 7033684, 7195575}. Similar to perturbation, the relevant fields in user data are obfuscated to mask any identifying information in the data, preventing anyone from inferring the original users' preferences and information based on the obfuscated data.

Anonymization schemes combine the two previous approaches by removing the identifying fields from the data, obfuscating the values in other fields, and adding randomness to the data all to guarantee a level of $k$-anonimity for each user in the dataset. Examples of this scheme in recent literature include \cite{6615356} and \cite{Casino2013490}.

Obfuscation-based techniques have the advantage of being scalable since the transformations usually only need to be applied to the data at the point of origin, after which the obfuscated data can be used directly. However, the security of these techniques is harder to prove since it relies on randomness and anonymity, and it is harder to prove that a clever inference attack might not be able to re-identify some of the users. Another concern for this family of techniques is the accuracy, since adding randomness to the data can lead to the loss of some key information that some CF algorithms might be able to benefit from in providing more accurate recommendations.

\subsubsection{Clustering-based techniques}\label{ssec:clustering}

Clustering-based techniques for PPCF rely on grouping the users into clusters or communities, then extracting a representation of that cluster and using it, providing anonymity for the users within each cluster. This can be used to protect against inference attacks by semi-honest or malicious adversaries, provided that the clusters are large enough and are chosen appropriately. It can also be used as a user profile privacy guarantee, if the user profile data is not stored in the system, and only the cluster information is used to provide recommendations.

Contributions under this category may overlap the other categories, often relying on cryptographic techniques  \cite{7044660,6570605,5066586,Ahmad2007} or hashing \cite{6406488} to extract the representation of each cluster and perform the CF calculations securely.

This approach has an obvious advantage in terms of scalability over the sole use of cryptographic techniques, since a reduced representation of the data is used instead of trying to encrypt the entire data set. However, care must be taken when implementing such a technique, to ensure that the accuracy of the results does not suffer due to this reduced representation.

\subsubsection{Other Approaches}\label{ssec:heuristic}

Some contributions in recent literature (e.g.: \cite{5616200,5474755,6785787,6680862,6682707,7095527}) resorted to the use of alternative approaches to provide a reasonable compromise that still maintains users' privacy without incurring the computation cost required by other methods. These approaches are often based on knowledge of the problem domain and rely on algorithms that are especially designed to leverage this knowledge.

In \cite{5616200}, the authors built a CF system based on expert opinions, in which the items are rated by domain experts instead of relying on user ratings, thus eliminating the privacy concerns altogether. Similarly, the authors in \cite{5474755} substituted item-similarity in place of user-similarity in their CF scheme, which removed the need for user profile data in the system and instead used the item data to calculate the recommendations. Another item-based approach was used in \cite{6785787} and \cite{7095527}. They used a distributed belief propagation approach that relies on statistical measures to provide the users with recommendations based on item-similarity without the need to store the user preferences in the system.

Alternative approaches to PPCF have the advantage of leveraging the problem domain knowledge to avoid the need for costly cryptographic operations or applying transformations to the data. However, they can potentially be of limited use since they rely on knowledge from specific domains. Item-based approaches may also sacrifice on recommendation accuracy since the users' preferences may contain key information that helps the CF system provide accurate and relevant recommendations.

\section{Conclusion}\label{sec:conclusion}

This survey examined the recent literature on PPCF recommendation systems from a broad perspective. The contributions were classified based on the vulnerability they address, then divided into a number of categories representing the type of approach used in each contribution. The different categories were discussed in terms of the different vulnerabilities they address, and a discussion of some of the potential advantages and disadvantages of each category was provided. Considering the rising popularity of CF systems and the equal rise of privacy awareness by users, this survey aims to assist researchers and practitioners interested in the development of practical and secure CF systems identify the most suitable approach for their particular problem based on the recent contributions in the field.

\section*{Acknowledgement}

Partial support for this research was received from the Missouri University of Science and Technology Intelligent Systems Center, the Mary K. Finley Missouri Endowment, the National Science Foundation, the Lifelong Learning Machines program from DARPA/Microsystems Technology Office, and the  Army  Research  Laboratory (ARL); and it was accomplished under Cooperative Agreement Number W911NF-18-2-0260. The views and conclusions contained  in  this  document  are  those  of  the  authors  and  should not  be  interpreted  as  representing  the  official  policies,  either expressed  or  implied,  of  the  Army  Research  Laboratory  or the  U.S.  Government.  The  U.S.  Government  is  authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.

The authors would like to thank Emma Powell for her invaluable assistance with the organizational effort put into this survey.

\IEEEtriggeratref{109}
\bibliographystyle{IEEEtran}

redpajama_set_name: RedPajamaArXiv


text:
\subsection{The Princeton RSE Group Mission}
The Princeton RSE group's mission is to work directly with researchers create the most efficient, scalable, and sustainable research codes possible to enable new scientific and scholarly advances.
This is done by working as an integral part of traditional academic research groups, providing leadership in the design and construction of complex and highly customized software systems.
The group is committed to creating a collaborative environment in which best software engineering practices are valued, and to sharing and applying cross-disciplinary computational techniques to new and emerging areas.

\section{Organizational Details}
The Princeton Research Software Engineering group is housed with the Research Computing Department within the Office of Information Technology (OIT).
As of September 2022, the RSE group is led by one Director and two Associate Directors under the Associate CIO for Research Computing.
The group formed in late December 2016 with two members (a single manager and a single RSE) and has subsequently grown to 18 full-time staff as of September 2022.
The rate of growth can be seen in Figure~\ref{growth-image}.
Over the upcoming months, thanks to a broader strategic initiative led by the offices of the Provost and the Dean for Research that aims to invest in selected areas of research, the RSE group will expand by 10 new positions, for a total of 28 members \cite{expansion}.

Multiple funding mechanisms exist within the group and are described in the section below.
Regardless of funding source, each individual RSE works with a specific research ``partner.''
This partner can be an academic department, institute, center, consortium, or individual principal investigator (PI) and are chosen through an internal review process described in the following section.
An RSE steering committee, comprised of Princeton University faculty members, provides periodic oversight to the group, and serves as the review committee for partner award and selection.

\begin{figure*}
\centerline{\includegraphics[width=26pc]{rse-group-membership-2.png}}
\caption{The growth of the Princeton RSE group since January 2017, the month after its inception. This is a count of active members only - not predicted growth or open positions. Down ticks indicate an RSE has left Princeton.}
\label{growth-image}
\end{figure*}

\subsection{RSE Partner Proposal Process and Selection}
As central funding becomes available, a university-wide call for proposals is advertised broadly.
The proposal process is broken into two stages.
First, a one page letter Letter of Intent (LOI) must be submitted explaining how the partner would benefit from increased expertise in research software engineering.
Following an initial screening process, members of the RSE leadership team meet with the submitters to discuss the preparation of a full proposal.
Depending on the number of received LOIs this is either individuals meetings (when the number of LOI submissions is small) or via town hall information meetings (when the number of LOI submissions is large).
The purpose of these meetings is to provide guidance to the potential partners on the preparation of the full proposals.
After this initial step - a letter of intent followed by informational meetings - was introduced, the final proposals had a noticeable increase in quality.
The final proposals must be no more than three pages in length and include: the research area, how a professional RSE can enhance the research, the expertise that would be expected of such a person, the requested appointment level (e.g. RSE vs. Senior RSE), the expected amount of resource needed (e.g., 50\% of an RSE for 3 years), and information about any matching funding that may be provided (e.g., 50\% co-funding).
Co-funding is not required but is encouraged where possible.

\subsection{Proposal Evaluation Criteria}
Proposals are evaluated by the review committee based on the following primary and secondary evaluation criteria.\\

\noindent
Primary:
\begin{itemize}
\item{\emph{Research impact} - Is there a clear opportunity for software-driven impact in the given research area?}
\item{\emph{RSE innovation} - Is there a clear case for which the skills of an RSE are needed to enable the software impact?}
\item{\emph{Deliverables} - Are there clear, achievable objectives for the RSE project for new or improved software?}
\end{itemize}
Secondary:
\begin{itemize}
\item{\emph{Collaborative} - Are there additional collaborative impacts such as  training and engaging students and/or postdocs?}
\item{\emph{Strategic} - Is there an opportunity for impact beyond the specific initial project?}
\item{\emph{Sustainability} - Will there be ongoing impact such as positioning the project and/or PI for external funding, easier subsequent software engagement by students and postdocs, and extensibility and maintainability?}
\end{itemize}

\section{Funding}
Funding for the central RSE program is provided by OIT, the Office of the Provost, and the Office of the Dean for Research.
The RSE group currently supports two partnership models: Co-Funding and Full Partner Funding.

\textbf{Co-Funding} These RSEs have a home department in Research Computing and report formally to a member of the RSE leadership team (either the Director or an Associate Director).
A formal relationship is defined between each RSE and their partner.
These partners agree to fund a fraction (typically 0-50\%) of the position for a finite term (typically 1-3 years) while Research Computing, via one of the offices listed above, provides the remainder.
If, after the finite term, all parties involved (the partner, RSE, and RSE leadership) agree the partnership can continue indefinitely.
If, however, the term reaches an end and the partnership ends, the process for finding a new partnership will begin for the current RSE.
It is worth noting that co-funded RSE positions are considered permanent employees of the University and not restricted to the term of the partnership.
This is often an important detail during the recruiting process.

\textbf{Fully-Partner Funded} These RSE positions have a home department outside of Research Computing.
A partner fully funds an RSE position and commits to the RSE's participation in the RSE group with the same expectations as for other RSE members (including those who are co-funded).
This includes meeting the expectations listed in the next section.
The partner and RSE group leadership "co-supervise" the RSE.
The RSE's home department and reporting structure will be formally within the partner department/PI and the details of the co-supervision will be agreed upon at the time of hire.
The fully-partner funded RSEs may have finite term appointments subject to funding availability.

Regardless of funding method, the expectation is an open and collaborative evaluation between partners to ensure a successful engagement and as such, feedback from partners and all participating PI's is regularly sought.

\section{RSE Expectations}
Regardless of funding, each member of the RSE group is expected to work on the research software projects as prioritized by the partner with an effort of roughly 85\% of their time.
The RSE group's leadership team will support the RSE in ensuring that the priorities laid out by the partner are followed.
RSEs are expected to be independent and capable of managing their own time and efforts.
Awareness of an RSE's expected weekly effort by the partner and the RSE group leadership will contribute to continued success, and as such, performance evaluations are based on impact and production rather than timekeeping.
In many cases, RSEs are strongly encouraged to track their time.
This is done on a case-by-case basis depending on the specific situation of each RSE.
RSEs who are participating in more than one initiative typically track their time to ensure that each project is getting an appropriate effort.

The remaining 15\% of resource time is allocated to RSE group activities and professional development.
The RSE, along with the RSE group leadership, will track effort to ensure that the 15\% target is met as closely as possible.
Too much would take away from project work, too little limits growth and minimizes the ability to leverage the collective RSE group knowledge.
These activities include, but are not limited to, those listed below.

\textbf{Meetings}
While meetings tend to be minimized, two are essential.
First, a one-on-one meetings with their direct manager from the RSE group leadership team and a bi-weekly 90-minute RSE group meeting.
Additionally, a monthly Research Computing all hands meeting occurs once a month.

\textbf{RSE Group Events}
The RSE group organizes occasional internal activities to promote cross-collaboration and sharing.
Occasional (2-4 times per year) participation in these events is encouraged.

\emph{RSE project sprints} These short 1-3 day events provide a small team (2-4 RSEs) an opportunity to collaborate on a particularly challenging or technically edifying task.
This leverages group expertise and provides one project the opportunity to benefit from others as well as an opportunity for RSEs to learn new techniques and strategies.

\emph{RSE code reviews} The best way to improve code quality is through code review, and one of the best ways to learn new programming approaches is to read other people's code.
RSE peers can take turns reviewing each other's code for specific feedback and advice.
Both the reviewer and reviewee benefit, as does the project whose code is the subject of the review.

\textbf{Professional Development}
As largely independent professionals, RSEs must spend time and effort to maintain knowledge on advancing technologies.
Research changes rapidly.
Software development best practices change even faster.
Learning is essential in these positions.
Sometimes this is facilitated by participation in RSE group conversations.
However, a number of other options should be explored on a regular basis including: attending training, tutorials, conferences, contributing to open source projects and other project-based learning opportunities, and collaborating on other Princeton RSE group member projects.

\textbf{Deliver Training}.
As mentioned in the RSE overview, RSEs have knowledge of software engineering best practices that are extremely useful to share with researchers.
One of the most impactful approaches to sharing such information is through tutorials and workshops.
These are organized on an individual basis where interested RSEs are encouraged to deliver workshops and mini-courses to share their knowledge with the broader community.
This is typically considered optional for RSEs, and usually considered only after an RSE has been in their position for more than one year, however, many, if not most, Princeton RSEs participate.

\section{RSE Project Intake, Assignment \& Management}
Allocating individual RSE time to specific projects can be managed in a manner best suited for all parties involved.
Requirements tend to be unique and therefore the structure of project intake and assignment should be organized in a way that addresses the needs of a partner organization while maximizing the ability of an individual RSE to make a meaningful impact.
An established process for RSE project intake is important, especially when multiple parties are involved.
RSE project intake typically falls into one of the following models:

\subsection{PI project proposal \& review}
For co-funded positions, this is the most typical form of project intake and assignment, especially when the partner is a department or center and has many potential constituents.
At regular intervals, an email request for software project proposals is sent to associated faculty in the partner department/center.
Interested faculty respond with a short description of a potential project, and a meeting with the RSE and RSE group leadership is scheduled.
During the meeting, goals and deliverables of the project, an estimated scope, and the expected time commitment are discussed while guidelines for productive collaboration are established.
After spending time reviewing each proposal, the partner leadership (department chair/manager, etc.) meets with the RSE and RSE group leadership to select successful proposals and define the project commitment.
The goal is to balance the strategic priorities of the partner and the feasibility of the project with the interests/skills of the individual RSE.
This commitment is typically for a fraction of the RSE's availability within a fixed project cycle (typically 6-9 months).
Regular project check-ins are scheduled throughout the project cycle to ensure project goals are being met.
Renewals and small maintenance projects are possible through subsequent calls for software projects cycles.

\subsection{Single project partnership}
For fully-funded partnerships, this is the most common and simplest form of project assignment.
As the RSE is funded in alignment with a single project, that project is the sole focus of the RSE's effort.
As such, there is no need for a process to request, review and select new projects.
Occasional project check-ins and regular communication are still important to ensure project goals are being met.

\subsection{Ad-hoc projects}
Occasionally, an RSE will take on a project by direct request or through a meeting.
Together with the RSE group leadership and/or the partner representative, they manage the commitment and timeline.
While this model advantageously allows for flexibility and autonomy of the RSE, maintaining this model increases difficulty with predicting workloads which challenge the ability to advertise to potential PIs and can lead to an overwhelming amount of "walk-ins" and requests.

\section{Defining Expectations}
We strongly encourage communication between partners and both the individual RSEs and the RSE group leadership in the case of questions about the pace of development or the prioritization of effort.
Several software best practices initially slow down the addition of "new science" but facilitate the development of new aspects of a project while ensuring correctness, reproducibility, and the long-term success of the project.
To retain excellent RSEs, we want to ensure our work environment is as positive and rewarding as possible.
Enabling intellectual challenges, providing work-life balance, and promoting valued collaboration are all aspects of working at Princeton that keep us competitive.
If expectations are not being met, then it behooves everyone to quickly address the problem to collaboratively work towards a solution.
Typically, a new RSE collaboration represents the first time working with an RSE for partners, PIs, and researchers.
As a result, it's proven effective to define the role and expectations ahead of time with all those involved.

A formal \emph{RSE Partnership Guide} is published online and shared with future RSE partners and long-term collaborators.
The purpose of the document is to detail the role of RSEs along with operational details while setting expectations of all those involved.
It outlines much of what has been discussed here, such as project intake and management and publishing guidelines, as well as fine grained details such as how Annual Performance Reviews are conducted and physical desk locations.
The guide is shared with all potential partners prior to establishing a formal partnership.
The introduction of the guide has significantly reduced the number of questions and provided a smoother partnership initiation.

\subsection{RSE Publishing Guidelines}
RSEs sit at the intersection of scholarly research and software development.
A common and extremely important output of scholarly research is the journal/conference publication.
RSEs, by design, are not directly measured or graded based on their publication output, however many RSEs, especially those from a pure research background, still value such publications.
We encourage RSEs, PIs, and/or partner managers to discuss authorship as soon as possible.
It is typically recommended that if a publication would not have been possible without the RSE, then the RSE should be included as a co-author.

RSEs will occasionally be interested in publishing their work in a manner appropriate for the field and individual and RSEs may want to publish their work for a number of reasons.
In general, we view this as positive and useful for both the individual and the broader community.
To ensure all parties of an RSE collaboration understand the RSE publishing process, we have published on our website an \emph{RSE Publishing Guidelines} document.
This document gives detailed guidance for publications in which the RSE is either (a) the lead author with or without other members of the research group or (b) a co-author without any other members of the research group.
The key element of the guideline is frequent, open, and honest communication between the RSE and all parties, including PIs, graduate students, post doctoral researchers, and any other contributors to the research endeavor from which the publication may have resulted.
The full guidelines are available as part of the previously mentioned \emph{RSE Partnership Guide}.

\section{Advantages \& Disadvantages}
The Princeton RSE model described here developed as a result of Princeton University's unique funding and organizational environment.
The model has a number of positives, and like any organizational model, some disadvantages.
They are briefly listed them here for the reader to decide if such a model would be advantageous at a particular institute or organization.
The items listed here are specific to the RSE partner model, rather than more general advantages (e.g. advancing research, improved software sustainability, maintainability, and performance) and disadvantages (e.g. expense, recruiting, retaining) of Research Software Engineers.

\subsection{Advantages}
\textbf{Embedded Expertise} The partnership model has the distinct advantage that the scope and domain are both known ahead of time and stay relatively fixed over a long period.
The result is a customized job description and hiring search that allows for a very specific set of skills and knowledge in a successful candidate.
Then, over time, RSEs acquire domain expertise that makes them increasingly valuable to the projects and partners.
We've observed that after 1-2 years with the same project/partner that the depth of contribution possible by RSEs increases significantly.

\textbf{Group Management} Hiring RSEs and the subsequent management can take a significant amount of effort - something that researchers and partners frequently would not have time to adequately devote to.
RSEs managed and supervised by a dedicated RSE leadership team that prioritizes the entire RSE endeavor results in more complete hiring searches, increased focus on professional development, and technical support for individual RSEs.
This aspect is one of the initial appeals of the partnership model and a major reason fully-partner funded RSEs join the group.

\textbf{RSE Collaboration} The centralized nature of the RSE group allows for easy sharing, collaboration, and cooperation amongst the group members.
The group activities mentioned above (sprints, code review, etc.) are routinely cited by the group members as a unique opportunity to learn new technologies and develop new skills by working on a real and meaningful project.

\textbf{Sharing Best Practices} A frequent comment from researchers is that over time RSEs begin to serve as leaders and mentors to novice software developers, including undergraduates, graduate students, and postdoctoral researchers.
By providing mentorship on research software projects, RSEs elevate domain scientists' development through exposure to professional best practices often neglected in research software (e.g. version control, building unit tests, and documentation).
Also, RSEs within the group have taken an active role in designing and delivering training courses and workshops on research software engineering best practices.
Sessions typically range from 1-4 hours and targeting students and researchers, include topics ranging from continuous integration and automation, advanced git, to performance tuning.
These training programs are uniquely beneficial as RSEs understand the technology, audience, and requirements of research software.

\subsection{Disadvantages}
\textbf{Single Point of Failure} While the long-term commitment to specific partners allows an RSE to develop a deep collaboration and make increasingly meaningful contributions, it also leaves the project vulnerable should the RSE leave the position.
RSEs have marketable skills, desirable in other research settings and industry, which inevitably means that there is turnover withing RSEs.
When an RSE leaves a project it can leave a massive hole in the project as after years the RSEs can become nearly impossible to replace.
The gap between when an RSE leaves and when a new RSE can be hired and onboarded is time when project work can become completely stalled.

\textbf{Lack of Flexibility} The partnership model limits the ability to take projects outside the given partnership regardless of fit or need.
For example, if a specific RSE would be better suited for a project with \emph{Partner B} but works with \emph{Partner A} it's extremely difficult to switch to \emph{Partner B}.

\section{CONCLUSION}
The Princeton Central RSE group's growth is a testament to its impact on advancing and accelerating research.
What started as something of an experiment has grown well beyond the proof-of-concept stage straight into the strategic investment stage.
While the partnership model has proven incredibly effective, as the group has grown, so has the need to address some of the weaknesses of the model discussed earlier.

\section{ACKNOWLEDGMENT}

The author would like to thank all the Princeton community members who have contributed to the success of the program in its early years.
The list is too long to write out, but includes partners from virtually every academic department, PICSciE, and OIT, each of whom have consistently contributed ideas and effort to help instantiate this new RSE initiative.
I would also like to directly thank each individual RSE member of the group.
The success of the program is a direct result and testament to their sustained hard work and the transformative impact they have had on project after project.
Finally, I'd like to gratefully acknowledge and thank Curt Hillegas, Jeroen Tromp, and Jay Dominick for the consistent and overwhelming support of the RSE group at Princeton.
Without them, the RSE group would not just be less successful, it wouldn't exist.