\section{Introduction}
In the era of big data, recommender systems are playing a key  role
in tackling information overload, which largely improve the user experiences in a variety of applications, ranging from e-commerce, video sharing to healthcare assistant and on-line education.
The huge business value makes recommender systems become a longstanding research topic, with a large number of new models proposed each year~\citep{DLRS-survey}.
With the rapid growth of recommendation algorithms, these algorithms are usually developed under  different platforms or  frameworks. Typically, an experienced  researcher often  finds it difficult to implement the compared baselines in a unified way or framework. Indeed, many common components or procedures of these recommendation algorithms  are  duplicate or highly similar, which should be reused or extended.
Besides, we are aware that
there is an increasing concern about model reproducibility in the research community.
Due to some reasons, many published recommendation algorithms still lack public  implementations. Even with open source code, many details are implemented inconsistently (e.g., with different loss functions or optimization strategies) by different developers.
There is a need to re-consider the implementation of recommendation algorithms in a unified way.

In order to alleviate the above issues, we initiate a project  to provide a unified framework for developing recommendation algorithms.
We implement an open source recommender system library, called \emph{RecBole\xspace} (pronounced as [r\textepsilon k'bo\textipa{U}l\textschwa r]) \footnote{Bole was a  famous Chinese judge of horses in Spring and Autumn period, who was the legendary inventor of equine physiognomy (``\emph{judging a horse's qualities from appearance}''). Bole is frequently associated with the fabled \emph{qianlima} (a Chinese word) ``thousand-\emph{li} horse'', which was supposedly able to gallop one thousand \emph{li} (approximately 400 km) in a single day. Read more details about Bole at the wikipedia page via the link: \url{https://en.wikipedia.org/wiki/Bo_Le}. Here, we make an analogy between identifying qianlima horses and making good recommendations.}.
Based on this library, we would like to enhance the  reproducibility of existing models and ease the developing process of new  algorithms.  Our work is also useful to standardize the evaluation protocol of recommendation algorithms.
Indeed, a considerable number of recommender system libraries have been released in the past decade~\citep{Librec,MyMediaLite,NeuRec,ReChorus,daisyRec}. These works have largely advanced the progress of open source recommender systems.
Many libraries have made continuous improvement with increasingly added features.
We have extensively surveyed these libraries and broadly fused their merits into RecBole\xspace.
To summarize,  the key features and capabilities
 of our RecBole\xspace library are summarized in the following five aspects:

$\bullet$ Unified recommendation framework. We adopt PyTorch~\citep{PyTorch} to develop the entire recommender system library, since it is one of the most popular deep learning frameworks, especially in the research community.
As three core components of our library, we design and develop data modules, model modules, and evaluation modules, and encapsulate many common components, functions or procedures shared by different recommendation algorithms.
In our library, for reusing existing models, one can easily compare different recommendation algorithms with built-in evaluation protocols via simple  yet  flexible configuration; for developing new models, one  only needs to focus on a small number of interface functions, so that common parts  can be reused and implementation details are made transparent to the users.

$\bullet$ General and extensible data structure. For unified algorithm development, we implement the supporting data structures at two levels. At the user level, we introduce  \emph{atomic files} to format the input of mainstream recommendation tasks in a flexible way.
The proposed atomic files are able to characterize the input of four kinds of mainstream recommendation tasks.
At the algorithm level, we introduce a general data structure \textsf{Interaction} to unify the internal data representations tailored to GPU-based environment.  The design of \textsf{Interaction} is particularly convenient to develop new algorithms with supporting mechanisms or functions, e.g.,  fetching the data  by referencing feature name. We implement  \textsf{Dataset} and  \textsf{DataLoader} (two python classes) to automate the entire data flow, which greatly reduces the efforts for developing new models.

\begin{figure}[t]
\centering
\setlength{\fboxrule}{0.pt}
\setlength{\fboxsep}{0.pt}
\fbox{
\includegraphics[width=1.\linewidth]{framework.pdf}
}
\caption{The overall framework of our library RecBole\xspace.}
\vspace{-0.1cm}
\label{fw}
\end{figure}

$\bullet$ Comprehensive benchmark models and datasets.
So far, we have implemented 73\xspace recommendation algorithms, covering
the categories of general recommendation, sequential recommendation, context-aware recommendation and knowledge-based recommendation.
Besides traditional recommendation algorithms, we  incorporate a large number of neural algorithms proposed in recent years.
We provide flexible supporting mechanisms via the configuration files or command lines to run, compare and test these algorithms.
We also implement rich auxiliary  functions to use these models, including automatic parameter tuning and break-point resume.
To construct a reusable benchmark, we incorporate 28\xspace commonly used datasets for evaluating recommender systems.
With original dataset copies, a user can simply transform the data into a form that can be used in our library with the provided preprocessing tools or scripts.
More datasets and methods will be incorporated into our library.

$\bullet$ Efficient GPU-accelerated  execution. We design and implement a number of efficiency optimization techniques that are tailored to the GPU environment. As two major sources of time costs, both the model training and  testing are accelerated with GPU-oriented implementations.
For model test,  a special acceleration strategy is proposed to improve the efficiency of the full ranking for top-$K$ item recommendation. We   convert the  top-$K$ evaluation for all the users into the computation based on a unified matrix form. With this matrix form, we can utilize the GPU-version $\textsf{topk()}$  function in PyTorch to directly optimize  the top-$K$ finding procedure. Furthermore, such a matrix form is particularly convenient for generating the recommendations and computing the evaluation metrics. We empirically show that it  significantly reduces the time cost of the straightforward implementation without our acceleration strategy.

$\bullet$ Extensive and standard evaluation protocols.  Our library supports a series of widely adopted evaluation protocols for testing and comparing  recommendation algorithms.
It incorporates the various  evaluation settings discussed in \citep{comparison}.
Specially, we implement different combinations of item sorting (i.e., how to sort the items before data splitting) and data splitting (i.e., how to derive the train/validation/test sets) for deriving the evaluation sets. We also consider both full ranking and sample-based ranking, which is recently a controversial issue in the field of recommender system~\citep{sampledmetrics}.  We encapsulate four basic interfaces (namely \emph{Group}, \emph{Split}, \emph{Order} and \emph{NegSample})  to support the above evaluation protocols, which is flexible to include other evaluation settings. We provide a few commonly used evaluation settings (e.g., ratio-based splitting plus random ordering for dataset splitting), which integrates the alternative settings  of the above four factors. Our library provides a possibility to evaluate models under different evaluation settings.

\renewcommand\arraystretch{1.2}
\begin{table}[t]
\centering
\caption{Collected datasets in our library RecBole\xspace.}
\vspace{-0.cm}
\label{tab:public-dataset}
\begin{threeparttable}
\begin{tabular}
{p{2.5cm}<{\centering}||p{1.4cm}<{\centering}|p{1.4cm}<{\centering}|p{1.7cm}<{\centering}}
\hline
Dataset & \#Users & \#Items & \#Interactions\\\hline\hline
MovieLens  & - & - & - \\
Anime & 73,515 & 11,200 & 7,813,737 \\
Epinions & 116,260 & 41,269 & 188,478\\
Yelp & 1,968,703 & 209,393 & 8,021,122  \\
Netflix & 480,189 & 17,770 & 100,480,507 \\
Book-Crossing & 105,284 & 340,557 & 1,149,780 \\
Jester & 73,421 & 101 & 4,136,360  \\
Douban & 738,701 & 28 & 2,125,056\\
Yahoo Music & 1,948,882 & 98,211 & 11,557,943 \\
KDD2010 & - & - & -  \\
Amazon & - & - & -  \\
Pinterest & 55,187 & 9,911 & 1,445,622 \\
Gowalla & 107,092 & 1,280,969 & 6,442,892 \\
Last.FM & 1,892 & 17,632 & 92,834  \\
DIGINETICA & 204,789 & 184,047 & 993,483  \\
Steam & 2,567,538 & 32,135 & 7,793,069 \\
Ta-Feng & 32,266 & 23,812 & 817,741 \\
FourSquare & - & - & - \\
Tmall & 963,923 & 2,353,207 & 44,528,127 \\
YOOCHOOSE & 9,249,729 & 52,739 & 34,154,697 \\
Retailrocket & 1,407,580 & 247,085 & 2,756,101 \\
LFM-1b & 120,322 & 3,123,496 & 1,088,161,692 \\
Criteo & - & - & 45,850,617 \\
Avazu & - & - & 40,428,967 \\
iPinYou & 19,731,660 & 163 & 24,637,657 \\
Phishing websites & - & - & 11,055 \\
Adult & - & - & 32,561 \\
MIND & - & - & -\\
\hline
\end{tabular}
\begin{tablenotes}
   \footnotesize
   \item[1] ``-'' means the dataset is either composed of many small subsets (e.g., Amazon, KDD2010), so that we refer the readers to our website for more detail statistics or the dataset is based on the features (e.g., Criteo, Avazu), instead of  users and items.
\end{tablenotes}
\end{threeparttable}
\vspace{-0.2cm}
\end{table}

\section{The Library --- RecBole\xspace}
The overall framework of our library RecBole\xspace is presented in Figure~\ref{fw}.
The bottom part is the configuration module, which helps users to set up the experimental environment (e.g., hyperparameters and running details).
The data, model and evaluation modules are built upon the configuration module, which forms the core code of our library.
The execution module is responsible for running and evaluating the model based on specific settings of the environment.
All the auxiliary functions are collected in the utility module, including automatic parameter tuning, logger and evaluation metrics.
In the following, we briefly present the designs of three core modules, and more details can be found in the library documents.

\subsection{Data Module}
A major development guideline of our library is to make the code highly
self-contained and unified. For this purpose, data module is indeed the most important part that supports the entire library by  providing fundamental data structures and functions.

\subsubsection{The Overall Data Flow}
For extensibility and reusability, our data module designs an elegant data flow that transforms raw data into the model input.

The overall data flow can be described as follows: \underline{raw input} $\rightarrow$ \underline{atomic files}  $\rightarrow$ \underline{\textsf{Dataset}$_{DataFrame}$}  $\rightarrow$ \underline{\textsf{Dataloader}$_{Interaction}$} $\rightarrow$ \underline{algorithms}.
The implementation of  class \textsf{Dataset} is mainly based on the primary data structure of  \textsf{pandas.DataFrame} in the library of pandas, and the implementation of  class   \textsf{Dataloader}  is based on a general internal data structure called  \textsf{Interaction}.

Our data flow involves two special data forms, which are oriented to users and algorithms, respectively.  For data preparation, we introduce and define six \emph{atomic file types} (having the same or similar file format) for unifying the input at the user level. While, for internal data representations, we introduce and implement a flexible data  structure  \textsf{Interaction} at the algorithm level.
The atomic files are able to characterize most forms of the input data  required by different recommendation tasks, and the \textsf{Interaction} data structure provides a unified internal data representation for different recommendation algorithms.

In order to help users transform raw input into atomic files,  we have collected more than 28\xspace commonly used datasets and released the corresponding conversion tools, which makes it quite convenient to start with our library. We present the statistics of these datasets in Table~\ref{tab:public-dataset}.
During the transformation step from atomic files to class \textsf{Dataset}, we provide many useful functions that support a series of preprocessing steps in recommender systems, such as $k$-core data filtering and  missing value imputation. We present the  functions supported by class \textsf{Dataset} in Table~\ref{tb-dataset}.

\renewcommand\arraystretch{1.2}
\begin{table}[t]
\centering
\caption{The functions supported by class \textsf{Dataset}.}
\vspace{-0.cm}
\label{tb-dataset}
\begin{tabular}{p{3.cm}<{\centering}||p{4.5cm}<{\centering}}
\hline
Function & Description\\
\hline
\hline
\textsf{\_filter\_by\_inter\_num} & frequency based user/item filtering\\
\textsf{\_filter\_by\_field\_value} & value based filtering\\
\textsf{\_remap\_ID} & map the features to IDs \\
\textsf{\_fill\_nan} & missing value imputation \\
\textsf{\_set\_label\_by\_threshold} & generate interaction labels\\
\textsf{\_normalize} & normalize the features\\
\textsf{\_preload\_weight\_matrix} & initialize embedding tables \\
\hline
\end{tabular}
\end{table}

\subsubsection{Atomic Files}

So far, our library introduces six atomic file types, which are served as basic components for characterizing the input of various recommendation tasks.
In the literature, there is a considerable number of recommendation tasks.
We try to summarize and unify the most basic input forms for mainstream  recommendation tasks.
Note that these files are only functionally different while their formats are rather similar.
The details of these atomic files are summarized in Table~\ref{tb-atom}.

\renewcommand\arraystretch{1.2}
\begin{table}[t]
\centering
\caption{Summarization of the atomic files.}
\vspace{-0.cm}
\begin{threeparttable}
\begin{tabular}{p{1.cm}<{\centering}||p{1.5cm}<{\centering}|p{4.2cm}<{\centering}}
\hline
Suffix&Data types&Content\\ \hline\hline
\textsc{.inter}&all types&User-item interaction\\
\textsc{.user}&all types&User feature \\
\textsc{.item}&all types&Item feature\\
\textsc{.kg}&int&Triplets in a knowledge graph \\
\textsc{.link}&int&Item-entity linkage data\\
\textsc{.net}&all types&Social graph data\\ \hline
\end{tabular}
\end{threeparttable}
\label{tb-atom}
\vspace{-0.cm}
\end{table}

We identify different files by their suffixes.
By summarizing existing recommendation models and datasets, we conclude with four basic data types, i.e., ``\textsf{token}'' (representing integers or strings), ``\textsf{token sequence}'', ``\textsf{float}'' and ``\textsf{float sequence}''.
``\textsf{token}'' and ``\textsf{token sequence}'' are used to represent discrete features such as ID or category, while ``\textsf{float}'' and ``\textsf{float sequence}'' are used to represent  continuous features, such as price.
Atomic files support sparse feature representations, so that the space taken by the atomic files can be largely reduced.
Most of atomic files support all the four data types except the \textsc{.kg} and \textsc{.link} files.
Next, we present the detailed description of each atomic file:

$\bullet$ \textsc{.inter} is a mandatory file  used in all the recommendation tasks. Each line is composed of the user ID (\textsf{token}), item ID (\textsf{token}), user-item rating (\textsf{float}, optional), timestamp (\textsf{float}, optional) and review text (\textsf{token sequence}, optional). Different fields are separated by  commas.

$\bullet$ \textsc{.user} is a user profile file, which includes users' categorical or continuous features. Each line is formatted as user ID (\textsf{token}), feature (\textsf{token} or \textsf{float}), feature (\textsf{token} or \textsf{float}), ..., feature (\textsf{token} or \textsf{float}).

$\bullet$ \textsc{.item} is an item feature file, which describes the item characteristics, and the format is as follows: item ID (\textsf{token}), feature (\textsf{token} or float), feature (\textsf{token} or \textsf{float}), ..., feature (\textsf{token} or \textsf{float}). \textsc{.user} and \textsc{.item} are used for context-aware recommendation.

$\bullet$ \textsc{.kg} is a knowledge graph file  used for knowledge-based recommendation. Each line corresponds to a $\langle head, tail, relation \rangle$ triplet, and the format is as follows: head entity ID (\textsf{token}), tail entity ID (\textsf{token}), relation ID (\textsf{token}).

$\bullet$ \textsc{.link} is also used for knowledge-based recommendation. It records the correspondence between the recommender systems items and the knowledge graph entities. The file format is as follows: item ID (\textsf{token}), entity ID (\textsf{token}), which denotes the item-to-entity mapping.

$\bullet$ \textsc{.net} is a social network file  used for social recommendation. The format is as follows: source user ID (\textsf{token}), target user ID (\textsf{token}), weight (\textsf{float}, optional).

The essence of the atomic files is feature-based data frames corresponding to different parts of the task input. They can cover the input of most mainstream recommendation tasks in the literature.
In case the atomic files are not sufficient to support new tasks, one can incrementally introduce new atomic files in a flexible way.

\subsubsection{Input Files for Recommendation Tasks}

Based on the above atomic files, we can utilize a series of file combinations to facilitate five mainstream recommendation tasks, namely \emph{general recommendation}, \emph{context-aware recommendation}, \emph{knowledge-based recommendation}, \emph{sequential recommendation} and \emph{social recommendation}.
Currently, we have implemented the supporting mechanisms for the first four kinds of recommendation tasks, while  the code for social recommendation is under development.

The correspondence between  atomic files and  recommendation models are presented in Table~\ref{tb-cor}.
A major merit of our input files is that atomic files themselves are not dependent on  specific tasks. As we can see, given a dataset, the user can reuse the same \textsc{.inter} file (without any modification  on data files) when switching between different recommendation tasks. Our library reads the configuration file and determines what to do with the data files.

Another note is that Table~\ref{tb-cor} presents the combination of mandatory atomic files in each task. It is also possible to use additional atomic files besides mandatory  files. For example,
for sequential recommendation, we may also need to use context features.  To support this, one can simply extend the original combination to $\langle$\textsc{.inter}, \textsc{.user}, \textsc{.item}$\rangle$ as needed.

\renewcommand\arraystretch{1.2}
\begin{table}[t]
\centering
\caption{Correspondence between the recommendation task and the atomic files.}
\vspace{-0.cm}
\begin{threeparttable}
\begin{tabular}{p{4.6cm}<{\centering}||p{3.cm}<{\centering}}
\hline
Tasks &Mandatory atomic files\\ \hline\hline
General Recommendation&\textsc{.inter}\\
Context-aware Recommendation&\textsc{.inter}, \textsc{.user}, \textsc{.item}\\
Knowledge-based Recommendation&\textsc{.inter}, \textsc{.kg}, \textsc{.link}\\
Sequential Recommendation&\textsc{.inter} \\
Social Recommendation&\textsc{.inter}, \textsc{.net} \\
 \hline
\end{tabular}
\end{threeparttable}
\label{tb-cor}
\vspace{-0.cm}
\end{table}

\renewcommand\arraystretch{1.2}
\begin{table}[t]
\centering
\caption{The functions that class \textsf{Interaction} supports.}
\vspace{-0.cm}
\label{tb-interaction}
\begin{tabular}{p{2.5cm}<{\centering}||p{5.cm}<{\centering}}
\hline
Function & Description\\
\hline
\hline
\textsf{to(device)} & transfer tensors to \textsf{torch.device} \\
\textsf{cpu} & transfer all tensors to CPU \\
\textsf{numpy} & transfer all tensors to \textsf{numpy.Array} \\
\textsf{repeat} & repeats along the \textsf{batch\_size} dimension \\
\textsf{repeat\_interleave} & repeat elements of a tensor\\
\textsf{update} & update an object with other \textsf{Interaction}\\
\hline
\end{tabular}
\vspace{-0.cm}
\end{table}

\subsubsection{The Internal Data Structure \textsc{Interaction}}
As discussed in Section 2.1.1, in our library, \textsc{Interaction} is the internal data structural that is fed into the recommendation algorithms.

In order to make it unified and flexible, it is implemented as a new abstract data type based on \textsf{python.dict}, which is a key-value indexed data structure.
The keys correspond to  \emph{features} from input, which  can be conveniently referenced with feature names when writing the recommendation algorithms; and the values correspond to \emph{tensors} (implemented by \textsf{torch.Tensor}), which  will be used for the update and computation in learning algorithms.  Specially, the value entry for a specific key stores all the corresponding tensor data in a batch or mini-batch.

With such a data structure, our library provides a friendly interface to implement the recommendation algorithms in a batch-based mode.
All the details of the transformation from raw input to internal data representations are transparent to the developers. One can implement  different algorithms easily based on unified internal data representation \textsf{Interaction}.
Besides, the value components are implemented based on \textsf{torch.Tensor}. We wrap many  functions of PyTorch to develop a GRU-oriented data structure, which can support batch-based mechanism (e.g., copying a batch of data to GPU). Specially, we summarize the important functions that \textsf{Interaction} supports in Table~\ref{tb-interaction}.

\renewcommand\arraystretch{1.1}
\begin{table*}[t]
\caption{Implemented 73\xspace recommender models in RecBole on 4 categories.}\label{models}
\vspace{-0.2cm}
	\footnotesize
	\scalebox{.9}{
	\begin{tabular}{p{1.8cm}<{\centering}|p{4.cm}<{\centering}|p{1.2cm}<{\centering}|p{1.cm}<{\centering}|p{6.7cm}<{\centering}}
		\hline
		\textbf{Category}                                                             & \textbf{Model}                                       & \textbf{Conference} & \textbf{Year} & \textbf{Typical Evaluation Dataset}                                                    \\ \hline
		\multicolumn{1}{c|}{\multirow{24}{*}{\normalsize{\rotatebox{90}{{General Recommendation}}}}}       & popularity                                           & -                   & -             & -                                                                   \\
		\multicolumn{1}{c|}{}                                                         & ItemKNN~\cite{itemKNN}                               & TOIS                & 2004          & ctlg, ccard, ecmrc, EachMovie, MovieLens, skill                     \\
		\multicolumn{1}{c|}{}                                                         & BPR~\cite{BPRMF}                                     & UAI                 & 2009          & Rossmann, Netflix                                                   \\
		\multicolumn{1}{c|}{}                                                         & SLIMElastic~\cite{SLIM}                              & ICDM                & 2011          & ctlg, ccard, ecmrc,Book-Crossing, MoiveLens, Netflix, Yahoo   Music \\
		\multicolumn{1}{c|}{}                                                         & FISM~\cite{FISM}                                     & SIGKDD              & 2013          & MovieLens, Netflix, Yahoo Music                                     \\
		\multicolumn{1}{c|}{}                                                         & LINE~\cite{NNRec,LINE}                                     & WWW                 & 2015          & NetWork~(Wikipedia, Flickr, Youtube, DBLP)                              \\
		\multicolumn{1}{c|}{}                                                         & CDAE~\cite{CDAE}                                     & WSDM                & 2016          & MovieLens, Netflix, Yelp                                            \\
		\multicolumn{1}{c|}{}                                                         & NeuMF~\cite{NeuMF}                                   & WWW                 & 2017          & MovieLens, Pinterest                                                \\
		\multicolumn{1}{c|}{}                                                         & ConvNCF~\cite{ConvNCF}                               & IJCAI               & 2017          & Yelp, Gowalla                                                       \\
		\multicolumn{1}{c|}{}                                                         & DMF~\cite{DMF}                                       & IJCAI               & 2017          & MovieLens, Amazon                                                   \\
		\multicolumn{1}{c|}{}                                                         & NNCF~\cite{NNCF}                                     & CIKM                & 2017          & Delicious, MovieLens, Rossmann                                      \\
		\multicolumn{1}{c|}{}                                                         & NAIS~\cite{NAIS}                                     & TKDE                & 2018          & MovieLens, Pinterest                                                \\
		\multicolumn{1}{c|}{}                                                         & SpectralCF~\cite{SpectralCF}                         & RecSys              & 2018          & MovieLens, HetRec, Amazon                                           \\
		\multicolumn{1}{c|}{}                                                         & MultiVAE~\cite{MultiVAE}                             & WWW                 & 2018          & MovieLens, Million Song, Netflix                                    \\
		\multicolumn{1}{c|}{}                                                         & MultiDAE~\cite{MultiVAE}                             & WWW                 & 2018          & MovieLens, Million Song, Netflix                                    \\
		\multicolumn{1}{c|}{}                                                         & GCMC~\cite{GCMC}                                     & SIGKDD              & 2018          & MovieLens, Flixster,Douban, Yahoo Music                             \\
		\multicolumn{1}{c|}{}                                                         & NGCF~\cite{NGCF}                                     & SIGIR               & 2019          & Gowalla, Yelp, Amazon                                               \\
		\multicolumn{1}{c|}{}                                                         & MacridVAE~\cite{MacridVAE}                           & NeurIPS             & 2019          & AliShop-7C, MovieLens, Netflix                                      \\
		\multicolumn{1}{c|}{}                                                         & EASE~\cite{EASE}                                     & WWW                 & 2019          & MovieLens, Million Song, Netflix                                    \\
		\multicolumn{1}{c|}{}                                                         & LightGCN~\cite{LightGCN}                             & SIGIR               & 2020          & Gowalla, Yelp, Amazon                                               \\
		\multicolumn{1}{c|}{}                                                         & DGCF~\cite{DGCF}                                     & SIGIR               & 2020          & Gowalla, Yelp, Amazon                                               \\
		\multicolumn{1}{c|}{}                                                         & RaCT~\cite{RaCT}                                     & ICLR                & 2020          & MovieLens, Million Song, Netflix                                    \\
		\multicolumn{1}{c|}{}                                                         & RecVAE~\cite{RecVAE}                                 & WSDM                & 2020          & MovieLens, Million Song, Netflix                                    \\
		\multicolumn{1}{c|}{}                                                         & ENMF~\cite{ENMF}                                     & TOIS                & 2020          & Ciao, Epinions, MovieLens                                           \\
		\hline
		\multicolumn{1}{c|}{\multirow{17}{*}{\normalsize{\rotatebox{90}{{Context-aware recommendation}}}}}  & LR~\cite{LR}                                         & WWW                 & 2007          & Microsoft web search dataset                      \\
		\multicolumn{1}{c|}{}                                                         & FM~\cite{FM}                                         & ICDM                & 2010          & CML/PKDD Discovery Challenge 2009, Netflix                          \\
		\multicolumn{1}{c|}{}                                                         & DSSM~\cite{DSSM}                                     & CIKM                & 2013          & Henceforth                                                          \\
		\multicolumn{1}{c|}{}                                                         & FFM~\cite{FFM}                                       & RecSys              & 2016          & Criteo, Avazu                                                       \\
		\multicolumn{1}{c|}{}                                                         & FNN~(DNN)~\cite{FNN}                                  & ECIR                & 2016          & iPinYou                                                             \\
		\multicolumn{1}{c|}{}                                                         & PNN~\cite{PNN}                                       & ICDM                & 2016          & Criteo, iPinYou                                                     \\
		\multicolumn{1}{c|}{}                                                         & Wide\&Deep~\cite{Wide}                               & RecSys              & 2016          & Google play dataset                                       \\
		\multicolumn{1}{c|}{}                                                         & XGBoost~\cite{XGB}                                   & KDD                 & 2016          & Allstate, Higgs Boson, Yahoo LTRC, Criteo                           \\
		\multicolumn{1}{c|}{}                                                         & NFM~\cite{NFM}                                       & SIGIR               & 2017          & Frappe,MovieLens                                                    \\
		\multicolumn{1}{c|}{}                                                         & DeepFM~\cite{DeepFM}                                 & IJCAI               & 2017          & Criteo, Company                                                     \\
		\multicolumn{1}{c|}{}                                                         & AFM~\cite{AFM}                                       & IJCAI               & 2017          & Frappe, MoiveLens                                                   \\
		\multicolumn{1}{c|}{}                                                         & DCN~\cite{DCN}                                       & ADKDD               & 2017          & Criteo                                                              \\
		\multicolumn{1}{c|}{}                                                         & LightGBM~\cite{LightGBM}                             & NIPS                & 2017          & Allstate, Flight Delay, LETOR, KDD10, KDD12                         \\
		\multicolumn{1}{c|}{}                                                         & xDeepFM~\cite{xDeepFM}                               & SIGKDD              & 2018          & Criteo, Dianping, Bing News                                         \\
		\multicolumn{1}{c|}{}                                                         & FwFM~\cite{FwFM}                                     & WWW                 & 2018          & Criteo, Oath                                                        \\
		\multicolumn{1}{c|}{}                                                         & DIN~\cite{DIN}                                       & SIGKDD              & 2018          & Amazon, MovieLens, Alibaba                                          \\
			\multicolumn{1}{c|}{}
		&DIEN~\cite{DIEN}                                     & AAAI                &
		2019		  & Amazon                                                               \\
		\multicolumn{1}{c|}{}                                                         & AutoInt~\cite{AutoInt}                               & CIKM                & 2019          & Criteo, Avazu, KDD Cup 2012, MovieLens                              \\
		\hline
		\multicolumn{1}{c|}{\multirow{23}{*}{\normalsize{\rotatebox{90}{{Sequential recommendation}}}}}     & FPMC~\cite{FPMC}                                     & WWW                 & 2010          & ROSSMANN                                                            \\

		\multicolumn{1}{c|}{}                                                         & HRM~\cite{HRM}                                       & SIGIR               & 2015          & Ta-Feng, BeiRen, Tmall                                              \\
		\multicolumn{1}{c|}{}                                                         & Improved GRU-Rec~\cite{ImprovedGRU4Rec}              & DLRS                & 2016          & YOOCHOOSE                                                           \\
		\multicolumn{1}{c|}{}                                                         & GRU4RecF(+feature embedding)~\cite{GRU4RecF}         & RecSys              & 2016          & coined VIDXL, CLASS                                                 \\
		\multicolumn{1}{c|}{}                                                         & Fossil~\cite{Fossil}                                 & ICDM                & 2016          & Amazon, Epinions, Foursquare                                        \\
		\multicolumn{1}{c|}{}                                                         & NARM~\cite{NARM}                                     & CIKM                & 2017          & YOOCHOOSE, DIGINETICA                                               \\
		\multicolumn{1}{c|}{}                                                         & TransRec~\cite{TransRec}                             & RecSys              & 2017          & Amazon, Epinions, Foursquare, Google Local                          \\
		\multicolumn{1}{c|}{}                                                         & STAMP~\cite{STAMP}                                   & SIGKDD              & 2018          & YOOCHOOSE, DIGINETICA                                               \\
		\multicolumn{1}{c|}{}                                                         & Caser~\cite{Caser}                                   & WSDM                & 2018          & MovieLens, Gowalla, Foursquare, Tmall                               \\
		\multicolumn{1}{c|}{}                                                         & SASRec~\cite{SASRec}                                 & ICDM                & 2018          & Amazon, Steam, MovieLens                                            \\
		\multicolumn{1}{c|}{}                                                         & KSR~\cite{KSR}                                       & SIGIR               & 2018          & LastFM, MovieLens, Amazon                                           \\
		\multicolumn{1}{c|}{}                                                         & SHAN~\cite{SHAN}                                     & IJCAI               & 2018          & Tmall, Gowalla                                                      \\
		\multicolumn{1}{c|}{}                                                         & NPE~\cite{NPE}                                       & IJCAI               & 2018          & Movielens, Online Retail, TasteProfile                              \\
		\multicolumn{1}{c|}{}                                                         & NextItnet~\cite{NextItNet}                           & WSDM                & 2019          & YOOCHOOSE, LastFM                                                   \\
		\multicolumn{1}{c|}{}                                                         & BERT4Rec~\cite{BERT4Rec}                             & CIKM                & 2019          & Amazon, Steam, MovieLens                                            \\
		\multicolumn{1}{c|}{}                                                         & SRGNN~\cite{SRGNN}                                   & AAAI                & 2019          & YOOCHOOSE, DIGINETICA                                               \\
		\multicolumn{1}{c|}{}                                                         & GCSAN~\cite{GCSAN}                                   & IJCAI               & 2019          & DIGINETICA, Retailrocket                                            \\
		\multicolumn{1}{c|}{}                                                         & SASRecF(+feature embedding)~\cite{SASRecF}           & IJCAI               & 2019          & -                                                                   \\
		\multicolumn{1}{c|}{}                                                         & FDSA~\cite{FDSA}                                     & IJCAI               & 2019          & Amazon, Tmall                                                       \\
		\multicolumn{1}{c|}{}                                                         & RepeatNet~\cite{RepeatNet}                           & AAAI                & 2019          & YOOCHOOSE, DIGINETICA, LastFM                                       \\
		\multicolumn{1}{c|}{}                                                         & HGN~\cite{HGN}                                       & SIGKDD              & 2019          & MovieLens, Amazon, Goodreads                                        \\
		\multicolumn{1}{c|}{}                                                         & S3Rec~\cite{S3Rec}                                   & CIKM                & 2020          & Meituan, Amazon, Yelp, LastFM                                       \\
		\multicolumn{1}{c|}{}                                                         & GRU+KG Embedding                                     & -                   & -             & -                                                                   \\   \hline
		\multicolumn{1}{c|}{\multirow{8}{*}{\normalsize{\rotatebox{90}{\tabincell{c}{Knowledge-based \\ recommendation}}}}} & CKE~\cite{CKE}                                       & SIGKDD              & 2016          & MovieLens, IntentBooks                                              \\
		\multicolumn{1}{c|}{}                                                         & CFKG~\cite{CFKG}                                     & MDPI                & 2018          & Amazon                                                              \\
		\multicolumn{1}{c|}{}                                                         & RippleNet~\cite{RippleNet}                           & CIKM                & 2018          & MovieLens, Book-Crossing, Bing-News                                 \\
		\multicolumn{1}{c|}{}                                                         & KTUP~\cite{KTUP}                                     & WWW                 & 2019          & MovieLens, DBbook2014                                               \\
		\multicolumn{1}{c|}{}                                                         & KGAT~\cite{KGAT}                                     & SIGKDD              & 2019          & Amazon, LastFM, Yelp2018                                            \\
		\multicolumn{1}{c|}{}                                                         & MKR~\cite{MKR}                                       & WWW                 & 2019          & MovieLens, Book-Crossing, LastFM, Bing-News                         \\
		\multicolumn{1}{c|}{}                                                         & KGCN~\cite{KGCN}                                     & WWW                 & 2019          & MovieLens, Book-Crossing, LastFM                                    \\
		\multicolumn{1}{c|}{}                                                         & KGNN-LS~\cite{KGNN-LS}                               & SIGKDD              & 2019          & MovieLens, Book-Crossing, LastFM, Dianping-Food                      \\ \hline
	\end{tabular}
}
\vspace{-0.cm}
\end{table*}

\renewcommand\arraystretch{1.2}
\begin{table}[t]
\caption{Example evaluation settings.}\label{tb-protocol}
\vspace{-0.cm}
\centering
\begin{tabular}{p{1.3cm}<{\centering}||p{6.4cm}<{\centering}}
\hline
{Notation} & {Explanation} \\
\hline
\hline
RO\_RS & Random  Ordering + Ratio-based Splitting\\
TO\_LS & Temporal  Ordering + Leave-one-out Splitting\\
RO\_LS & Random  Ordering + Leave-one-out Splitting\\
TO\_RS & Temporal  Ordering + Ratio-based Splitting\\
full & Full ranking with all item candidates\\
uni$N$ & One positive item is paired with $N$ negative items\\
\hline
\end{tabular}
\vspace{-0.cm}
\end{table}

\subsection{Model Module}

Based on the data module, we organize the implementations of recommendation algorithms in a separate model module.

\subsubsection{Unified Implementation Interface}
By setting up the model module, we can largely decouple the algorithm implementation from other components, which is particularly important to collaborative  development of this library.
To implement a new model within the four tasks in Table~\ref{tb-cor}, one only needs
to follow the required interfaces to connect with input and evaluation modules, while the details of other parts can be ignored.
In specific, we utilize the interface function of \textsf{calculate\_loss}$(\cdot)$ for training and the
interface function of \textsf{predict}$(\cdot)$ for testing. To implement a model, what a user needs to do is to implement these important interface functions, without considering other details.
These  interface functions are indeed general to various recommendation algorithms, so that we can implement various algorithms in a highly unified way. Such a design mode enables  quick development of new algorithms.
Besides, our model module further encapsulates many important model implementation details, such as the learning strategy. For code reuse, we implement several  commonly used loss functions (e.g., BPR loss, margin-based loss, and regularization-based loss), neural components (e.g., MLP, multi-head attention, and graph neural network)  and initialization methods (e.g., Xavier's normal and uniform initialization) as individual components, which can be directly used when building complex models or algorithms.

\subsubsection{Implemented Models}
Until now, we have implemented 73\xspace recommendation models in the four categories of
general recommendation, sequential recommendation, context-aware recommendation and knowledge-based recommendation.
We refer the readers to {Table~\ref{models}} for more details on these models.
When selecting the models to be implemented, we have carefully surveyed the recent literature and selected the commonly used recommendation models and their associated variants (which may not receive high citations) in our library.
We mainly focus on the recently proposed neural methods, while also keep some classic traditional methods such as ItemKNN and FM.
In the future, more methods will also be incorporated in regular update.
For all the implemented models, we have tested their performance on two or four selected datasets, and invited a code reviewer to examine the correctness of the implementation.

\subsubsection{Rich Auxiliary Functions}
In order to better use the models in our library, we also implement a series of useful functions.
A particularly useful function is  automatic parameter tuning. The user is allowed to provide a parameter set for searching an optimal value leading to the best performance.
Given a set of parameter values, we can indicate four types of tuning methods, i.e.,
``\emph{Grid Search}'', ``\emph{Random Search}'', ``\emph{Tree of Parzen Estimators~(TPE)}'' and ``\emph{Adaptive TPE}''. The tuning procedure is implemented based on the library of \textsf{hyperopt}~\citep{hyperopt}.
Besides, we add the functions of model saving and loading to store and reuse the learned models, respectively.
Our library also supports the resume of model learning from a previously stored break point.
In the training process, one can print and monitor the change of the loss value and apply training tricks such as early-stopping. These tiny tricks largely improve the usage experiences with our library.

\subsection{Evaluation Module}

The function of evaluation module is to implement commonly used evaluation protocols for recommender systems. Since different models can be compared under the same evaluation module, our library is useful to standardize the evaluation  of recommender systems.

\subsubsection{Evaluation Metrics}
Our library supports both value-based and ranking-based evaluation metrics.
The value-based metrics (for rating prediction) include Root Mean Square Error~(RMSE) and Mean Average Error~(MAE), measuring the prediction difference between the true and predicted values.
The ranking-based metrics (for top-$K$ item recommendation) include the most widely used ranking-aware metrics, such as Recall$@K$, Precision$@K$, NDCG, and MRR, measuring the ranking performance of the generated recommendation lists by an algorithm.

\begin{figure*}[htbp]
    \begin{minipage}[b]{0.47\linewidth}
    \centering
    \subfigure[Full Ranking]{
        \includegraphics[width=1.25\linewidth]{eval_full.png}
        \label{fig-full}
    }
    \subfigure[Sample-based Ranking]{
        \includegraphics[width=1\linewidth]{eval_sample.png}
        \label{fig-sample}
    }
    \end{minipage}
    \medskip
    \begin{minipage}[b]{0.52\linewidth}
    \centering
    \subfigure[Indexing]{
        \includegraphics[width=0.5\linewidth]{eval_index.png}
        \label{fig-index}
    }
    \end{minipage}
    \vspace{-0.cm}
    \caption{Illustration of the proposed acceleration strategy for top-$K$ item evaluation. Here, $u_0,\cdots, u_5$ denote six users; black, blue and grey boxes denote training items, test items and other candidate items, respectively.}
\label{fig-acc}
\end{figure*}

\subsubsection{Evaluation Settings}

In recent years, there are more and more concerns on the appropriate evaluation of recommender systems~\citep{sampledmetrics,comparison}. Basically speaking, the divergence mainly lies in the ranking-based evaluation for top-$K$ item recommendation.
Note that the focus of our library is not to identify the most suitable evaluation protocols. Instead, we aim to provide most of the widely adopted evaluation protocols (even the most critical ones) in the literature. Our library provides a possibility to compare the performance of various models under different evaluation protocols.

For top-$K$ item recommendation, the implemented evaluation settings cover various settings of our earlier work in \citep{comparison}, where we have studied the influence of different  evaluation protocols on the performance comparison of models.
In particular, we mainly consider the combinations between item sorting (i.e., how to sort the items before data splitting) and data splitting (i.e., how to derive the train/validation/test sets) for constructing evaluation sets. We also consider both full ranking and sampling-based ranking, which is recently a controversial issue in the field of recommender system~\citep{sampledmetrics}. We summarize the supporting evaluation settings by our library in Table~\ref{tb-protocol}.

In order to facilitate  various evaluation settings, we encapsulate the related functions into four major parts, namely \emph{Group}, \emph{Split}, \emph{Order} and \emph{NegSample}.
With these  implementations, we can effectively support different evaluation protocols, which is also an appealing feature to use our library.

\subsubsection{Acceleration Strategy for Top-$K$ Evaluation}
Computing Top-$K$ evaluation metrics is usually time consuming.
The basic reason lies in that one need to exhaustively estimate the score for each user-item pair.
Since the method of score estimation varies across different models, it is not easy to optimize the entire evaluation procedure in a general way.
Therefore, we mainly focus on the step of selecting and generating top $K$ items given the ranking scores.

A problem is that different users have a varying number of ground-truth items in test set (resulting in different-sized user-by-item matrices), which is not suitable for parallel GPU computation in a unified manner.
Our approach is to consider all the items, including the ones in the training set (called \emph{training items}).
Given $n$ users and $m$ items for consideration, when performing full ranking, we can obtain a $n \times m$ matrix $\bm{D}$ consisting of the confidence scores from a model over the entire item set. When performing sample-based ranking, we create an $n \times m$ matrix $\bm{D}$, initializing all elements to negative infinity. Then, we fill matrix $\bm{D}$ with the confidence scores over sampled items. This step is called \emph{reshaping}.
When performing full ranking with all item candidates, we provide an option to mask the score of training items. If the user choose to mask, the matrix $\bm{D}$ obtained in the above step cannot be directly used for top-$K$ prediction. Our solution is to set the scores of training items to negative infinity, and perform the full ranking over the entire item set without removing training items. This step is called \emph{filling}.
In this way, all the users correspond to equal-sized evaluation matrices (i.e., $n \times m$) for subsequent computation in full ranking and sample-based ranking and the following steps are the same for both cases.

Then, we utilize the GPU-version \textsf{topk()} function provided by PyTorch to find the top $K$ items with the highest scores for users. The GPU-version \textsf{topk()} function has been specially optimized based on CUDA, which is very efficient in our case. This step is called \emph{topk-finding}.
With the \textsf{topk()} function, we can obtain a matrix $\bm{A}$ with size $n \times K$, which records the original index of the selected top $K$ items.
We further generate a binary matrix $\bm{B}$ of size $n \times m$ to indicate the existence of an item in the test set (blue boxes in Figure ~\ref{fig-full}) and Figure ~\ref{fig-sample}). Next, we use each row of matrix $\bm{A}$ to index the same row in matrix $\bm{B}$ and obtain a binary matrix $\bm{C}$ of size $n \times K$, which can be implemented efficiently through \textsf{gather()} function provided by PyTorch. We take the case of full ranking as an example in Figure ~\ref{fig-index}. This step is called \emph{indexing}.
Finally, we concatenate the matrix $\bm{C}$ of all the batches. The generated result consists of zeros and ones, which is particularly convenient for computing evaluation metrics. As will be shown next, such an acceleration strategy is able to improve the efficiency for both full ranking and sample-based ranking item recommendation.

\begin{figure}[t]
\centering
\setlength{\fboxrule}{0.pt}
\setlength{\fboxsep}{0.pt}
\fbox{
\includegraphics[width=1.\linewidth]{evaluator.png}
}
\caption{Architecture and dataflow of the evaluation model.}
\label{fig-evaluator-flow}
\end{figure}

\subsubsection{Efficiency and Scalability}
In this part, we empirically analyze the efficiency improvement yielded by our acceleration strategy and the scalability of the evaluation architecture.
Specifically,  the classic BPR model~\citep{BPRMF} is selected for efficiency analysis, since it is one of the most commonly used baselines for top-$K$ recommendation.
Besides, its model architecture is pretty simple without the influence of other factors, which is suitable for efficiency analysis.
 We compare its performance  \emph{with} and \emph{without} the acceleration strategy in our implementation.
We measure the model performance by the total time that (1) it generates a recommendation list of top ten items for users and (2) computes the metrics (NDCG@10 and Recall@10) over the recommendation list, on all the users.
To further analyze the model efficiency on  datasets of varying sizes, we use three Movielens datasets~\footnote{https://grouplens.org/datasets/movielens/} (i.e., Movielens-100k, Movielens-1M, and MovieLens-10M) to conduct the experiments. We split one original dataset into train, validation and test sets with a ratio of $8:1:1$. We only count the time for generating top ten recommendations (with full ranking) on the test set.
We average the time of ten runs of different implementations.
Our experiments are performed on a linux PC with CPU (Intel(R) Xeon(R) Silver 4216, 16 cores, 32 threads, 2.10GHz) and GPU (Nvidia RTX 3090 24G).
The results of efficiency comparison are shown in Table~\ref{t:timecost}.
From the result we can see that by applying the acceleration strategy, we can significantly speed up the evaluation process.
In particular, on the largest dataset MovieLens-10M, the accelerated model can perform the full ranking about two seconds, which indicates that   our implementation is rather efficient.
Currently, we only compare the entire  time with all the acceleration techniques. As future work, we will analyze the contribution of each specific technique in detail.
Apart from the superiority of efficiency, the evaluation of \emph{Recbole} is also flexible and extendable. As shown in Figure~\ref{fig-evaluator-flow}, the \emph{evaluator} is decoupled with model and data, and all the required resources for calculating metrics are well-wrapped by a \emph{collector}.
In this way, it is flexible to develop other customized metrics with these unified interfaces: implement new metrics and sign them in the \emph{register} (see Figure~\ref{fig-evaluator-flow}).

\renewcommand\arraystretch{1.2}
\begin{table}[t]
\caption{Time cost comparison (in second) on different-sized of Movielens datasets with the acceleration strategy or not. BPR$^{acc}$ denotes the model with acceleration strategy.}\label{t:timecost}
\centering
\begin{tabular}{c||c|c|c}
\hline
\multirow{2}{*}{Model} & \multicolumn{3}{c}{Dataset} \\ \cline{2-4}
& MovieLens-100k & MovieLens-1M & MovieLens-10M \\ \hline\hline
BPR & 0.245s & 2.478s & 29.900s \\ \hline
BPR$^{acc}$ & 0.009s & 0.090s & 2.210s \\ \hline
\end{tabular}
\end{table}

\section{Usage Examples of the Library}
In this section, we show how to use our library with code examples.
We detail the usage description in two parts, namely running existing models in our library and implementing  new models based on the interfaces provided in our library.

\begin{figure}[t]
\centering
\setlength{\fboxrule}{0.pt}
\setlength{\fboxsep}{0.pt}
\fbox{
\includegraphics[width=1.\linewidth]{example.png}
}
\caption{An illustrative usage flow of our library.
}
\vspace{-0.cm}
\label{usage}
\end{figure}

\subsection{Running Existing Models}
The contained models in our library can be run with either fixed parameters or auto-tuned parameters.

\subsubsection{Model Running with Fixed Parameters}
Figure~\ref{usage} presents a general procedure for running existing models in our library.
To begin with, one needs to download and format the raw public dataset based on our provided utils.
The running procedure relies on some experimental configuration, which can be obtained from the files, command line or parameter dictionaries.
The dataset and model are prepared according to the configured parameters and settings, and the execution module is responsible for training and evaluating the models.
The detailed steps are given as follows.

(\romannumeral1) Formatting the dataset.
A dataset is firstly selected by a user, and then it is formatted based on the scripts, which can generate required atomic files for different datasets.
This procedure takes the following code: $\textsf{\text{atomic\_file}=\text{PreProcess}(\text{dataset})}$.

(\romannumeral2) Generating the configuration.
In our library, the experiment configurations can be generated in different ways.
One can write a configuration file, and then read this file in the main function as in line (1) of Figure~\ref{usage}.
Another way for configuration is to include parameters in the command line, which is useful for specially focused parameters.
At last, one can also directly write the parameter dictionaries in the code.

(\romannumeral3) Filtering and splitting the dataset. We provide rich
auxiliary functions to filter and split the datasets.
For example, one can filter the dataset by keeping the users/items with at least $K$ interactions, removing the data occurred in some fixed time period.
Different filtering methods can be applied in a unified function (line (2) of Figure~\ref{usage}).
When splitting the dataset, one can indicate ratio-based method or leave-one-out method. Then, one can use line (3) of Figure~\ref{usage} to generate the training, validation and testing sets.

(\romannumeral4)  Loading the model.
The next step is to build a recommender model.
Given a target model in mind, the user can obtain a model instance according to line (4) of Figure~\ref{usage}.

(\romannumeral5) Training and evaluation.
Once the dataset and model are prepared, the user can, at last, train and evaluate the model based on line (5) of Figure~\ref{usage}.

\renewcommand\arraystretch{1.2}
\begin{table*}[t]
\caption{Comparison with existing recommender system libraries.}
\vspace{-0.cm}
\centering
\setlength{\tabcolsep}{3.pt}
\begin{threeparttable}
\begin{tabular}
{p{4cm}<{\centering}||p{1.6cm}<{\centering}p{1.2cm}<{\centering}p{1.2cm}<{\centering}p{1.1cm}<{\centering}p{1.1cm}<{\centering}p{1.1cm}<{\centering}p{1.6cm}<{\centering}p{1.cm}<{\centering}p{1.4cm}<{\centering}}\hline
Library&Language&\#Models&\#Datasets&\#Fork&\#Star&\#Issues&Release time&Neural&PT\\\hline\hline
MyMediaLite~(\cite{MyMediaLite})&C\#&61&5&199&477&451&2010&No&manual\\\hline
LibFM~(\cite{LibFM})&C++&1&-&415&1400&32&2014&No&manual\\\hline
LibRec~(\cite{Librec})&Java&93&11&1000+&3009&252&2014&No&manual\\\hline
RankSys~(\cite{RankSys})&Java&8&-&58&259&38&2016&No&manual\\\hline
Crab~(\cite{Crab})&Python&2&4&381&1122&75&2011&No&manual\\\hline
Surprise~(\cite{Surprise})&Python&11&3&888&4989&333&2015&No&manual\\\hline
LightFM~(\cite{LightFM})&Python&1&2&610&3741&425&2015&No&manual\\\hline
Case Recommender~(\cite{CaseRecommender})&Python&27&-&75&354&24&2015&No&manual\\\hline
Recommenders~(\cite{Recommenders})&Tensorflow&31&5&1900+&10000+&602&2018&Yes&automatic\\\hline
Cornac~(\cite{Cornac})&Tensorflow&42&14&75&397&58&2018&Yes&automatic\\\hline
NeuRec~(\cite{NeuRec})&Tensorflow&33&3&199&816&29&2019&Yes&manual\\\hline
Elliot~(\cite{Elliot})&Tensorflow&50&-&19&108&8&2021&Yes&automatic\\\hline
Spotlight~(\cite{Spotlight})&PyTorch&8&5&389&2552&109&2017&Yes&automatic\\\hline
DaisyRec~(\cite{daisyRec})&PyTorch&20&14&59&354&8&2019&Yes&automatic\\\hline
ReChorus~(\cite{ReChorus})&PyTorch&12&2&47&214&16&2020&Yes&manual\\\hline
Beta-recsys~(\cite{Beta-recsys})&PyTorch&22&21&25&75&120&2020&Yes&manual\\\hline\hline
\textbf{RecBole\xspace}&\textbf{PyTorch}&\textbf{73\xspace}&\textbf{28\xspace}&\textbf{193}&\textbf{1179}&\textbf{163}&\textbf{2020}&\textbf{Yes}&\textbf{automatic}\\\hline
\end{tabular}
\begin{tablenotes}
   \footnotesize
   \item[1] Neural means the libarary support deep recommender models, and PT denotes parameter tuning.  The statistics were collected on the date of Aug 21, 2021.
\end{tablenotes}
\end{threeparttable}
\vspace{-0.cm}
\label{tab:compare}
\end{table*}

\subsubsection{Parameter Tuning}
Our library is featured in the capability of automatic parameter (or hyper-parameter) tuning.
One can readily optimize a given model according to the provided hyper-parameter range.
The general steps are given as follows.

(\romannumeral1) Setting the parameter range.
The users are allowed to provide candidate parameter values in the file ``hyper.test''.
In this file, each line is formatted as: $\textsf{\text{parameter}=[\text{value $1$},\text{value $2$},...\text{value $n$}]}$.
Instead of a fixed value, the users can empirically indicate a value set, which will be explored in the following tuning steps.

(\romannumeral2) Setting the tuning method.
Our parameter tuning function is implemented based on the library \textsf{hyperopt}.
Given a set of parameter values, we can indicate four types of tuning methods, i.e.,
``\emph{Grid Search}'', ``\emph{Random Search}'', ``\emph{Tree of Parzen Estimators~(TPE)}'' and ``\emph{Adaptive TPE}''.
The tuning method is invoked by the following code: $\textsf{\text{hy} = \text{HyperTuning}(\text{objective},\ \text{tuning\_method},\ \text{range\_file})}$,
where the parameter range file is used to indicate parameter values.

(\romannumeral3) Starting the tuning process.
The user can start the running process by the following code: $\textsf{\text{hy.run()}}$.
With the tuning ranges and method, our library will run the model iteratively, and finally output and save the optimal parameters and the corresponding model performance.

\subsection{Implementing a New Model}
Based on RecBole, it is convenient to implement a new model by instantiating three functions as follows:

(\romannumeral1) Implementing the ``\textsf{\_\_init\_\_()}'' function.
In this function, the user performs parameter initialization, global variable definition and so on.
The new model should be a sub-class of the abstract model class provided in our library.
Until now, we have implemented the abstract classes for  general recommendation, knowledge-based recommendation, sequential recommendation and context-aware recommendation.

(\romannumeral2) Implementing the ``\textsf{calculate\_loss()}'' function.
This function calculates the loss to be  optimized by the new model.
Based on the return value of this function, the library will automatically invoke different optimization methods to learn the model according to the pre-set configurations.

(\romannumeral3) Implementing the ``\textsf{predict()}'' function.
This function is used to predict a score from the input data (e.g., the rating given a user-item pair).
This function can be used to compute the loss or derive the item ranking during the model testing phase.

\section{Comparison with Existing Libraries}
In recent years, a considerable number of open source recommender system libraries have been released for research purpose. We summarize and compare the characteristics of existing recommender system libraries in Table~\ref{tab:compare}, from which, we can see: the programming language of these libraries gradually evolves from C/C++/JAVA to Python/Tensorflow/PyTorch. From the model perspective, recent libraries mostly support neural recommender models, which agrees with the advance trend in the recommendation domain. In our framework, we select PyTorch as the basic deep learning framework  for development, due to its friendly features like easy debugging, compatible with numpy and etc.

RecBole provides the most comprehensive models and benchmark datasets among existing libraries, which can better free the users from the heavy model re-programming work.
In addition to reproduce the existing models, we aim to ease the developing process of new algorithms. We design general and extensible underlying data structures to support the unified development framework. By providing a series of useful tools, functions and scripts (e.g., automatic parameter tuning), our library is particularly convenient to be used for scientific research.

At last, we believe that implementation is only the first step for open source recommendation library, as more efforts are needed to maintain and update the library according to users' feedbacks and suggestions.
Our team is working hard to respond to the GitHub issues and fix possible bugs (134 issues were solved until Aug 21, 2021).
After release, our library has received much attention from the users. To the date of publication, it is ranked at the third and fourth places based on the number of received stars for the topic of ``recommender system'' and ``recommendation system'', respectively.

\ignore{At last, we believe implementing the library is only the first step towards providing the recommendation researchers with a better experiment environment.
Much more efforts are needed to maintain this library, and receive suggestions from the users.
To this end, we keep on fixing bugs and solving issues since the first release of our library.
Until now, we have solved nearly 130 issues, and obtained 1049 stars and 160 forks in only 8 months.
Because of our effort in the past few months, our library is ranked at 5 on Github under the ``recommendation'' and `recommendation system'' queries, respectively, which is a positive response on our hard work and confirms the popularity and acceptability of our library.
In the future, we will actively improve our library to continually serve the recommendation community, and we believe this is not hard for us to promise, since the project leaders are all faculties in the universities, who can easily stick to their research interest in the recommendation domain without too much disturbation.
}

\section{Conclusion}
In this paper, we have released a new recommender system library called RecBole\xspace.
So far, we have implemented 73\xspace recommendation algorithms on 28\xspace commonly used datasets.
We design general and extensible data structures to offer a unified development framework for new recommendation algorithms.
We also support extensive and standard evaluation protocols to compare and test different recommendation algorithms.
Besides, our library is implemented in a GPU-accelerated way, involving a series of optimization techniques for achieving efficient execution.
The RecBole\xspace library is expected to improve the  reproducibility of recommendation models, ease the developing process of new algorithms,  and set up a benchmark framework for the field of recommender system.
In the future, we will make continuous efforts to  add more  datasets and models.
We will also consider adding more utils for facilitating the usage of our library, such as result visualization and algorithm debugging.

\section{Acknowledgment}
This work was partially supported by the National Natural Science Foundation of China under Grant No. 61872369, 61802029 and 61972155, Beijing Academy of Artificial Intelligence (BAAI) under Grant No. BAAI2020ZJ0301, and Beijing Outstanding Young Scientist Program under Grant No. BJJWZYJH012019100020098.
We also sincerely thank non-author team members (Chen Yang, Zheng Gong, Chenzhan Shang, Zihan Song, Ze Zhang, Jingsen Zhang, Lanling Xu, Zhen Tian and Wenjing Yue) for testing our library.

\newpage
\bibliographystyle{ACM-Reference-Format}
\balance


\section{Introduction}
\label{sec:intro}
Speaker verification refers to the task of verifying a user identity based on their voiceprint.
This technology has received increasing attention in recent years, partially due to its application to voice assistants. Speaker verification enables the contextualisation of spoken queries and tailored assistant responses to personalised content (\textit{e.g.} ``add an event to \textit{my} calendar``).

The aggregation of the variable-length sequential audio input into a fixed length embedding plays a crucial role for any practical application. In the first approaches to time aggregation, the embeddings are computed over fixed segments of audio~\cite{ivector} with additional steps required to aggregate the representations. More recently, with the advent of neural networks, end-to-end solutions have been proposed that directly handle variable-length input~\cite{heigold_end--end_2015}. Using properly designed layers, the temporal statistics can be accumulated internally in the network~\cite{heigold_end--end_2015,sp,asp}.
While reaching good accuracy, these end-to-end, neural-based methods have typical computational footprints that require offline or server-side execution.
Although some speaker verification engines with low execution latency~\cite{ge2e}  or the ability to run on mobile devices~\cite{angular_prototypical}  have been proposed,
they remain too large for embedded applications where memory and computing power are further limited.

In this work, we propose a speaker verification system specifically tailored to embedded use cases.
We budget CPU and memory resources to match that of typical keyword spotting systems~\cite{snips_wakeword} designed to run continuously and in real time on device.
Our approach allows to decouple streamed time-series features extraction from aggregation, providing an optimal balance between representation quality and inference latency.
The features extraction stage is based on the QuartzNet~\cite{quartznet} model -- never used in the context of speaker verification to our knowledge -- and the aggregation stage on Ghost Vector of Locally Aggregated Descriptors (GVLAD)~\cite{gvlad}, with key modifications, \textit{e.g.} the inclusion of Max  Features Map~\cite{mfm} operations and a more computationally efficient method for descriptor aggregation.

We demonstrate that our approach reaches performances comparable with the state of the art (3.31\% EER on the VoxCeleb1 verification test set) with a number of learning parameters orders of magnitude smaller, making it fit for embedded applications. Voice being a highly sensitive biometric identifier, our lightweight approach grants speaker verification abilities to small devices typical of IoT systems, while fully respecting the privacy of users.

The paper is structured as follows.  In Section ~\ref{sec:spkidmodel}, we detail our system in terms of neural network architectures and training method.
In Section~\ref{sec:experiments}, we describe our experimental settings including datasets and hyper-parameters selection, and computational performances. In Section~\ref{sec:results}, we compare our solution with other SOTA approaches. We give our conclusions in Section~\ref{sec:conclusion}.

\begin{figure*}[ht!]
    \centering
    \includegraphics[height=8.9cm]{horizontal_diagram_seqwise_embedwise_237-1.png}
    \caption{Network architecture. Left panel: sequence-wise network (blue). $B$ is the number of stacked blocks. In each block, $R$ is the number of TCSConv1d+BN modules followed by Max Feature Map activation stacked before a last TCSConv1d+BN and residual connection. Right panel: embedding-wise network (yellow). $T$ is the time dimension and $C$ is the channel dimension. In the GVLAD module, $O=K+G$ is the total number of clusters, defined as the sum of the number of contributing clusters $K$ and the number of Ghost clusters $G$.}
    \label{fig:network_diagram}
\end{figure*}

\section{Speaker Embedding Model}
\label{sec:spkidmodel}

We propose a fixed-size speaker embedding model performed in two stages. In the first stage, a streaming neural network, inspired by the QuartzNet~\cite{quartznet} architecture, takes as input an arbitrarily long time series of acoustic features and outputs another series of higher-level features. This network is referred to as sequence-wise network. The second stage consists of an aggregator neural network, called embedding-wise network, built upon the GVLAD~\cite{quartznet} architecture, that aggregates the outputs of the streaming stage along the time dimension to build a fixed-size embedding of the audio signal.

\subsection{Model Architecture}
\label{subsec:archi}

The sequence-wise network of our solution is pictured in blue on the leftmost part of Figure~\ref{fig:network_diagram}.
One basic component, taken from the QuartzNet~\cite{quartznet} model, is a Time Channel Separable 1-dimensional Convolution (TCSConv1d) module. It is composed of a 1-dimensional depthwise convolution, where each kernel operates only across the time axis, and a pointwise convolution acting on all filters but independently on each frame.
The first TCSConv1d module is followed by Batch Normalisation (BN) and Parametric ReLU (PReLU)~\cite{prelu}. The next layer performs max-pooling to halve subsequent computations.
Successive blocks are also composed of TCSConv1d followed by BN and a Max Features Map (MFM) operation~\cite{mfm} that, at each location, optimally selects the output of distinct filters. As shown in Figure~\ref{fig:network_diagram}, the TCSConv1d+BN followed by MFM constitutes a block that is repeated $R$ times with a residual connection and followed by a PReLU activation.

Time aggregation techniques are crucial for creating efficient embeddings. Historically, aggregation was performed using a simple mean pooling mechanism~\cite{tap}, later refined by statistical pooling~\cite{sp} and attention mechanisms~\cite{asp, sap}, or using that last output of the recurrent cells~\cite{heigold_end--end_2015}. Statistical pooling being too demanding for our constrained budget, we compared Self Attentive Pooling (SAP), recurrent cells, and GLVAD and got better results with the latter.
To further limit the computations, we replace the last linear projection in GVLAD by a simple cluster-wise projection averaged on the cluster dimension $K$. This comes with almost no performance drop, while making the cost of adding a new cluster linear instead of quadratic in the number of learning weights in the projection layer.
The rightmost part of Figure~\ref{fig:network_diagram} (in yellow) displays a detailed view of the embedding-wise network. $O=K+G$ is the total number of clusters and is defined as the sum of the numbers of clusters $K$ contributing to the embedding and the number of "Ghost" clusters $G$, named that way because they are not included in the final concatenation (see~\cite{gvlad} for more details).

\subsection{Training loss function}
\label{subsec:loss}

Speaker recognition can be seen as a metric learning or a classification problem; both approaches have been shown to be successful.
We explored triplet loss techniques for metric learning (like soft triplet loss based on softmax~\cite{triplet_soft}), and cross-entropy based methods for classification, like
angular prototypical~\cite{angular_prototypical} or ArcFace~\cite{arcface} losses.
We found that ArcFace was the most efficient, especially when coupled with focal loss~\cite{focal_loss}.

\section{Experiments}
\label{sec:experiments}

\subsection{Datasets}
\label{subsec: dataset}

The proposed approach is evaluated on two datasets. We first train our network on the VoxCeleb2~\cite{voxceleb2} dev split only. This dataset contains $5994$ speakers of $145$ different nationalities, with over $1,092,009$ utterances.
We compared this model with previously published papers in Section~\ref{subsec:results_clean} using the VoxCeleb1 verification test, following the protocol proposed by~\cite{voxceleb2}.

We also train an additional model using the same architecture, but augmenting the training data with the MUSAN~\cite{musan} dataset, a corpus of music, speech, and noise.
Reverberation effects are obtained by applying rooms simulation from Pyroomacoustics~\cite{Scheibler2017Oct}. We simulate over $5,000$ rooms and apply the augmentation randomly eight times for each original audio sample with variation of gain, SNR, noise type, and noise source location. We also include a small amount of speed augmentation as a approximate means of accounting for within-speaker speech tempo variability.
The model trained with augmented data is analysed
in challenging acoustic conditions in Section~\ref{subsec:results_noisy} on the Voices Obscured in Complex Environmental Settings (VOiCES) corpus~\cite{richey2018voices}. The corpus addresses challenging noisy and far-field acoustic environments known to strongly impact the final performance of speaker verification systems.

\subsection{Experimental setup}

\subsubsection{Model implementation}
The acoustic features are $64$-dimensional Mel filterbank energies, extracted from the input audio every $10$ms over a window of 20ms. Mean and variance normalisation (MVN) is applied but no Voice Activity Detection (VAD) pre-processing is done. We experimented with several combinations of depth and width of the architecture (while keeping the total number of parameters fixed) and converged to a configuration with a total of $22$ TCSConv1d each with $96$ filters ($B=5$ and $R=3$ in Figure~\ref{fig:network_diagram}). All the kernels have a constant size of $15$, tuned to reach a good balance between performance, computational load and final size of the receptive field.

Compared with the original GVLAD work~\cite{gvlad}, we select a higher number of clusters $K$ at the aggregation stage (see Section~\ref{subsec:archi}) as we find it yields better results. The number of Ghost clusters $G$ seems to have little impact as soon as there are at least 3. We therefore set $K=32$ and $G=3$ for the following experiments. The embedding network has $237,499$ learning parameters and an output dimension of $96$. Despite our constrained budget, we do not apply compression techniques such as quantization, teacher-student, or pruning. We of course expect these refinements to further improve our model, but we choose to focus solely on optimizing the architecture in this paper.

\subsubsection{Training details}

Each batch contains $S$ speakers, each with $N$ utterances of duration $D$ in seconds. Best results are obtained with: $S=75$, $N=5$, and $D=2\mathrm{to}5$. $2\mathrm{to}5$ means a random uniform sampling of each sample from 2 to 5 seconds.
The same definition of training epoch as in~\cite{angular_prototypical} is used: every speaker in the dataset is seen once. Given the memory constraints, we train with mixed precision and auto scaling loss which we find useful to stabilise the training and avoid exploding gradients.
We use the Adam optimizer with a learning rate of $0.001$ and a weight decay of $0.0005$. A scheduler is applied to reduce the learning rate on loss plateau while monitoring the EER on the VoxCeleb2 test set. The scheduler patience is set to $250$ epochs.
The scaling and margin parameters of the ArcFace~\cite{arcface} loss function are respectively set to $s=15$ and $m=0.5$.
Masking on the time-dimension is applied on the sequence-wise part and during the first stages of the GVLAD process.

\subsection{Inference \& Streaming}

\begin{table}
    \centering
     \begin{tabular}{c| c c c c}
     \toprule
     Network Part & FMA* & Div* & FLOPS & params \\ [0.5ex]
     \midrule

     Sequence wise & $10850.4$ & $0$ & $10.8$M & $211.6$K  \\

     Embedding wise & $659.7$ & $8.0$ & $0.7$M &  $25.8$K \\
     \midrule
     TOTAL & $11509.4$ & $8.0$ & $11.5$M &  $237.5$K \\
     \bottomrule
    \end{tabular}
    \caption{Computational Inference Cost (* K over 1s).
    \textit{FMA}: fused multiply–add operation.
    \textit{Div}: div operation.
    \textit{FLOPS}: floating point operations per second.
    \textit{params}: number of learning parameters.
    }
    \label{fig:computation_cost_table}
\end{table}

Separating the model in two distinct stages allows to set a decoupled memory and latency budget. Thanks to \texttt{tract}\footnote{available at \url{ https://github.com/sonos/tract} version used: \texttt{0.11.1}}, an open source neural network inference library, we are able to run the first stage in streaming mode and drastically reduce the delay after end-pointing. Some numbers for each stage of the proposed approach are displayed in Table~\ref{fig:computation_cost_table}.

Latency is a cornerstone to real-time applications of speaker verification systems. It depends on the audio duration to embed and the computational power given a fixed model. Both dimensions are displayed in Table~\ref{fig:latency}. These results show that the system can run on a single core of Raspberry Pi3 B. This latency could be further improved by streaming part of the embedding-wise network
(dashed orange box in the rightmost part of Figure~\ref{fig:network_diagram}). This will be the object of a future work.

\begin{table}
    \centering
     \begin{tabular}[width=\textwidth]{c| c c c c c}
     \toprule
      CPU \& inference & 1s & 5s & 10s \\ [0.5ex]
     \midrule
     Intel i7-8750H - Batch & 14.8 &  33.7 & 58.40 \\

     Intel i7-8750H - Stream & 1.11 & 5.09 & 9.97 \\
     \midrule
     Raspberry Pi 3B - Batch & 198.8 & 438.2 & 733.8 \\
     Raspberry Pi 3B - Stream & 45.1 & 184.4 & 221.5 \\
     \bottomrule
    \end{tabular}
    \caption{Mean embedding latency in milliseconds on a mono-core CPU (Raspberry Pi running on 64-bit Ubuntu) for various audio lengths,
    \texttt{Batch} is the latency if we apply the whole embedding process once end-pointing is triggered,
    \texttt{Stream} is the latency when the first stage is performed in streaming with a Real Time Factor (RTF) lower than 1.
    }
    \label{fig:latency}
\end{table}

\section{Results}
\label{sec:results}

The proposed approach is evaluated by computing a cosine similarity score between embeddings on two datasets.

\subsection{VoxCeleb1 verification test}
\label{subsec:results_clean}

The proposed model (trained on the VoxCeleb2 dev split) is compared to other existing works in terms of EER and number of parameters, following the VoxCeleb1 verification test protocol proposed in~\cite{voxceleb2}. Figure~\ref{fig:research_comparison} shows that despite its constrained budget, our model is almost on par with the original GVLAD approach~\cite{gvlad} with 32 times less number of parameters. Other solutions have lower EER, but require computational capabilities incompatible with embedded systems.
Contrary to the number of parameters, the computation cost is more rarely reported in the literature, but critical to real life applications. The second smallest model displayed on Figure~\ref{fig:research_comparison}, denoted \texttt{2020-03 Chung}, has only $1.437$M learning parameters ($6$ times more than the proposed approach) but actually requires $353.3$ MFLOPS, or $30$ times more operations at processing time than our solution\footnote{To compute these numbers, we built the ONNX model from the source code available at \url{https://github.com/clovaai/voxceleb_trainer}}.

\begin{figure}[ht!]
    \centering
    \includegraphics[scale=0.65]{voxceleb_verif_1_research_comparison.jpeg}
    \caption{
        EER\% comparison on the VoxCeleb1 verification test
        with several previous approaches:
        \texttt{2018-06 Chung}~\cite{voxceleb2}, \texttt{2019-02 Xie}~\cite{gvlad}, \texttt{2020-03 Chung}~\cite{angular_prototypical},
        \texttt{2019-10 Yadav}~\cite{Yadav2019Oct}, \texttt{2019-10 Zeinali}~\cite{Zeinali2019Oct}
        , \texttt{2019-10 Zeinali*}~\cite{Zeinali2019Oct} (models fusion). EER for \texttt{2020-03 Chung} has been recomputed by building an embedding model from the provided source code, adding a cosine distance scorer and running the benchmark, all others EERs are reported from the corresponding papers.
    }
    \label{fig:research_comparison}
\end{figure}

\subsection{VOiCES 2019 challenge}
\label{subsec:results_noisy}

We compare the proposed approach with existing works following the \textit{fixed} training conditions of the VOiCES benchmark (see \cite{Nandwana2019Feb} for more details). The results reported in Table~\ref{table:voices_performance_table} for our approach refer to a  model trained on VoxCeleb2 augmented as detailed in Section~\ref{subsec: dataset}. Even in challenging acoustic conditions, the increase in EER compared to other approaches remains limited ($+2.57$ percentage points from the best performing approach) while the number of parameters is drastically reduced ($26$ times smaller). It should be noted that the best scoring systems in Table~\ref{table:voices_performance_table} employ additional VAD and scoring mechanisms that are expected to significantly improve accuracy especially in challenging conditions.

\begin{table}
    \centering
    \begin{tabular}[width=\textwidth]{c| c c }
    \toprule
    Model &  EER\% & params (K) \\
    \midrule
    STC-Innovations Ltd. \cite{Novoselov2019Apr} & 5.04 & 5953.5\\
    BUT from Brno University \cite{Zeinali2019Jul} & \textbf{4.90} & 6083.0 \\

    Ours & 7.47 & \textbf{237.5}\\
    \bottomrule
    \end{tabular}
    \caption{Comparison in the \textit{fixed} training conditions~\cite{Nandwana2019Feb} of EER\% and learning parameters to the best reported single models on the VOiCES from a Distance 2019 Challenge.}
    \label{table:voices_performance_table}
\end{table}

\section{Conclusion}
\label{sec:conclusion}
We propose an efficient model architecture for speaker verification suited for embedded systems. Our results demonstrate that our solution yields a limited increase of EER on well established benchmarks, while drastically reducing the number of parameters and operations. Rarely reported in the literature, the inference properties of the model have been studied, highlighting a good level of responsiveness. Future work will be centered on further improving the accuracy in challenging acoustic environments, first by integrating the proposed solution with a low-resource VAD and advanced post-embedding scoring techniques as in~\cite{Zeinali2019Jul}.

\bibliographystyle{IEEEbib}