\section{Conclusion}
\label{sec:conclusion}

This paper reported on 6D pose estimation from a single RGB image by introducing novel primitive learning associated with each object. This paper presents a substantial improvement in the pose estimation even in the occluded case and comparable performance even using only synthetic images for training. The proposed method was validated using three public benchmark datasets yielding \ac{SOTA} performance for occluded and small objects.

\section{Experiment}
\label{sec:experiment}

In this section, we evaluate the proposed method using three different datasets.

\input{tab_linemod_occ.tex}

\subsection{Dataset}

In total, three datasets were used for the evaluation: LINEMOD \cite{hinterstoisser2012model}, Occlusion LINEMOD \cite{hinterstoisser2012model}, and YCB-Video \cite{xiang2017posecnn}. The LINEMOD and Occlusion LINEMOD datasets are widely exploited benchmarks for 6D object pose estimation and cover various challenges, including occlusion and texture-less. The LINEMOD dataset comprises 13 objects with 1,312 rendered images on each for the training and about 1,200 images per object for the test. The Occlusion-LINEMOD dataset shares the training images with the LINEMOD dataset. For the test, they provide pose information for eight occluded objects. A recently released dataset, the YCB-Video dataset is composed of 21 high-quality 3D models, offering 92 annotated video sequences per frame. These video sequences include various lighting conditions, noise in the capture, and occlusion.

\subsection{Training Detail}

We sampled 50,000 object poses and rendered 50,000 training images per object together with their associated primitives. The rendered object images were further augmented using the method proposed by \cite{mjeon-2019-cams} using domain randomization. We train our network mainly from synthetic images and additionally train with 180 real images for three types of dataset.

All of the networks were trained with the Adam optimizer on Titan V. The ResNet backbone was initialized with pre-trained weights provided in PyTorch. All networks were trained with a batch size of 50 and a 0.0001 learning rate for 40 epochs.

\subsection{Evaluation Metric}

\textbf{(i) 2D Projection error metric} To evaluate the pose estimation in terms of the 2D projection error we also use the same metric as in \cite{pvnet-2019-cvpr} and measure the mean pixel distance between projection of the 3D model and the image pixel points. We also count the estimation is correct if the error is less than 5 pixels.

\textbf{(ii) 3D Projection error metric} Similar to \cite{xiang2017posecnn}, two metrics were selected for evaluation. We chose the average distance metric (ADD)~\cite{hinterstoisser2012model} as the evaluation metric, which computes the distance of transformed 3D model points between the ground truth pose and predicted pose using ADD(-S) metric.
When the average distance value is less than 10\% of the 3D model's diameter, the predicted 6D pose is considered to be correct. This ADD metric, however, is not suitable to use with symmetrical objects. For symmetrical objects, the average distance is computed between the closest points.

\textbf{(iii) Pose estimation metric} For the original LINEMOD dataset, we directly measure the \ac{MAE} for rotation and translation together with other metrics.

\input{tab_ycb.tex}
\vspace{-3mm}

\subsection{6D Object Pose Estimation Evaluation}

We evaluate the performance of our model on the three datasets. The proposed method is compared against other \ac{SOTA} methods from holistic, keypoint-based and part-based approaches. We evaluate our method when training only with synthetic images (PrimA6D-S) and when training using 180 additional real images (PrimA6D-SR). Other existing methods were trained using additional real images. Qualitative results for all three datasets are presented in \figref{fig:linemod_result_fig} and \figref{fig:linemod_occ_result_fig}. Please also refer to \texttt{prima6d.mp4}. We excluded the comparison to the case without the primitive decoder due to drastically large error when no primitive decoder was used.

\subsubsection{LINEMOD Dataset Results}

\tabref{tab:linemod_add} and \tabref{tab:linemod_2dp} exhibit the evaluation for all 13 objects in the LINEMOD dataset. Among the keypoint or the part-based approaches, PVNet is the most recently released method with \ac{SOTA} performance and thus used as the main baseline method for comparison.

Over the LINEMOD dataset, PrimA6D-SR outperformed PVNet in terms of the ADD(-S) score and presented a comparable result in terms of the 2D projection metric. The different performance can be understood from the \ac{MAE} metric in \tabref{tab:linemod_2dp}. If the object center point in the image is found properly, the 2D projection metric is affected more by the rotational estimation accuracy and ADD(-S) is more critical to the translation inference performance. As can be seen, PVNet exhibited higher performance than PrimA6D-S in terms of the rotation \ac{MAE} metric, which results in a performance difference on the 2D projection metric. On the contrary, PrimA6D-S outperformed PVNet in terms of the translation \ac{MAE} metric, which results in a performance difference on the ADD(-S) score.

In addition, the PrimA6D(-S/-SR) presents the strength for small and texture-less objects that the drawback in the PVNet such as ape and cat. Especially, the PrimA6D(-S/-SR) has the forte on the translation estimation.

\subsubsection{Occlusion LINEMOD Dataset Results}

Tables in \tabref{tab:occ_add} list the performance on the Occlusion LINEMOD dataset. Here, we only evaluate the performance of models that do not have the refinement step. As can be seen in two tables, the PrimA6D(-S/-SR) outperformed PVNet in terms of the ADD(-S) score and presented a comparable result in terms of the 2D projection metric. As the PrimA6D(-S/-SR) reconstruct the corresponding object and primitive, we can affirm notable improvement on the occluded case. Moreover, the PrimA6D(-S/-SR) demonstrates striking accuracy for objects that are difficult to recognize by other methods due to small size such as cat and holepuncher.

\subsubsection{YCB-Video Dataset Result}

We further evaluate the pose estimation performance over the YCB dataset as shown in \tabref{tab:ycb}. The PrimA6D(-S/-SR) achieved enhanced inference capability even when trained solely from synthetic dataset (PrimA6D-S). The proposed method is capable of inferring 6D pose of objects that are difficult to recognize by other methods such as wood block.

Another notable point in YCB dataset is the overall down performance from LINEMOD; all approaches including PVNET and ours shows lower accuracy in YCB than LINEMOD. This is because of the challenging and realistic dataset nature of YCB. Despite the challenging dataset, the proposed method shows the smallest performance drop from LINEMOD both in terms of ADD(-S) and 2D projection error even when solely trained from synthetic data. This reveals the generalization capability of the proposed method while others lacking.

\subsection{Ablation Studies}

We further evaluate the proposed method by conducting ablation studies in terms of the effect of real images, effect of training with \ac{GAN} and bounding box accuracy.

\textbf{1) Effect of real image} The proposed method was designed to exploit synthetic data from CAD model. However, considering the potential discrepancy between CAD and real data, additional weight refinement using real training image shows very meaningful performance improvement. We train the network only from synthetic data (PrimA6D-S) and compare the performance improvement by adding real images in the training set (PrimA6D-SR). This comparison is provided for three datasets. Obviously we witness accuracy improvement in all metrics by adding real images in the training set.

\begin{figure}[!t]
	\centering
	\includegraphics[width=0.95\columnwidth]{figs/fig_5.pdf}

	\caption{\bl{Effect of GAN on occluded and symmetrical object.}}

	\label{fig:effect_gan}
	\vspace{-4mm}
\end{figure}

\textbf{2) Training with GAN} The adversarial loss and training strategy using \ac{GAN} provides a detailed refinement of the object and primitive reconstruction. As can be seen in \figref{fig:effect_gan}, the reconstructed result shows a blurry and obscure reconstruction when without using \ac{GAN}. This indicates the potential ambiguity in the rotation and translation estimation. Additional training with \ac{GAN} enables the \ac{VAE} to build discriminability. \bl{Also, regarding symmetrical objects with ambiguity for one axis, incorporating adversarial training can solve the potential axis ambiguity in the reconstructed primitive and completely form three orthogonal axes. In the experiment, we confirmed that the ambiguous axis in the primitive becomes the clear orthogonal axis for the other two axes according to the qualitative result.} Testing over Occlusion LINEMOD in \tabref{tab:occ_add} reveals this substantial improvement from the adversarial loss.

\input{tab_bb_ablation.tex}

\textbf{3) Bounding box accuracy} The input of the network is a cropped image from an object detection module hence the PrimA6D estimation performance may be dependent on the detection accuracy. We have examined the effect of the detected bounding box accuracy by reducing the \ac{IoU} to the original bounding box in terms of  ADD(-S) as this is more sensitive to the translational estimation accuracy. As can be seen in \tabref{tab:bb_ablation}, the performance drops as we reduce the \ac{IoU} from 1.0 to 0.75. As can be seen from \tabref{tab:bb_ablation}, \tabref{tab:linemod_add} and \tabref{tab:linemod_2dp}, the PrimA6D-SR still outperforms all other existing methods despite the performance degradation. More notably, the PrimA6D-S shows the exceptional result on the Occlusion LINEMOD dataset even with decreased IoU, being solely trained from synthetic data.

\subsection{Inference Time}

Since our method is applied after object detection, the total inference time is also affected by the running time of the object detection model. For example, we used M2Det \cite{DBLP:journals/corr/abs-1811-04533} that runs at 33.4 \ac{fps}. For the 6D object pose estimation of a single object in an image, the total process takes 31ms on average using a GTX 1080Ti GPU. The entire 6D pose estimation supports 16 to 17 \ac{fps}.

\subsection{Failure Cases}

\figref{fig:fail} shows failure cases when one axis is entirely obscured by the other two axes of the rotation primitive. This hidden primitive axis prevents the algorithm from extracting the correct keypoint yielding a larger rotation error. In future work, we aim to include estimated depth from the primitive to resolve this issue.

\begin{figure}[!t]
	\centering
	\includegraphics[width=0.85\columnwidth]{figs/fig_6.jpg}

	\caption{The examples of failure case.}

	\label{fig:fail}
	\vspace{-5mm}
\end{figure}

\begin{figure*}[!t]
	\centering
	\includegraphics[trim=0 60 0 0, clip, width=0.95\textwidth]{figs/fig_7.png}
    \caption{Qualitative pose estimation evaluation on the LINEMOD dataset. The red box indicates the ground truth pose while blue is the estimated pose using PrimA6D. Next to the inferred pose is reconstructed object and the predicted primitive associated to each object.}

	\label{fig:linemod_result_fig}
	\vspace{-4mm}
\end{figure*}

\begin{figure*}[!t]
	\centering
	\includegraphics[width=0.95\textwidth]{figs/fig_8.png}
    \caption{Qualitative pose estimation evaluation on the Occlusion LINEMOD dataset. The red box indicates the ground truth pose while blue is the estimated pose using PrimA6D. Next to the inferred pose is reconstructed object and the predicted primitive associated to each object.}

	\label{fig:linemod_occ_result_fig}
	\vspace{-4mm}
\end{figure*}

\section{Introduction}
\label{sec:intro}

The 6D object pose estimation intends to recover the orientation and translation of the object relative to the camera.
When solving 6D pose estimation, reaching accurate, fast and robust solutions has been a key element of augmented reality \cite{pvnet-2019-cvpr}, robotic tasks \cite{billings2018silhonet, periyasamy2018robust, wang2019densefusion}, and autonomous vehicles \cite{manhardt2019roi}. Despite the widespread use of RGB-D cameras, single image-based pose estimation is still essential when an RGB-D camera is not applicable.

Recent advances in deep learning-based object pose estimation has been groundbreaking. Direct end-to-end 6D pose regression suffered from the non-linearity of the rotation, and studies tackled this issue through orientation representation \cite{saxena-2009-icra} or combining \ac{PnP} and \ac{RANSAC}. The latter approaches reported current \acf{SOTA} performance by leveraging a two-step approach for pose estimation \cite{pvnet-2019-cvpr, brachmann2017dsac, learning-less-2017}. These approaches extract keypoints from the RGB image using deep learning and compute the 6D pose of the object using 2D-3D correspondence matching, such as the \ac{PnP} algorithm.

\begin{figure}[!t]
	\centering
	\includegraphics[width=0.9\columnwidth]{figs/fig_1.pdf}

	\caption{We train VAE to reconstruct objects and the associated primitive followed by the keypoints extraction over the inferred primitive. Keypoints are extracted from each corner plus the center, the same as the object center (i.e., 21).}

	\label{fig:oveview}
	\vspace{-5mm}
\end{figure}

We were also partially motivated by the finding in \cite{sundermeyer2018implicit} that an \ac{AAE} could implicitly represent a rotation when training an \ac{AAE} from a 3D CAD. The main advantage of this approach is that there is no need for manual annotation to prepare ground pose labeled training images. While \cite{sundermeyer2018implicit} only solved for the orientation retrieval given a discretized orientation set, we reconstruct a primitive from which keypoints are extracted under occlusion and complete a full orientation regression.

This paper solves for the direct 6D pose regression by introducing a novel primitive descriptor in \figref{fig:oveview}. The proposed solution, primitive associated 6D (PrimA6D) pose estimation, combines a direct and holistic understanding of an image with keypoint based PnP for the final orientation regression. More specifically, we newly introduce the primitive decoder in addition to the \ac{VAE} to increase the discriminability of the orientation inference. Via this step, like the aforementioned studies, we aim to find reliable keypoints in an image for pose regression extracting from the reconstructed primitives. Differing from previous methods, our method presents the following contributions.

\begin{figure*}[!t]
	\centering
	\includegraphics[width=0.95\textwidth]{figs/fig_2.pdf}

	\caption{The reconstruction stage of PrimA6D. In the reconstruction stage, an encoder using ResNext50 \cite{DBLP:journals/corr/XieGDTH16} and an reconstruction decoder consisted of fully deconvolutional layers form the \ac{VAE}. We reconstruct both object and primitive in a separate decoder. To increase the discriminability, we additionally introduce adversarial loss and train using \ac{GAN}.}

	\label{fig:network}
	\vspace{-4mm}
\end{figure*}

\begin{itemize}

	\item We propose a novel 6D object pose estimation network called PrimA6D, which introduces a rotation primitive reconstruction and its associated keypoints to enhance the orientation inference.

	\item The proposed estimation scheme mitigates the manual effort of preparing label annotated images for training. The proposed primitive is straightforward to generate given rotation in 3D CAD; the trained \ac{VAE} allows us to learn from synthetic images via domain randomization robust to occlusion and symmetry.

	\item We verify the meaningful performance improvement of 6D object pose estimation compared to the existing \ac{SOTA} 6D pose regression methods. The proposed method improved estimation over LINEMOD as 97.62\% (PVNet 86.27\%), Occlusion LINEMOD as 59.77\% (PVNet 40.77\%), and YCB-Video as 94.43\% (PVNet 73.78\%) in terms of ADD(-S).

	\item The proposed method is more generalizable than existing methods showing less performance variance between datasets. This superior generalization capability is essential for robotic manipulation and automation tasks.

\end{itemize}

\section{Method}
\label{sec:method}

\begin{figure}[!t]
	\centering
	\includegraphics[width=1\columnwidth]{figs/fig_3.pdf}
  \caption{The rotation and translation inference. We use keypoints learned from the reconstructed primitive for the orientation estimation using PnP. In the translation inference stage, the network input is created using the output of the reconstruction decoder through object relocalization. }

	\label{fig:pose}
	\vspace{-1mm}
\end{figure}

Given an RGB image, we compute the 6D pose of an object relative to the camera in terms of the its relative transform ${\mathbf{T}}^{C}_{O}$ between the camera coordinate frame ${C}$ and the object coordinate frame ${O}$. The 6D pose is represented by quaternion and translation vectors. Similar to \cite{pix2pose} and \cite{sundermeyer2018implicit}, the proposed method can be accompanied with other detection modules such as M2Det \cite{DBLP:journals/corr/abs-1811-04533}. We use a cropped image using a bounding box detected from other methods as input to our network.

\subsection{Training Set Prepration}

As indicated earlier, our aim is at enabling 6D pose regression without requiring the real images. For each pair of a 3D CAD object and the associated camera pose, we prepare the training set consists of a bounding box, rendered primitive, the target keypoints (the 20 corner points and a center point), and rendered training image. The primitive images represent the rotation of the object having different colors per axis as in \figref{fig:oveview}.

\subsection{Object and Primitive Reconstruction}

Using the training data, the first phase of the proposed method consists of object and primitive reconstruction as shown in \figref{fig:network}. We train the \ac{VAE} having an encoder and two reconstruction decoders. This step is partially similar to \ac{AAE} in \cite{sundermeyer2018implicit}. In \cite{sundermeyer2018implicit}, the authors verified that a trained \ac{AAE} using a geometric augmentation technique learns the representation of the object's orientation and generates the latent code with data for the object orientation. We adopt the same technique in the training phase by rendering the training images with various poses of the object. The training images are augmented using the domain randomization technique \cite{mjeon-2019-cams} to bridge the gap between real and rendered images. Augmented images are fed into the encoder, and clean target images are obtained through the reconstruction decoder using four losses as follows. Differing from \cite{sundermeyer2018implicit}, our training strategy is to use \ac{GAN} by introducing a discriminator for the reconstructed object and primitive.

\subsubsection{Object Reconstruction Loss}

The first loss focuses on the reconstruction of an object by comparing the pixel-wise loss. To prevent overfitting in the reconstruction stage, we use the top-k pixel-wise L2 loss as follow.
\begin{equation}
	\label{eq:reconst_loss}
	\mathcal{L}_{O}=\frac{1}{K}\sum_{i=1}^{K} \ell_{[i]}(\left \| x-\hat{x} \right \|^2),
\end{equation}
where $\ell_{[i]}$ is a function that extracts the pixel with the $i^\text{th}$ largest error \cite{atkloss-2017-nips}. We use $K=128$ in this paper. In \eqref{eq:reconst_loss}, $x$ is the target image pixel and $\hat{x}$ is the predicted image pixel respectively. This initial latent code encompasses the orientation but it is not sufficiently discriminative for regression. To increase the discriminability, we introduce the rotational primitive decoder to embolden the orientation of the code.

\subsubsection{Primitive Reconstruction Loss}

The rendered primitive image becomes the prediction target of the primitive decoder. Using the color-coded axes in primitive image, we designed the color based axis-aligning loss function for the primitive decoder. When computing the loss for the channels, we consider a per-pixel intensity difference $d_c$ for each channels $c \in \mathcal{C} = [R,G,B]$.
\begin{eqnarray}
	\label{eq:primitive}
	d_{c} &=& x_{c}-\hat{x}_{c}\\
	S_{c} &=&\frac{1}{K}\sum_{i=1}^{K}\ell_{[i]}(\exp({\alpha \left | d_c \right |}) \cdot \left \|  d_c \right \|^{2}) \\
	C_{c} &=&\frac{1}{K}\sum_{i=1}^{K}\ell_{[i]}( \left \|  d_c \right \|^{2})\\
	\mathcal{L}_{P}&=&\sum_{c \in \mathcal{C}}S_{c} \exp \left( \frac{C_{c}}{\sum_{k \in \mathcal{C}}C_{k}} \right)
\end{eqnarray}
Here, the $S_{c}$ measures misalignment of each axis in terms of the pixel intensity difference per channel. The $C_{c}$ measures the overall misalignment per channel and discerns between channels by penalizing the channel showing a larger error when the primitive reconstruction loss $L_P$ is constructed. The $\alpha$ is a weight constant ($\alpha=5$ in this paper).

\subsubsection{Overall VAE Loss}

Using these two reconstruction losses with \ac{KL} divergence loss, we train the \ac{VAE} to generate both an object image and a primitive image. By adopting \ac{VAE}, the network encodes input image $x$ to the parameters of a Gaussian distribution $q(z|x)$. In doing so, we minimize the \ac{KL} divergence between $q(z|x)$ and $\mathcal{N}(0,I)$. The overall loss function for the reconstruction stack becomes as below.
\begin{equation}
	\label{eq:loss_kld}
	\mathcal{L}_{R} = \mathcal{L}_{O} + \mathcal{L}_{P} + D_{KL} \left( q(z|x)||\mathcal{N}(0,I) \right)
\end{equation}

\subsubsection{Adversarial Loss}

We further aim to improve the quality of the reconstruction using training strategy of \ac{GAN}. For instance, the reconstructed primitive sometimes becomes ambiguous as shown in \figref{fig:effect_gan} and may result in incorrect keypoints learning. Regarding this issue, introducing adversarial loss improved the primitive/object prediction accuracy and thereby the overall pose inference performance. For the target primitive/object image $X$ and the reconstructed image $\hat{X}$, we consider the following loss to train using \ac{GAN} \cite{goodfellow2014generative}.
\begin{equation}
	\label{eq:loss_adv}
\mathcal{L}_{A} = \mathbb{E}_{\hat{X}}[\log D(\hat{X})] + \mathbb{E}_{X}[\log (1-D(G(X)))].
\end{equation}
Here we use $D{(\cdot)}$ to distinguish if the predicted primitive/object is the target primitive/object from 3D model (true) or generated by $G{(\cdot)}$ (fake). Following the training strategy of \ac{GAN}, we solve for $G$ and $D$ by minimizing $\mathcal{L}^G_{A}=\mathbb{E}_{\hat{X}}[\log D(\hat{X})]$ and maximizing $\mathcal{L}^D_{A}=\mathbb{E}_{X}[\log (1-D(G(X)))]$.

\begin{figure}[!t]
	\centering
	\includegraphics[width=0.8\columnwidth]{figs/fig_4.pdf}
	\caption{The input image of the reconstruction network is ($\kappa$=1.3) larger than the actual bounding-box allowing complete object reconstruction even for inaccurate bounding-box.}

	\label{fig:synthetic_dataset}
	\vspace{-4mm}
\end{figure}

\subsection{Orientation Regression via PnP}

From the reconstructed primitive we estimate the rotation first by learning the keypoints from it. Using generated keypoints in the training set preparation and via pixel-wise L2 loss, we adopt ResNet18 \cite{resnet} and extract the keypoints of the reconstructed primitive using the loss as
\begin{equation}
	\label{eq:pnp_R}
	\mathcal{L}_{rot}=\sum\left \| x_k-\hat{x_k} \right \|^2.
\end{equation}
Since the keypoints consists of corners and the center point of the object which is the same as the primitive center, we utilize this estimated center point, $C=x_c = (u_c,v_c)$ for the translation estimation.

\subsection{Translation Regression}

\input{tab_linemod_add.tex}
\input{tab_linemod_2dp_mae.tex}

Given the estimated orientation, we then infer the 3D translation by computing the scene depth $T_{z}$ of the object center (i.e., primitive center). The reconstructed image is relocated to the original image using the bounding box of the object. This process is called \textit{object relocalization} (\figref{fig:pose}).

As the sample illustration in \figref{fig:synthetic_dataset} shows, the reconstruction occurs \textit{without} assuming the bounding box is centered and \bl{thereby is robust to inaccurate object detection.} The reconstruction phase handles the inaccurate bounding box location by properly reconstructing the object even at the off-centered position. The inflation scaling factor $\kappa$ was chosen to handle the detected bounding box with \ac{IoU} \bl{0.75}. A small $\kappa$ fails to handle inaccuracy in objection detection phase, and a large $\kappa$ might deteriorate the performance with increased empty space near borders. Because this re-localization replaces an object with a reconstructed object, the translation can be regressed over occlusion-free objects. From this relocalized full image, we again use ResNet18 to regress the scene depth $T_{z}$ to the object center.

Using $T_{z}$, the bounding-box, and the camera intrinsic matrix $K$, we compute the $T_{x}$ and $T_{y}$ components of translation as follows from the estimated object center $(u_c, v_c)$. The focal length $f$ and principal point $(u_p,v_p)$ are obtained from the calibration matrix while the bounding box is known when preparing the training data.
\begin{eqnarray}
	\label{eq:trans}
	u_c=f_{u}\frac{T_x}{T_z}+u_p \quad\text{and}\quad
	u_v=f_{v}\frac{T_y}{T_z}+v_p.
\end{eqnarray}

\section{Related Works}
\label{sec:related}

PnP based approaches establish 2D-3D correspondences from keypoints or patches and solve for the pose from these correspondences. Keypoint-based approaches define specific features to infer the pose from an image, such as the corner points of 3D bounding-box. These approaches chiefly involve two steps: first, the 2D keypoint of an object in the image is computed, and then an object pose is regressed using the \ac{PnP} algorithm. In BB8 \cite{rad2017bb8}, the authors predicted the 2D projection corner points of the object's 3D bounding box. Then, they computed the object pose from 2D-3D correspondences using \ac{PnP} algorithm. Similarly, \cite{tekin2018real} additionally added the central point of object to the keypoints of BB8 \cite{rad2017bb8}. Unfortunately, \cite{rad2017bb8,tekin2018real} revealed the weakness to the occlusion that prevents defining the correct bounding box of an object. To alleviate this issue, PVNet \cite{pvnet-2019-cvpr} predicted the unit vectors that point to predefined keypoints for each pixel in an object. By computing the locations of these keypoints using unit vector voting, the authors substantially improved their accuracy. However, these keypoint-based approaches may suffer when an object lacks texture, whereas the proposed primitive can be robustly reconstructed even for small and textureless objects.

The part-based approaches extract parts of an object called patches for pose regression. This method is  frequently used with depth data. For instance, \cite{brachmann2016uncertainty} jointly predicted the object labels and object coordinates for every pixel, then computed the object pose from the predicted 2D-3D correspondence matching. In \cite{oberweger2018making}, the authors extracted heatmaps of multiple keypoints from local patches then computed the object pose using the \ac{PnP} algorithm. Depth data may not always be available and we propose using a single RGB as input. Recently, \cite{pix2pose} exploited \ac{AE} to generate an image representing 3D coordinate per each pixel. Together with the reconstructed image containing 3D information, the expected error per pixel was also estimated to sort meaningful 2D-3D correspondence.

Unlike the aforementioned approaches, which consist of steps, holistic approaches utilize the overall shape of the object appearing in an image to directly regress the object pose relative to the camera. In Deep-6DPose \cite{do2018deep}, the authors found the regions where the objects were located through the \ac{RPN}. Despite solving for both object detection and 6D pose estimation, their method was vulnerable to small and symmetrical objects since they relied on the detected regions. In SSD-6D \cite{kehl2017ssd}, the authors used the discrete 6D pose space to represent the discriminable viewpoint. SSD-6D is a \ac{SSD}-style pose estimation model that extends the work of \cite{liu2016ssd}. Their method had a faster inference time than other methods though it sacrificed accuracy.

Incorporating a mask has been widely adopted. For example, \cite{billings2018silhonet,xiang2017posecnn,periyasamy2018robust,wu2018real} utilized an mask of the object to solve the pose estimation task. The well-known PoseCNN \cite{xiang2017posecnn} required an \ac{ICP} refinement for better accuracy. Stemming from PoseCNN, \cite{wu2018real} enabled a real-time performance with RGB data. The authors of \cite{xiang2017posecnn,wu2018real} also reported meaningful results, but their works were sensitive to the issue of object occlusion. Tackling the occlusion issue, SilhoNet \cite{billings2018silhonet} constructed an occlusion-free mask even from the occluded region, through CNN. This type of mask level restoration enhanced the performance over occlusion, however, yielded low accuracy for symmetrical objects. Unlike these methods relying on masks, we exploit the reconstructed object to not only regress the translation but further tackle the occlusion issue.

All the abovementioned methods require true pose-labeled images for training. Another stream of studies has focused on leveraging latent code without the need for labeled training images via \ac{AAE} to solve this issue. Without requiring the pose label from object 6D pose estimation, \cite{sundermeyer2018implicit} utilized the latent code generated from the object recovery process using \ac{AE}. The object pose was determined by a similarity comparison with the predefined latent codebook for each pose, but the accuracy was dominated by the size of the codebook with a discretized orientation class. We have overcome this accuracy limitation and achieved substantially improved performance from a synthetic dataset.

\section*{ACKNOWLEDGMENT}

\balance
\small
\bibliographystyle{IEEEtranN}


\section{Introduction}
It is difficult to label large-scale data accurately. Therefore, datasets with label noise are ubiquitous in the era of big data. However, label noise will degenerate the performance of deep networks, because deep networks will easily overfit label noise \citep{zhang2016understanding}. Almost all existing methods deal with the label noise problem in \textit{pointwise} manners. Namely, these methods use pointwise losses (e.g., cross-entropy loss), and pointwise noise corrections (e.g., sample selection, loss correction, label correction, and others) \citep{xia2021instance, li2019learning,  zhang2018deep, xia2020parts, han2020survey}.

On the other hand, methods employing \textit{pairwise} manners are very prevailing and have made a great success in machine learning, e.g., supervised metric learning and unsupervised contrastive learning \citep{qi2019simple, boudiaf2020unifying, chen2020simple, he2020momentum}, where relationships between data points are exploited.
Intuitively, the pairwise manners require less pointwise supervision information, i.e., class labels, and might be robust to label noise. In this paper, we naturally ask a question: does learning in a pairwise manner mitigate label noise? This motivates us to introduce a pairwise manner to deal with label noise.

Here we propose a noise reduction perspective on handling label noise: \textit{Class2Simi}, i.e., transforming training data with noisy class labels into data pairs with noisy similarity labels. A class label shows the class that an instance belongs to, while a similarity label indicates whether or not two instances belong to the same class. We theoretically prove that through this transformation, the noise rate becomes lower (see Theorem \ref{lower_rate}). This is because, given a data pair, of which if one point has an incorrect class label or even if both points have incorrect class labels, the similarity label could be correct. Moreover, this transformation also reduces a multi-class classification problem into a binary classification problem.
In label noise learning, the binary problem is easier to handle and a lower noise rate usually results in higher classification performance \citep{patrini2017making}.

\begin{figure*}[h]
	\vspace{-0pt}
	\centering
	\includegraphics[width=1.0\linewidth]{fig/example.pdf}
	\caption{An illustration of the transformation from class labels to similarity labels. Note that $\bar{y}$ stands for the noisy class label and $y$ for the latent clean class label. The labels marked in red are incorrect. If we assume the class label noise is generated according to the transition matrix presented in the upper part of the right column, it can be calculated that the noise rate for the noisy class labels is 0.5 while the noise rate for the noisy similarity labels is 0.25. Note that the transition matrix for similarity labels can be calculated by exploiting the class transition matrix as in Theorem \ref{transform_T}.} \label{fig:label_example}
	\vspace{-10pt}
\end{figure*}

Specifically, we illustrate the transformation and the robustness of similarity labels in Figure \ref{fig:label_example}. In the middle column, we can see the noisy similarity labels of example-pairs $(x_2,x_5)$ and $(x_2,x_4)$ are correct, although there is one mislabeled point in $(x_2,x_5)$, and two mislabeled points in $(x_2,x_4)$. Moreover, if we assume that the noisy class labels in Figure \ref{fig:label_example} are generated according to the latent clean class labels and the class transition matrix (the $ij$-th entry of this matrix denotes the probability that the clean class label $i$ flips into the noisy class label $j$), the noise rate of class labels is $0.5$. Meanwhile, the corresponding similarity transition matrix can be derived from the class transition matrix with the class-priors (see Theorem \ref{transform_T}). The noise rate of similarity labels is $0.25$, which is the proportion of the number of incorrect similarity labels to the number of total similarity labels.

To handle the transformed data pairs with noisy similarity labels, the connection between noisy similarity posterior and clean class posterior should be established. Intuitively, noisy similarity posterior can be linked to clean similarity posterior, and then clean class posterior can be inferred from clean similarity posterior. For the first part, we can draw on the philosophy of dealing with noisy class labels, e.g., selecting reliable data pairs for training, and correcting the similarity loss to learn a robust similarity classifier. For the second part, plenty of similarity metrics can be adopted. As an example, we could adapt the \textit{Forward} \citep{patrini2017making} to learn clean similarity posterior from data with noisy similarity labels. Then, by using the inner product of the clean class posterior \citep{hsu2019multi} to approximate clean similarity posterior, the clean class posterior (and thus the robust classifier) can thereby be learned.

It is obvious that Class2Simi suffers information loss because we can not recover the class labels from similarity labels, which implies that learning only from similarity labels can only cluster data points but can not identify the semantic classes of clusters. In \citet{hsu2019multi}, a pointwise cluster can be learned from similarity labels.
However, in our case, the pairs with similarity labels are constructed from points with class labels, and we could acquire the semantic class information of clusters by pretraining the model from points with class labels without any additional information.
Note that when class labels of points are corrupted, leading to noisy similarity labels, the proposed pretraining still works because the noisy class is assumed to be dominated by its clean class in label noise learning. Thus we do not suffer the major information loss in noisy similarity learning.

It is worthwhile to mention Class2Simi increases the computation cost very slightly, compared with the standard pointwise training. As shown in Figure \ref{fig:net}, most computation is still pointwise. Only the computation of the pairwise enumeration layer \citep{Hsu18_L2C} and the loss are pairwise, while both the forward and backward propagation are pointwise. The pairwise enumeration layer was verified to only introduce a negligible overhead to the training time \citep{hsu2019multi}. Moreover, the transformation is on-the-fly in mini-batches, which means the pairs are quadratic on the batch size other than the whole sample size.

The contributions of this paper are summarized as follows:
\begin{itemize} [topsep=0pt,itemsep=-1ex,partopsep=1ex,parsep=1ex, leftmargin=0.4cm]

	\item We propose a noise reduction perspective on learning with noisy labels, which transforms class labels into similarity labels, reducing the noise rate.

	\item We provide a way to estimate the similarity transition matrix $T_s$ by theoretically establishing its relation to the class transition matrix $T_c$. We show even if the $T_c$ is roughly estimated, the induced $T_s$ still works well.

	\item We design a deep learning method to learn robust classifiers from data with noisy similarity labels and theoretically analyze its generalization ability.

	\item We empirically demonstrate that the proposed method remarkably surpasses the baselines on many datasets with both synthetic noise and real-world noise.
\end{itemize}

The rest of this paper is organized as follows: In Section 2, we formalize the noisy multi-class classification problem. In Section 3, we propose the Class2Simi method and practical implementation. Experimental results are discussed in Section 4. We conclude our paper in Section 5.

\section{Problem Setup and Related Work}
\label{sec:2}
Let $(X,Y)\in\mathcal{X}\times\{1,\ldots,c\}$ be the random variables for instances and clean labels, where $\mathcal{X}$ represents the instance space and $c$ is the number of classes. However, in many real-world applications \citep{zhang2016understanding,zhong2019graph, li2019learning, tanno2019learning, zhang2018deep,xia2021robust,feng2020learning,chou2020unbiased,wu2020multi,zhu2020second, yu2020label, berthon2020confidence}, the clean labels cannot be observed. The observed labels are noisy. Let $\bar{Y}$ be the random variable for the noisy labels. What we have is a sample $\{(x_1,\bar{y}_1),\ldots,(x_n,\bar{y}_n)\}$ drawn from the noisy distribution $\mathcal{D}_\rho$ of the random variables $(X,\bar{Y})$. We aim to learn a robust classifier that could assign clean labels to test data by exploiting the sample with noisy labels.

Existing methods for learning with noisy labels can be divided into two categories: algorithms that result in statistically inconsistent or consistent classifiers.
Methods in the first category usually employ heuristics to reduce the side-effect of noisy labels, e.g., selecting reliable samples \citep{han2018co,yu2019does,Wei_2020_CVPR,wu2020topological,xia2021sample}, reweighting samples \citep{ren2018learning,jiang2018mentornet,ma2018dimensionality,kremer2018robust,reed2014training}, correcting labels \cite{tanaka2018joint,zheng2020error}, designing robust loss functions \cite{zhang2018generalized,xu2019l_dmi,liu2019peer,ma2020normalized}, employing side information \citep{vahdat2017toward,li2017learning}, and (implicitly) adding regularization \citep{li2021provably, li2017learning,veit2017learning,vahdat2017toward,han2018masking,zhang2017mixup,guo2018curriculumnet,hu2020simple,zhang2021learning,han2020sigua}.
Those methods empirically work well in many settings.
Methods in the second category aim to learn robust classifiers that could converge to the optimal ones defined by using clean data. They utilize the transition matrix, which denotes the probabilities that the clean labels flip into noisy labels, to build consistent algorithms  \citep{natarajan2013learning,scott2015rate,liu2016classification,patrini2017making,northcuttlearning,yu2018learning,kremer2018robust,hendrycks2018using,liu2019peer,yao2020dual,xia2020extended}.
The idea is that given the noisy class posterior probability and the transition matrix, the clean class posterior probability can be inferred.

Note that the noisy class posterior and the transition matrix can be estimated by exploiting the noisy data, where the transition matrix additionally needs anchor points \citep{liu2016classification,patrini2017making}.
Some methods assume anchor points have already been given \citep{yu2018learning}. There are also methods showing how to identify anchor points from the noisy training data \citep{liu2016classification}.

\begin{figure*}[!t]
	\centering
	\includegraphics[width=0.98\textwidth]{fig/net.pdf}
	\caption{An overview of the proposed method. We add a pairwise enumeration layer and similarity transition matrix to calculate and correct the predicted similarity posterior. By minimizing the proposed loss $L_{c2s}$, a classifier $f$ can be learned for assigning clean labels. The detailed structures of the Neural Network are provided in Section 4.}
	\label{fig:net}
	\vspace{-10pt}
\end{figure*}

\section{Class2Simi meets noisy supervision}
\label{sec:3}
In this section, we propose a new perspective for learning from noisy data. Our core idea is to transform class labels to similarity labels first, and then handle the noise manifested on similarity labels.

\subsection{Transformation on labels and the transition matrix}
As in Figure \ref{fig:label_example}, we combine every 2 instances in pairs, and if the two instances have the same class label, we assign this pair a similarity label $1$, otherwise $0$. If the class labels are corrupted, the generated similarity labels also contain noise.

The definition of the similarity transition matrix is similar to the class one. The elements in a similarity transition matrix denote probabilities that clean similarity labels $H$ flip into noisy similarity labels $\bar{H}$, i.e., $T_{s,mn} \coloneqq P(\bar{H}=n|{H}=m)$. The dimension of the similarity transition matrix is always ${2\times 2}$. Since the similarity labels are generated from class labels, the similarity noise is determined and, thus can be calculated, by the class transition matrix.

\begin{theorem}
	\label{transform_T}
	Assume that the dataset is balanced (each class has the same amount of instances, and c classes in total), and the noise is class-dependent. Given a class transition matrix $T_c$, such that $T_{c,ij}=P(\bar{Y}=j|Y=i)$. The elements of the corresponding similarity transition matrix $T_s$ can be calculated as
	\begin{align*}
		T_{s,00} &= \frac{c^2 - c - \big(\sum_j(\sum_iT_{c,ij})^2-||T_c||_{\mathrm{Fro}}^2\big)}{c^2 - c}, \\
		T_{s,01} &= \frac{\sum_j(\sum_iT_{c,ij})^2-||T_c||_{\mathrm{Fro}}^2}{c^2 - c}, \\
		T_{s,10} &= \frac{c - ||T_c||_{\mathrm{Fro}}^2}{c}, \qquad
		T_{s,11} = \frac{||T_c||_{\mathrm{Fro}}^2}{c}.
	\end{align*}
\end{theorem}
A detailed proof is provided in Appendix A.

\begin{remark}
	Theorem \ref{transform_T} can easily extend to the setting where the dataset is unbalanced in classes by multiplying each $T_{c,ij}$ by a coefficient $n_i$. $n_i$ is the number of instances from the $i$-th class.
\end{remark}

Note that the similarity labels are only dependent on class labels. If the class noise is class-dependent, the similarity noise is also `class-dependent' (class means similar and dissimilar). Under class-dependent label noise, a binary classification is learnable as long as $T_{00}+T_{11}>1$ \citep{menon2015learning}, where $T$ is the corresponding binary transition matrix; a multi-class classification is learnable if the corresponding transition matrix $T_c$ is invertible. For Class2Simi, in the most general sense, i.e., $T_c$ is invertible, $T_{s,00}+T_{s,11}>1$ holds. Namely, the learnability of the pointwise classification implies the learnability of the reduced pairwise classification. A proof is provided in Appendix B. However, the latter cannot imply the former: As shown in Figure \ref{fig:label_example}, the class transition matrix is not invertible, and thus the pointwise classification is not learnable while the reduced pairwise classification is learnable. Note that this `learnable' is only for the binary pairwise classification in this case. Technically, two conditions must be met to learn a pointwise classifier from pairwise data: (1) The reduced pairwise classification is learnable; (2) The semantic class information is learnable. Generally, the second condition is equivalent to the learnability of the pointwise classification. Thus the learnability for a pointwise classifier of the two learning manners is consistent.

\begin{theorem} \label{lower_rate}
	Assume that the dataset is balanced (each class has the same amount of samples), and the noise is class-dependent. When the number of classes $c \geq 8$, the noise rate of noisy similarity labels is lower than that of the noisy class labels.
\end{theorem}
A detailed proof is provided in Appendix C.

In multi-class classification problems, the number of classes is usually larger than 8. As $c$ becomes larger, the range of `dissimilarity' of data pairs becomes larger, which is conducive to the reduction of the noise rate. Through Class2Simi, the number of \textit{d-pairs} (with similarity label 0) is $(c - 1)$ times as much as that of \textit{s-pairs} (with similarity label 1). Meanwhile, compared with the original noise rate of noisy class labels, the noise rate of noisy similarity labels of s-pairs is higher and that of d-pairs is lower, while the overall noise rate of data pairs is lower, which partially reflects that the impact of label noise is less bad. Notably, the flip from `dissimilar' to `similar' should be more adversarial and thus more important. In practice, it is common that one class has more than one clusters, while it is rare that two or more classes are in the same cluster. If there is a flip from `similar' to `dissimilar' and based on it we split a (latent) cluster into two (latent) clusters, we still have a high chance to label these two clusters correctly later. If there is a flip from `dissimilar' to `similar' and based on it we join two clusters belonging to two classes into a single cluster, we nearly have zero chance to label this cluster correctly later. As a consequence, the flip from `dissimilar' to `similar' is more adversarial and important, thus deserving a larger weight when calculating the noise rate. Here we assign all data pairs the same weight, otherwise, there would be a more reduction of the noise rate. On balance, considering the reduction of the overall noise rate is meaningful.

When dealing with label noise, a low noise rate has many benefits. The most important one is that the noise-robust algorithms will consistently achieve higher performance when the noise rate is lower \citep{han2018co, xia2019anchor,patrini2017making}. Another benefit is that, when the noise rate is low, the complex instance-dependent label noise can be well approximated by class-dependent label noise \citep{cheng2017learning}, which is easier to handle.

\begin{figure*}[!t]
	\vspace{-20pt}
	\begin{center}
		\subfigure[\textit{Similar example}]
		{\includegraphics[width=0.46\textwidth]{fig/pre_simi.png}}
		\qquad
		\subfigure[\textit{Dissimilar example}]
		{\includegraphics[width=0.46\textwidth]{fig/pre_dissimi.png}}
	\end{center}
	\caption{Examples of predicted noisy similarity. Assume class number is $10$; $f(X_i)$ and $f(X_{j})$ are categorical distribution of $X_i$ and $X_{j}$ respectively, which are shown above in the form of area charts. $\hat{{S}}_{ij}$ is the predicted similarity posterior between two instances, calculated by the inner product between two categorical distributions.}
	\label{fig:pre_simi}
	\vspace{-8pt}
\end{figure*}

\subsection{Learning with noisy similarity labels}
\label{risk}

In order to learn a multi-class classifier from similarity labeled data, we should establish relationships between class posterior probability and similarity posterior probability. Here we employ the relationship established in \citep{hsu2019multi}, which is derived from a likelihood model. As in Figure \ref{fig:net}, we denote the predicted clean similarity posterior by the inner product between two categorical distributions: $\hat{S}_{i j} = f\left(X_{i}\right)^\top f\left(X_{j}\right)$. Intuitively, $f(X)$ outputs the predicted categorical distribution of input data $X$ and $f(X_i)^\top f(X_{j})$ can measure how similar the two distributions are. For clarity, we visualize the predicted similarity posterior in Figure \ref{fig:pre_simi}. If $X_i$ and $X_{j}$ are predicted belonging to the same class, i.e., $\operatornamewithlimits{argmax}_{m \in c} f_m(X_i) = \operatornamewithlimits{argmax}_{n \in c} f_n(X_{j})$, the predicted similarity posterior should be relatively high ($\hat{{S}}_{ij} = 0.30$ in Figure 3(a)). By contrast, if $X_i$ and $X_{j}$ are predicted belonging to different classes, the predicted similarity posterior should be relatively low ($\hat{{S}}_{ij} = 0.0654$ in Figure 3(b)). Note that the noisy similarity posterior $P({\bar{H}}_{ij}|X_i,X_j)$ and clean similarity posterior $P({{H}}_{ij}|X_i,X_j)$ satisfy
\begin{align}
	P({\bar{H}}_{ij}|X_i,X_j) = T_s^{\top}P({{H}}_{ij}|X_i,X_j). \label{eq:n2c}
\end{align}
Therefore, we can infer the predicted noisy similarity posterior $\hat{\bar{S}}_{ij}$ from the predicted clean similarity posterior $\hat{{S}}_{ij}$ with the similarity transition matrix. To measure the error between the predicted noisy similarity posterior $\hat{\bar{S}}_{ij}$ and noisy similarity label $\bar{H}_{ij}$, we employ a binary cross-entropy loss function. The final optimization function is
\begin{align*}
	L_{c2s}&(\bar{H}_{ij},\hat{\bar{S}}_{ij}) \\
	&= - \sum_{i,{j}} \bar{H}_{ij} \log \hat{\bar{S}}_{ij} +   (1-\bar{H}_{ij})\log (1-\hat{\bar{S}}_{ij}).
\end{align*}

The pipeline of the proposed Class2Simi is summarized in Figure \ref{fig:net}. The softmax function outputs an estimation for the clean class posterior, i.e., $f(X)=\hat{P}(Y|X)$, where $\hat{P}(Y|X)$ denotes the estimated class posterior. Then a pairwise enumeration layer is added to calculate the predicted clean similarity posterior $\hat{S}_{ij}$ of every two instances. According to Equation (\ref{eq:n2c}), by pre-multiplying the transpose of the noise similarity transition matrix, we can obtain the predicted noisy similarity posterior $\hat{\bar{S}}_{ij}$. Therefore, by minimizing $L_{c2s}$, we can learn a classifier for predicting noisy similarity labels. Meanwhile, before the transition matrix layer, the pairwise enumeration layer will output a prediction for the clean similarity posterior, which guides $f(X)$ to predict clean class labels.

\begin{remark}
    For a better understanding, we formulate Class2Simi in the form combined with \textit{Forward} as an illustration. However, Class2Simi is a meta method that can be applied on top of sample selection, loss correction, label correction, and many other label noise learning methods. We provide another implementation with \textit{Reweight} in Appendix D.
\end{remark}
\subsection{Implementation}
The proposed algorithm is summarized in Algorithm \ref{alg:c2s}. Since learning only from similarity labels will lose the semantic class information, we load the model trained on the data with noisy class labels to provide the semantic class information for similarity learning in Stage 2.

\subsection{Generalization error}

We formulate the above problem in the traditional risk minimization framework \citep{mohri2018foundations}. The expected and empirical risks of employing estimator $f$ can be defined as
\begin{align*}
	{R}(f) = {E}_{(X_i, X_{j}, \bar{Y}_{i}, \bar{Y}_{j}, \bar{H}_{ij}, T_s)\sim {\mathcal{D}_\rho}}[{\ell}(f(X_i), f(X_{j}), {T}_s, \bar{H}_{ij})],
\end{align*}
and
\begin{align*}
	{R}_n(f) = \frac{1}{n^2}\sum_{i=1}^n\sum_{j=1}^n {\ell}(f(X_i), f(X_{j}),{T}_s, \bar{H}_{ij}),
\end{align*}
where $n$ is the training sample size of the noisy data.
Assume that the neural network has $d$ layers with parameter matrices $W_1,\ldots,W_d$, and the activation functions $\sigma_1,\ldots,\sigma_{d-1}$ are Lipschitz continuous, satisfying $\sigma_{j}(0)=0$. We denote by $H: X\mapsto W_d\sigma_{d-1}(W_{d-1}\sigma_{d-2}(\ldots \sigma_1(W_1X)))\in\mathbb{R}$ the standard form of the neural network. $H =\operatornamewithlimits{argmax}_{i\in\{1,\ldots,c\}} h_i$. Then the output of the softmax function is defined as $f_i(X)=\exp{(h_i(X))}/\sum_{j=1}^{c}\exp{(h_j(X))}, i=1,\ldots,c$. We can then obtain the following generalization error bound.
\begin{theorem} \label{thm:bound}
	Assume the parameter matrices $W_1,\ldots,W_d$ have Frobenius norm at most $M_1,\ldots, M_d$, and the activation functions are 1-Lipschitz, positive-homogeneous, and applied element-wise (such as the ReLU). Assume the transition matrix is given, and the instances $X$ are upper bounded by $B$, i.e., $\|X\|\leq B$ for all $X$, and the loss function $\ell$ is upper bounded by $M$. Then, for any $\delta>0$, with probability at least $1-\delta$,
	\begin{align}
		{R}&(\hat{f})- {R}_n(\hat{f}) \leq  M\sqrt{\frac{\log{1/\delta}}{2n}} + \nonumber \\
		& \quad  \frac{(T_{s,11} - T_{s,01})2Bc(\sqrt{2d\log2}+1)\Pi_{i=1}^{d}M_i}{T_{s,11}\sqrt{n}}.
	\end{align}
\end{theorem}
A detailed proof are provided in Appendix E.
\begin{algorithm}[t]
	{\bfseries Input}: training data with noisy class labels; validation data with noisy class labels.

	\textbf{Stage 1: Learn $\hat{T}_s$}

	1: Learn $g(X)=\hat{P}(\bar{Y}|X)$ by training data with noisy class labels, and save the model for Stage 2;

	2: Estimate $\hat{T}_c$ following the optimization method in \citep{patrini2017making};

	3: Transform $\hat{T}_c$ to $\hat{T}_s$.

	\textbf{Stage 2: Learn the classifier $f(X)=\hat{P}({Y}|X)$}

	4: Load the model saved in Stage 1, and train the whole pipeline showed in Figure \ref{fig:net}.

	{\bfseries Output}: classifier $f$.
	\caption{Class2Simi}
	\label{alg:c2s}
\end{algorithm}

Theorem \ref{thm:bound} implies that if the training error is small and the training sample size is large, the expected risk ${R}(\hat{f})$ of the representations for noisy similarity posterior will be small. If the transition matrix is well estimated, the clean similarity posterior as well as the classifier for the clean class will also have a small risk according to Equation (\ref{eq:n2c}) and the Class2Simi relations. This theoretically justifies why the proposed method works well. In the experiment section, we will show that the transition matrices are well estimated and that the proposed method significantly outperforms the baselines.

In Class2Simi, a multi-class classification is reduced to a pairwise binary classification. For data pairs, if a surrogate loss is classification-calibrated, minimizing it leads to minimizing the zero-one loss on the pointwise random variables in the limit case. Otherwise, we cannot guarantee the worst-case learnability of learning pointwise labels from pairwise labels, but it cannot imply the average-case non-learnability either. Theoretically, \cite{bao2020similarity} proved that when the pairwise labels are all correct, for the special case $c =2$, a good model for predicting s-/d-pairs must also be a good model for predicting the original classes, under mild assumptions. In practice, it seems fine to use non-classification-calibrated losses. According to \cite{tewari2007consistency}, the multi-class margin loss (i.e., one-vs-rest loss) and the pairwise comparison loss (i.e., one-vs-one loss) are proved to be non-calibrated, but they are still the main multi-class losses in \cite{mohri2018foundations, shalev2014understanding}.

\begin{table*}[!t]
\vspace{8pt}
\centering
	\begin{center}

		\caption{Means and Standard Deviations of Classification Accuracy over 5 trials on image datasets.}

		\renewcommand{\arraystretch}{1.4}
		\vspace{0pt}
		\label{tab:syn1}
		\scalebox{0.95}
		{
			\begin{tabular}{c|ccc|ccc}
				\hline
				\textbf{\textit{MNIST}}  & Sym-0.2 & Sym-0.4 & Sym-0.6 & Asym-0.2 & Asym-0.4 & Asym-0.6 \\ \hline
				Co-teaching & 97.34$\pm$0.26 & 94.68$\pm$0.52 & 93.36$\pm$0.47 & 97.37$\pm$0.20 & 96.63$\pm$0.41 & 91.33$\pm$0.38\\
				JoCor & 97.48$\pm$0.12 & 96.31$\pm$0.20 & 93.18$\pm$0.27 & 97.31$\pm$0.09 & 95.73$\pm$0.29 & 91.43$\pm$0.28\\
				PHuber-CE & 98.65$\pm$0.18 & 98.17$\pm$0.15 & 97.63$\pm$0.36 & 98.73$\pm$0.09 & 98.36$\pm$0.25 & 97.37$\pm$0.41 \\
				APL & 98.77$\pm$0.21 & 97.06$\pm$0.37 & 97.67$\pm$0.35 & 98.72$\pm$0.10 & 98.45$\pm$0.29 & 97.58$\pm$0.25\\
				S2E & 98.96$\pm$0.27 & 93.27$\pm$2.18 & 89.37$\pm$0.70 & 99.19$\pm$0.05 & 94.47$\pm$1.08 & 92.36$\pm$2.40\\
				Revision & 98.92$\pm$0.09 & 98.42$\pm$0.50 & 98.10$\pm$0.37 & 98.97$\pm$0.06 & 98.58$\pm$0.19 & 98.21$\pm$0.19 \\ \hline
				Reweight & 98.78$\pm$0.16 & 98.26$\pm$0.22 & 97.02$\pm$0.58 & 98.62$\pm$0.19 & 98.12$\pm$0.31 & 96.98$\pm$0.29  \\
				Forward & 98.76$\pm$0.03 & 98.37$\pm$0.25 & 96.89$\pm$0.49 & 98.61$\pm$0.22 & 98.08$\pm$0.33 & 97.43$\pm$0.25  \\ \hline
				R-Class2Simi & {99.04$\pm$0.06} & {98.87$\pm$0.06} & {98.40$\pm$0.17} & {99.06$\pm$0.05} & {98.75$\pm$0.08} & {98.23$\pm$0.20}\\
				F-Class2Simi & \textbf{99.26$\pm$0.07} & \textbf{99.18$\pm$0.06} & \textbf{98.91$\pm$0.09} & \textbf{99.26$\pm$0.05} & \textbf{99.08$\pm$0.07} & \textbf{98.91$\pm$0.07}\\ \hline

				\textbf{\textit{CIFAR10}}   & Sym-0.2 & Sym-0.4 & Sym-0.6 & Asym-0.2 & Asym-0.4 & Asym-0.6 \\ \hline
				Co-teaching & 88.92$\pm$0.45 & 85.97$\pm$1.02 & 75.97$\pm$1.33 & 89.14$\pm$0.36 & 84.77$\pm$1.08 & 76.07$\pm$1.27 \\
				JoCor & 88.46$\pm$0.25 & 85.19$\pm$0.75 & 77.03$\pm$0.92 & 88.96$\pm$0.70 & 85.19$\pm$0.58 & 75.76$\pm$1.31\\
				PHuber-CE & 90.37$\pm$0.26 & 86.05$\pm$0.37 & 74.06$\pm$0.92 & 90.73$\pm$0.22 & 86.06$\pm$0.53 & 73.25$\pm$1.04\\
				APL & 89.07$\pm$0.92 & 85.77$\pm$0.84 & 70.06$\pm$1.06 & 89.97$\pm$0.19 & 85.60$\pm$0.91 & 72.33$\pm$1.68 \\
				S2E & 90.04$\pm$1.22 & 82.05$\pm$1.95 & 57.96$\pm$4.70 & 90.12$\pm$0.97 & 83.16$\pm$1.58 & 64.77$\pm$3.06 \\
				Revision & 90.02$\pm$0.48 & 85.47$\pm$0.71 & 73.92$\pm$2.02 & 89.77$\pm$0.28 & 85.32$\pm$1.36 & 75.24$\pm$1.87  \\ \hline
				Reweight & 89.05$\pm$0.32 & 84.60$\pm$0.45 & 74.87$\pm$1.18 & 89.28$\pm$0.26 & 84.61$\pm$0.62 & 72.77$\pm$1.91 \\
				Forward & 89.63$\pm$0.20 & 87.08$\pm$0.31 & 73.24$\pm$1.33 & 90.03$\pm$0.41 & 86.64$\pm$0.71 & 77.41$\pm$0.43  \\ \hline
				R-Class2Simi & 90.91$\pm$0.26 & 87.80$\pm$0.23 & 79.19$\pm$1.65 & 91.07$\pm$0.21 & 87.78$\pm$0.33 & 78.56$\pm$0.63 \\
				F-Class2Simi & \textbf{91.38$\pm$0.19} & \textbf{88.22$\pm$0.19} & \textbf{79.45$\pm$0.53} & \textbf{91.24$\pm$0.27} & \textbf{87.79$\pm$0.36} & \textbf{79.05$\pm$0.56} \\
				\hline

				\textbf{\textit{CIFAR100}}   & Sym-0.2 & Sym-0.4 & Sym-0.6 & Asym-0.2 & Asym-0.4 & Asym-0.6 \\ \hline
				Co-teaching & 57.14$\pm$0.49 & 52.62$\pm$1.03 & 37.32$\pm$1.67 & 57.82$\pm$0.37 & 51.32$\pm$0.83 & 35.32$\pm$1.68 \\
				JoCoR & 58.32$\pm$0.71 & 51.76$\pm$1.07 & 37.02$\pm$1.33 & 58.61$\pm$0.30 & 49.18$\pm$1.05 & 37.09$\pm$1.82\\
				PHuber-CE & 57.90$\pm$0.31 & 52.36$\pm$0.77 & 37.93$\pm$0.86 & 57.33$\pm$0.71 & 51.29$\pm$0.96 & 36.03$\pm$1.34\\
				APL & 54.03$\pm$0.92 & 49.06$\pm$0.93 & 36.06$\pm$2.02 & 55.62$\pm$0.92 & 48.37$\pm$0.94 & 35.02$\pm$1.72\\
				S2E & 59.37$\pm$1.09 & 43.29$\pm$1.94 & 30.08$\pm$3.91 & 58.92$\pm$1.21 & 42.88$\pm$2.16 & 29.93$\pm$4.05 \\
				Revision & 59.62$\pm$0.97 & 53.26$\pm$0.84 & 35.82$\pm$2.06 & 58.77$\pm$0.93 & 52.72$\pm$1.38 & 37.72$\pm$1.75  \\ \hline
				Reweight & 49.59$\pm$0.74 & 39.72$\pm$0.57 & 22.79$\pm$1.35 & 48.87$\pm$0.96 & 36.65$\pm$0.90 & 17.24$\pm$1.97  \\
				Forward & 48.68$\pm$0.57 & 39.78$\pm$1.23 & 27.01$\pm$0.89 & 47.90$\pm$0.23 & 37.89$\pm$0.57 & 21.71$\pm$1.53  \\ \hline
				R-Class2Simi & {55.45$\pm$0.55} & {50.38$\pm$0.49} & {35.57$\pm$0.75} & {54.95$\pm$0.65} & {47.56$\pm$0.72} & {34.82$\pm$0.58} \\
				F-Class2Simi & \textbf{60.26$\pm$0.18} & \textbf{54.85$\pm$0.60} & \textbf{40.38$\pm$0.58} & \textbf{59.10$\pm$0.13} & \textbf{52.99$\pm$0.78} & \textbf{38.69$\pm$2.84} \\ \hline
			\end{tabular}
		}
	\end{center}
	\vspace{0pt}
\end{table*}

\begin{table*}[!t]
\vspace{8pt}
\centering
	\begin{center}

		\caption{Means and Standard Deviations of Classification Accuracy over 5 trials on text datasets.}

		\renewcommand{\arraystretch}{1.4}
		\vspace{0pt}
		\label{tab:text}
		\scalebox{0.95}
		{
			\begin{tabular}{c|ccc|ccc}
				\hline
				\textbf{\textit{NEWS20}}  & Sym-0.2 & Sym-0.4 & Sym-0.6 & Asym-0.2 & Asym-0.4 & Asym-0.6 \\ \hline
				Co-teaching & 55.32$\pm$0.28 & 51.09$\pm$1.06 & 47.07$\pm$0.83 & 55.29$\pm$0.41 & 53.08$\pm$0.26 & 45.63$\pm$0.75   \\
				JoCor & 52.21$\pm$0.70 & 49.84$\pm$0.92 & 48.83$\pm$0.43 & 55.58$\pm$0.27 & 49.35$\pm$0.62 & 46.21$\pm$0.73\\
				PHuber-CE & 55.73$\pm$0.38 & 54.33$\pm$0.92 & 45.05$\pm$0.49 & 56.76$\pm$0.26 & 51.15$\pm$0.65 & 41.59$\pm$1.05 \\
				APL & 56.91$\pm$0.21 & 53.12$\pm$1.21 & 43.60$\pm$1.28 & 56.11$\pm$0.23 & 50.93$\pm$1.05 & 43.60$\pm$1.28\\
				S2E & 57.93$\pm$0.37 & 47.16$\pm$1.32 & 28.53$\pm$5.04 & 54.89$\pm$1.92 & 50.42$\pm$1.71 & 30.67$\pm$3.12\\
				Revision & 58.06$\pm$0.19 & 52.30$\pm$1.73 & 46.84$\pm$1.09 & 56.41$\pm$0.77 & 53.44$\pm$0.83 & 43.77$\pm$1.08\\ \hline
				Reweight & 53.34$\pm$1.08 & 50.15$\pm$1.33 & 44.73$\pm$0.79 & 53.37$\pm$0.66 & 49.82$\pm$0.44 & 39.46$\pm$1.27  \\
				Forward & 57.30$\pm$0.32 & 53.94$\pm$0.42 & 46.91$\pm$1.48 & 53.58$\pm$0.54 & 49.90$\pm$1.44 & 42.55$\pm$3.81  \\ \hline
				R-Class2Simi & \textbf{58.67$\pm$0.38} & 56.59$\pm$0.74 & 50.48$\pm$0.97 & 58.44$\pm$0.66 & \textbf{55.03$\pm$1.55} & \textbf{47.75$\pm$2.17}\\
				F-Class2Simi & 58.27$\pm$0.47 & \textbf{56.70$\pm$1.13} & \textbf{50.18$\pm$0.89} & \textbf{58.46$\pm$0.68} & 54.92$\pm$1.66 & 46.07$\pm$3.54\\ \hline
			\end{tabular}
		}
	\end{center}
	\vspace{0pt}
\end{table*}

\section{Experiments}
\textbf{Experiment setup.}\ \
We employ three widely used image datasets, i.e., \textit{MNIST} \citep{lecun1998mnist}, \textit{CIFAR-10}, and \textit{CIFAR-100} \citep{krizhevsky2009learning}, one text dataset \textit{News20}, and one real-world noisy dataset \textit{Clothing1M} \citep{xiao2015learning}. \textit{News20} is a collection of approximately 20,000 newsgroup documents, partitioned nearly evenly across 20 different newsgroups. \textit{Clothing1M} has 1M images with real-world noisy labels and additional 50k, 14k, 10k images with clean labels for training, validation and test, and we only use noisy training set in the training phase. Note that the similarity learning method of Class2Simi is based on \textit{clustering} because there is no class information.
Intuitively, for a noisy class, if most instances in it belong to another specific class, we can hardly identify it. For example, assume that a class with noisy labels $\bar{i}$ contains $n_i$ instances with ground-truth labels $i$ and $n_j$ instances with ground-truth labels $j$. If $n_j$ is bigger than $n_i$, the model will cluster class $i$ into $j$. Unfortunately, in \textit{Clothing1M}, most instances with label `5' belong to class `3' actually. Therefore, we merge the two classes and denote the modified dataset by \textit{Clothing1M*} which contains 13 classes. For all the datasets, we leave out 10\% of the training data as a validation set, which is for model selection.

For \textit{MNIST}, \textit{CIFAR-10}, and \textit{CIFAR-100}, we use LeNet \citep{lecun1998gradient}, ResNet-26 with shake-shake regularization \citep{Gastaldi17ShakeShake}, and ResNet-56 with pre-activation \citep{he2016identity}, respectively. For \textit{News20}, we first use GloVe \citep{pennington2014glove} to obtain vector representations for the raw text data, and employ a 3-layer MLP with the Softsign active function. For \textit{Clothing1M*}, we use pre-trained ResNet-50 \citep{he2016deep}. Further details for the experiments are provided in Appendix F.1.

\textbf{Noisy labels generation.}\ \
For clean datasets, we artificially corrupt the class labels of training and validation sets according to the class transition matrix. Specifically, for each instance with clean label $i$, we replace its label by $j$ with a probability of $T_{c,ij}$. In this paper, we consider both symmetric and asymmetric noise settings which are defined in Appendix F.2. \textit{Sym-0.2} means symmetric noise type with a 0.2 noise rate and \textit{Asym-0.2} means asymmetric noise type with a 0.2 noise rate.

\textbf{Baselines.}\ \
In this paper, we compare our method with the following baselines: \textit{Reweight} \citep{liu2016classification}, \textit{Forward} \citep{patrini2017making}, and \textit{Revision} \citep{xia2019anchor}, which utilize a class-dependent transition matrix to model the noise, and learn a robust classifier. Besides, we externally conduct experiments on \textit{Co-teaching} \citep{han2018co}, which is a representative algorithm of selecting reliable samples for training; \textit{JoCoR} \citep{Wei_2020_CVPR}, which employs a joint loss function to select small-loss samples; \textit{PHuber-CE} \citep{menon2020can}, which introduces gradient clipping to mitigate the effects of noise; \textit{APL} \citep{ma2020normalized}, which applies simple normalization on loss functions and makes them robust to noisy labels; \textit{S2E} \citep{yao2020searching}, which properly controls the sample selection process so that deep networks can benefit from the memorization effect. Besides, we conduct experiments on another implementation of the proposed method, which employs \textit{Reweight} (More details are provided in Appendix D). To distinguish these two methods, we call them `F-Class2Simi' and `R-Class2Simi'.

\textbf{Results on noisy image datasets.}\ \ The results in Table \ref{tab:syn1} and Figure \ref{fig:tnoise} demonstrate that Class2Simi achieves distinguished classification accuracy and is robust against the estimation errors on the transition matrix.

From Table \ref{tab:syn1}, overall, we can see that after the transformation, better performance are achieved due to a lower noise rate and the similarity transition matrix being robust to noise. Even for challenging noise rates of 0.6, Class2Simi achieves good accuracy, uplifting about 5 and 10 points on \textit{CIFAR-10} and \textit{CIFAR-100} respectively, compared with the corresponding pointwise methods.

In Figure \ref{fig:tnoise}, we show that the similarity transition matrix is robust against estimation errors. To verify this, we add some random noise to the ground-truth $T_c$ through multiplying every element in class $T_c$ by a random variable $\alpha_{ij}$. We control the noise rate on the $T_c$ by sampling $\alpha_{ij}$ in different intervals, i.e., 0.1 noise means that $\alpha_{ij}$ is uniformly sampled from $\pm[1.1,1.2]$. Then we normalize $T_c$ to make its row sums equal to 1. From Figure \ref{fig:tnoise}, we can see that the accuracy of Forward drops dramatically with the increase of the noise on $T_c$. By contrast, there is only a slight fluctuation of F-Class2Simi, indicating Class2Simi is robust against the estimation errors on the transition matrix.

\begin{figure}
    \centering
    \includegraphics[width=\columnwidth]{fig/noise_on_T.pdf}
    \caption{Means and Standard Deviations of Classification Accuracy over 5 trials on \textit{MNIST} and \textit{CIFAR10} with perturbational ground-truth $\hat{T}_c$.}
    \label{fig:tnoise}
\end{figure}

\vspace{0pt}

\begin{table}[t]
\vspace{0pt}
\centering
	\caption{Classification Accuracy on \textit{Clothing1M*}. }

	\label{tab:real1}
	\renewcommand{\arraystretch}{1.4}
	\scalebox{1.0}
	{
		\begin{tabular}{c|c|c|c}
			\hline
			Co-teaching & 74.70 & JoCoR & 74.98 \\\hline
			PHuber-CE & 73.16 & APL & 58.93 \\\hline
			S2E & 72.30 & Revision & 74.65 \\ \hline
			Forward & 73.88 & F-Class2Simi & 75.41\\ \hline
			Reweight & 74.44 & R-Class2Simi & \textbf{75.76}
			\\ \hline
		\end{tabular}
	}
\end{table}

\begin{table}[!h]
\vspace{0pt}
\centering
	\caption{Classification Accuracy on clean datasets. CE uses class labels and the cross-entropy loss function. C2S refers to Class2Simi.}
	\vspace{-0pt}
	\label{tab:abaltion}
	\renewcommand{\arraystretch}{1.4}
	\scalebox{0.8}
	{
		\begin{tabular}{c|c|c|c|c}
			\hline
			Dataset & \textbf{\textit{MNIST}} & \textbf{\textit{CIFAR10}} & \textbf{\textit{CIFAR100}} & \textbf{\textit{News20}}\\				\hline
			CE & \textbf{99.30$\pm$0.02} & 94.03$\pm$0.14  &58.74$\pm$0.51& \textbf{59.86$\pm$0.39} \\
			\hline
			C2S & 99.24$\pm$0.05 & \textbf{94.05$\pm$0.27} &\textbf{60.36$\pm$0.89}& 59.74$\pm$0.20\\ \hline
		\end{tabular}
	}
	\vspace{-8pt}
\end{table}

\textbf{Results on the noisy text dataset.}\ \ Results in Table \ref{tab:text} show that the proposed method works well on the text dataset under both symmetric and asymmetric noise settings.

\textbf{Results on the real-world noisy dataset.}\ \ Results in Table \ref{tab:real1} show that the proposed method also performs well against agnostic noise.

\textbf{Ablation study.}\ \ To investigate how the similarity loss function influences the classification accuracy, we conduct experiments with the cross-entropy loss function and the similarity loss function on clean datasets over 3 trials, where the $T_c$ is set to an identity matrix. All other settings are kept the same. As shown in Table \ref{tab:abaltion}, on \textit{MNIST}, \textit{CIFAR10}, and \textit{News20}, the similarity loss function does not improve the classification accuracy on clean data, and on \textit{CIFAR100}, the improvement is marginal. However, in Table \ref{tab:syn1} and \ref{tab:text}, the improvements are significant, which reflects the improvements are mainly benefited from the lower noise rate and the reduced noisy binary paradigm.

\section{Conclusion}
This paper proposes a noise reduction perspective on dealing with class label noise by transforming training data with noisy class labels into data pairs with noisy similarity labels. We establish the connection between noisy similarity posterior and clean class posterior and propose a deep learning framework to learn classifiers from the transformed noisy similarity labels. The core idea is to transform pointwise information into pairwise information, which makes the noise rate lower. We also prove that not only the similarity labels but the similarity transition matrix is robust to noise. Experiments are conducted on benchmark datasets, demonstrating the effectiveness of our method. In future work, investigating different types of noise for diverse real-life scenarios might prove important.

\section*{Acknowledgments}
SHW, XBX, and TLL were supported by Australian Research Council Project DE-190101473. BH was supported by the RGC Early Career Scheme No. 22200720, NSFC Young Scientists Fund No. 62006202 and HKBU CSD Departmental Incentive Grant. NNW was supported by National Natural Science Foundation of China Grant 61922066, Grant 61876142. GN and MS were supported by JST AIP Acceleration Research Grant Number
JPMJCR20U3, Japan. MS was also supported by Institute for AI and Beyond, UTokyo.


\section{Introduction}
Interactive recommendation systems capture dynamic personalized user preferences by improving their strategies continuously \cite{mahmood2007learning,thompson2004personalized, Taghipour2008hybrid}.
They have attracted enormous attention and been applied in leading companies like Amazon, Netflix, and Youtube.
The traditional methods to model user-system interactions include Multi-Armed Bandit (MAB) or Reinforcement Learning (RL).
The former views action choices as a repeated single process, while the latter considers immediate and future rewards to model behavors' long-term benefits. RL-based systems employ a Markov Decision Process (MDP) agent that estimates the value based on both actions and states, rather than merely on actions as done by MAB.

However, reinforcement learning-based models can be fooled by small disturbances on the input data \cite{Szegedy2013Intriguing, Goodfellow2014Explaininga}.
Small imperceptible noises, such as adversarial examples, may increase prediction error or reduce reward in supervised and RL tasks---the input noise can be transferred to attack different parameters even different models, including recurrent network and RL \cite{gao2018black,Huang2017Adversarial}.
Besides, the embedding vectors of users, items and relations are piped into RL-based recommendation models, making it challenging for humans to tell the true value or to dig out the real issues in the models.
Attackers can easily leverage such characteristics to disrupt recommendation systems silently,
making defending adversarial attacks a non-trivial task for RL-based recommendation systems.

In this work, we aim to develop a general detection model to detect attacks and increase the defence ability, which provides a practical strategy to overcome the dynamic `arm-race' of attacks and to defend in the long run.
We make the following contributions:
\begin{itemize}
    \item We systematically investigate adversarial attacks and detection approaches with a focus on reinforcement learning-based recommendation systems and demonstrate the effectiveness of the designed adversarial examples and strategically-timed attack.
   \item We propose an encoder-classification detection model for attack-agnostic detection. The encoder captures the temporal relationship among sequence actions in reinforcement learning. We further use an attention-based classifier to highlight the critical time steps out of ample interactive space.
   \item We empirically show that even small perturbations can reduce the performance of most attack methods significantly.
   Our statistical validation shows that multiple attack methods generate similar actions of the attacked system, providing insights into improving the detection performance.
\end{itemize}

\begin{figure}[!ht]
  \centering
    \includegraphics[width=0.5\textwidth]{images/system_structure.png}
  \centering
  \caption
  Our proposed Adversarial Attack and Detection Approach for RL-based Recommender Systems.

  }
  \label{fig-attention-0}
\end{figure}

\section{Methodology}
\label{method}

\subsection{RL-based Interactive Recommendation}
\label{problem-definition}
Interactive recommendation systems suggest items to users and receives feedback. Given a user $u_{j} \in U= \{u_0, u_1, u_2, ..., u_n\}$, a set of items $I = \{i_0, i_1, i_2, ..., i_n\}$, and the user feedback history $i_{k_1}, i_{k_2}, ..., i_{k_{t-1}}$, the recommendation system suggests a new item $i_{k_{t}}$.
This problem represents a Markov Decision Process as follows:
\begin{itemize}
    \item State ($s_t$): a historical interaction between a user and the recommendation system computed by an embedding or encoder module.
    \item Action ($a_t$): an item or a set of items recommended by the RL agent.
    \item Reward ($r_t$): a variable related to a user's feedback to guide the reinforcement model towards true user preference.
    \item Policy ($\pi(a_t | s_t)$): a conditional probability distribution of items which the agent might recommend to a user $u_i$ given the state of last time step $s_{t-1}$.
    The learning process aims to get an optimal policy.
    \item Value function ($Q(s_t, a_t)$): the agent's estimation of reward of current states $s_t$ and recommended item $a_t$. We define the reward as the cosine similarity between user and item embedding vectors.
\end{itemize}
The reinforcement agent could follow REINFORCE with baseline or Actor-Critic algorithm that both consist of a value network and a policy network \cite{xian2019reinforcement}. The attack model may generate adversarial examples using either the value network \cite{Huang2017Adversarial} or the policy network\cite{Pattanaik2017Robust}.

\subsection{Attack Model}
\label{method-attack}

\vspace{1mm}\noindent\textbf{FGSM-based attack.} We define an adversarial attack as an additional small perturbation $\delta$ on benign examples $x$, which can be a composition of embedding vectors of users, relations and items \cite{xian2019reinforcement}.
Unlike perturbations on images or texts, $\delta$ can be large due to the enormous manual work to check the embedding vectors. We define an adversarial example as
\begin{equation}
\begin{aligned}
    & \min\limits_{\delta} R_T = \sum\limits_{t=1}\limits^{T} Q(s_t + \delta,\ a_t).\\
    & a_t = \pi^{*}(a_t | s_t + \delta) \quad  \text{subject to}\  S(s_t, s_t+\delta) \leq l
\end{aligned}
\end{equation}

\vspace{1mm}\noindent\textbf{Attack with smaller frequency.} The strategically-timed attack \cite{Lin2017Tacticsc} aims to decreases the attack frequency without sacrificing the performance of the un-targeted reinforcement attack.
We formally present it below:
\begin{equation}
\begin{aligned}
    & \delta_t = \delta_t * c_t \quad c_t \in \{0, 1\},\quad \frac{\sum_{t=1}^{T}c_t}{T} < d\\
\end{aligned}
\end{equation}
where $c_t$ is a binary variable that controls when to attack; $d < T$ is the frequency of adversarial examples. There are two approaches to generate the binary sequence $c_{1:T}$ optimizing a hard integer programming problem and generating sequences via heuristic methods. Let $p_0, p_1$ be the two maximum probability of an policy $\pi$, $c_t$ be the attack mask on time step, which is different from \cite{Lin2017Tacticsc}:
$$
c_t = (p_0 - p_1) > threshold
$$
We pick out actions that have the max distance between the highest two probabilities, which means we attack on the most confident actions of the agent. Experiments show that this strategy works.
In contrast, Jacobian-based Saliency Map Attack (JSMA) \cite{papernot2016limitations} and Deepfool \cite{moosavi2016deepfool} are based on the gradient of actions rather than the gradient of $Q$ value. One key component of JSMA is saliency map computation, which decides which dimension of vectors (in Image classification is pixels) are modified. Deepfool pinpoints the attack dimension by comparison of affine distances between some class and temporal class.

\subsection{Detection Model}

The detection model is a supervised classifier detects adversarial examples with actions of the reinforcement agent.
Suppose the action distributions of an agent are shifted by adversarial examples (Section \ref{exp} shows statistical evidence of the drift).
Given an abnormal action sequence $a = \pi^{*}(a|s + \delta)$, the detection model aims to establish separating hyperplane between adversarial examples and normal examples, thereby measuring the probability $p(y | a, \theta)$ or $p(y | \pi^{*}, s, \delta, \theta)$, where $y$ is a binary variable indicating whether the input data are attacked.

To detect the adversarial examples presented in the last section, we employ an attention-based classifier.
The detection model consists of two parts. The first is a GRU encoder, to encode the action methods into a low dimensional feature vector. The second is an attention based decoder with classifier to separate different data. This encoder-decoder model has a bottleneck structure that filters out noisy information. The formulation of GRU is as follows:
\begin{equation}
\begin{aligned}
    & z_t = \sigma_g(W_z a_t + U_z h_{t-1})\\
    & r_t = \sigma_g(W_r a_t + U_r h_{t-1})\\
    & \hat{h_t} = tanh(W_h a_t + U_h \circ h_{t-1})\\
    & h_t = (1 - z_t) \circ h_{t-1} + z_t \circ \hat{h_t}
\end{aligned}
\end{equation}

We use $a_{1:T}$ to denote action sequence, which is a series of user relation vectors or item embedding vectors. $h_{t}$ is the output of encoder.

The attention-based decoder is formulated below.
\begin{equation}
\begin{aligned}
    & \alpha_{t} = Softmax(W_{e} e + b_{e})_{t}\\
    & out_{t+1}, hid_{t+1} = GRU(\alpha_{t} a_t, hid_{t}) \\
    & p_t = Sigmoid(W_{out} out_t + b_{out})
\end{aligned}
\end{equation}
where $e$ is the combined vector of action embedding, hidden states $hid$ and encoder output, the combination method is a linear unit with ReLU activation. GRU is reused to generate multiple hidden states at each time steps. Loss is computed at each time step.
After processed through the attention model and GRU, the vector is then piped into a linear unit with sigmoid function to predict if the agent is polluted. The loss function is the cross entropy between the true label and corresponding probability,
$$
J(Att(a_{1:T}), y) = - y \circ log(p)
$$

\section{Experiments}
\label{exp}
In this section, we report our experiments to evaluate attack methods and our detection model.

\subsection{Dataset and Experiment Setup}
\label{exp-dataset}
We conduct experiments following \cite{chen2019large} and \cite{xian2019reinforcement} over a \textit{Amazon dataset} \cite{he2016ups}. This public dataset contains user reviews and metadata of the Amazon e-commerce platform from 1996 to 2014. We utilize three subsets, namely Beauty, Cellphones, and Clothing, which are originally provided by \cite{xian2019reinforcement} on Github . Details about Amazon dataset analysis can be found in \cite{xian2019reinforcement}.

Our experiments are based on \cite{xian2019reinforcement}.
During dataset preprocessing,feature words with higher TF-IDF scores than 0.1 are filtered out.
70\% of data in each dataset comprises the training set (the rest are the test set). We take actions of reinforcement agent as the detection data.
We define the actions of PGPR \cite{xian2019reinforcement} as heterogeneous graph paths that start from users and have a length of 4.
The three Amazon sub-datasets (Beauty, Cellphones, and Clothing) contain 22,363, 27,879, and 39387 users, respectively.
To accelerate experiments, we use the first 10,000 users of each dataset to produce adversarial examples. Users in Beauty get, on average, 127.51 paths. The counterparts for Cellphones and Clothing are 121.92 and 122.71. As the number of paths is large, we utilize the first 100,000 paths for train and validation with split ratio 80/20.  We randomly sampled 100,000 paths from each action file to form the test set.

We attack trained RL agent with methods in section \ref{method-attack}. We slightly modify JSMA and Deepfool for our experiments---we create the saliency map by calculating the product of the target label and temporal label and achieve both effectiveness and higher efficiency (by 0.32 seconds per iteration) of JSMA;
we decrease the computation load on a group of gradients on Deepfool by sampling.
Besides, we set the hidden size of the GRU to 32 for the encoder, the drop rate of the attention-based classifier to 0.5, the maximum length of a user-item path to 4, the learning rate and weight decay of the optimization solver, Adam, to 5e-4 and 0.01, respectively.

\subsection{Attack Experiments}

\vspace{1mm}\noindent\textbf{Adversarial attack results.} We are interested in how vulnerable the agent is to perturbation in semantic embedding space. An attack will be effective if a small perturbation leads to a notable performance reduction.
We reuse the evaluation metrics of \cite{xian2019reinforcement}, namely Normalized Discounted Cumulative Gain (NDCG), Recall, Hit Ratio (HR), and Precision for evaluation. All metrics are computed based on the top 10 sorted predictions for each user, and they are presented in percentage without specific notion.

Table \ref{tab-basic-attack} shows the performance of different attack methods. Attack results share the same trend with the distribution discrepancy. Most attack methods significantly reduce the performance of the reinforcement system. FGSM $l_1$ achieves the best performance. It reveals that single dimension attack can change the agent's action drastically. While FGSM $l_2$ is less effective, where the metrics just fluctuates around the original baseline (Table \ref{tab-basic-attack}), partly because of small disturbance created by $l_2$ method. Specifically, JSMA achieves comparable results as $l_{inf}$ with a small attack region.
Attacks on Clothing and Cellphones sub-datasets show similar effects.

\begin{table*}
  \caption{Adversarial attack results, MMD between benign distribution and adversarial distribution on Amazon Beauty}
  \centering
  \label{tab-basic-attack}
    \begin{tabular}{c c c c c c|| c c c}
    \toprule
    Data &Parameters &NDCG &Recall &HR &Precision &MMD-org &MMD-$l_1$\\
    \hline
    Original &- &4.654 &6.572 &13.993 &1.675 &0.121 &0.620\\
    FGSM $l_1$ &$\epsilon=$0.1 &2.695 &3.714 &6.599 &0.693 &0.604 &0.010\\
    FGSM $l_2$ &$\epsilon=$1.0 &4.567 &6.555 &13.751 &1.653 &0.016 &0.573\\
    FGSM $l_{inf}$ &$\epsilon=$0.5 &2.830 &3.909 &7.351 &0.787 &0.570 &0.011 \\
    JSMA &- &2.984 &3.844 &8.254 &0.931 &0.412 &0.034\\
    Deepfool &- &3.280 &4.352 &9.548 &1.050 &0.177 &0.458\\
    \bottomrule
\end{tabular}
\end{table*}

\vspace{1mm}\noindent\textbf{Impact of attack frequency.}
We conduct two experiments on attack frequency, random attack and strategically attack. The difference is if adversarial examples are crafted with a frequency parameter $p_{freq}$, or generated by the method shown in Section \ref{method-attack}. The NDCG metric is presented in Figure \ref{fig-attack-time}; other metrics have a similar trend. It can be seen from \ref{fig-attack-time} that the random attack performs worse than the strategically-timed attack. With strategically timed method, attacking $\frac{1}{3} - \frac{1}{2}$ time steps achieves a significant reduction in all metrics.

\begin{figure}[!htb]
  \centering
    \includegraphics[width=0.5\textwidth]{images/attack_time_NDCG.png}
  \centering
  \caption{NDCG of attack frequency on Beauty and Clothing subsets. Dashdot lines represent random attacks, solid lines are strategically-timed attacks. Blue and green lines are FGSM $l_{inf}$ and  $l_1$ attacks respectively.}
  \label{fig-attack-time}
\end{figure}

\vspace{1mm}\noindent\textbf{Analysis of adversarial examples.}
We use Maximum Mean Discrepancy as statistical measures to capture distribution distance.
This divergence is defined as:
$$
MMD(k, X_{org}, X_{adv}) = \sup\limits_{k \in K}\Bigg(\frac{1}{n}\sum\limits_{i=1}^{n}k(x_{org,i}) - \frac{1}{m}\sum\limits_{i=1}^{n}k(x_{adv,i})\Bigg)
$$
where $k$ is the kernel function, i.e., a radial basis function,
$X_{org}, X_{adv}$ are benign and adversarial examples.

MMD-org reveals the discrepancy between the original and adversarial datasets. And MMD-$l_1$ presents the discrepancy among different attack methods. The results (Table \ref{tab-basic-attack}) show that the adversarial distribution is different from the original distribution. Also, the disturbed distributions are closed to each other regardless of the attack type.
This insight makes it clear that we can use a classifier to separate benign data and adversarial data and it can detect several attacks at the same time, which might be transferred to other reinforcement learning attack detection tasks.

\subsection{Detection Experiments}

From a statistical perspective, the above analysis shows that one classifier can detect multiple types of attacks.
We evaluate the detection performance using Precision, Recall and F1 score.

Our attention-based detection model is trained on FGSM $l_1$ attack with $\epsilon$ at 0.1 and detects all attack types. The results (Table \ref{tab-detection-basic}) show that the attack are stronger, the model achieves better performance.
$l_{\infty}$ attack validates this trend, which shows that our model can detect weak attacks as well. The result of detection on $l_2$ attack can be reasoned with MMD analysis shown above, high precision and low recall show that most $l_2$ adversarial examples are close to benign data which confuses the detector.
The $l_1$ attack with $\epsilon=1.0$ validates that our detector performs well yet achieves worse performance on other tests of Cellphones dataset.

\begin{table}[!htb]
  \caption{Detection Result \& Factor Analysis}
  \centering
  \label{tab-detection-basic}
  \begin{tabular}{cccccc}
    \toprule
    Dataset &Attack &Precision &Recall &F1 Score\\
    \hline
    Beauty &$l_1$ 0.1 &0.919 &0.890 &0.904\\
           &$l_2$ 1.0 &0.605 &0.119 &0.199\\
           &$l_{inf}$ 0.5 &0.918 &0.871 &0.894\\
           &JSMA &0.910 &0.793 &0.848\\
           &Deepfool &0.915 &0.840 &0.876\\
    \hline
    Cellphones &$l_1$ 0.1 &0.801 &0.781 &0.791\\
           &$l_2$ 1.0 &0.754 &0.593 &0.664\\
           &$l_{inf}$ 0.5 &0.795 &0.752 &0.773\\
           &$l_1$ 1.0 &0.810 &0.825 &0.817\\
    \hline
    Clothing &$l_1$ 0.1 &0.911 &0.866 &0.888\\
           &$l_2$ 1.0 &0.541 &0.099 &0.168\\
           &$l_{inf}$ 0.5 &0.912 &0.879 &0.895\\
    \midrule
    Dataset &Frequency &Precision &Recall &F1 Score\\
    \hline
    Beauty &$l_1$ 0.02 &0.823 &0.362 &0.503\\
           &$l_1$ 0.08 &0.918 &0.872 &0.894\\
           &$l_1$ 0.3 &0.922 &0.927 &0.924\\
    \midrule
    Dataset &Frequency &Precision &Recall &F1 Score\\
    \hline
    Beauty &$l_1$ 0.579 &0.921 &0.912 &0.917\\
           &$l_1$ 0.316 &0.918 &0.879 &0.898\\
           &$l_1$ 0.118 &0.837 &0.401 &0.543\\
    \bottomrule
\end{tabular}
\end{table}

Our results on factor analysis (Table \ref{tab-detection-basic}) show that the detection model can detect attacks even under low attack frequencies.
But the detection accuracy decreases as the attack frequency drops---the recall reduces significantly to 40.1\% when 11.8\% examples represent attacks.

\section{Conclusion}
Adversarial attacks on reinforcement learning-based recommendation system can degrade user experience.
In this paper, we systematically study adversarial attacks and their factor impacts. We conduct statistical analysis to show classifiers, especially an attention-based detector, can well separate the detection data. Our extensive experiments show both our attack and detection models achieve satisfactory performance.

\bibliographystyle{ACM-Reference-Format}