text:
\section{Introduction}}

\IEEEPARstart{D}{eep} learning has shown very impressive performance on various vision problems, e.g., classification, detection and semantic segmentation. Although there are many factors for the success of deep learning, one of the most important is the availability of large-scale datasets with clean annotations like ImageNet~\cite{Deng2009CVPR_ImageNet}.

However, collecting a large scale dataset with clean labels is expensive and time-consuming. On one hand, expert knowledge is necessary for some datasets such as the fine-grained CUB-200~\cite{Wah2011CIT_CUB_200_2011}, which demands knowledge from ornithologists. On the other hand, we can easily collect a large scale dataset with noisy annotations from various websites~\cite{Fergus2010PIEEE_CollectDataset,Krause2016ECCV_CollectDataset,Schroff2011TPAMI_CollectDataset}. These noisy annotations can be obtained by extracting labels from the surrounding texts or using the searching keywords~\cite{Xiao2015CVPR_Clothing1M}. For a huge dataset like JFT300M (which contains 300 million images), it is impossible to manually label it and inevitably about 20\% noisy labels exist in this dataset~\cite{Sun2017ICCV_JFT300M}. Hence, being able to deal with noisy labels is essential.

The label noise problem has been studied for a long time~\cite{Angluin1988ML_PreviousWork,Quinlan_1986ML_PreviousWork}. Along with the recent successes of various deep learning methods, noise handling in deep learning has gained momentum, too~\cite{Reed2015ICLR_Bootstrapping,Sukhbaatar2014_AN,Xiao2015CVPR_Clothing1M}. However, existing methods often have prerequisites that may not be practical in many applications, e.g., an auxiliary set with clean labels~\cite{Xiao2015CVPR_Clothing1M} or prior information about the noise~\cite{Patrini2017CVPR_Forward}. Some methods are very complex~\cite{Yao2018TIP_QualityEmbedding}, which hurts their deployment capability. Overfitting to noise is another serious difficulty. For a DNN with enough capacity, it can memorize the random labels~\cite{Zhang2017ICLR_NetworkMemory}. Thus, some noise handling methods may finally still overfit and their performance decline seriously, i.e., they are not robust. Their accuracies on the clean test set reach a peak in the middle of the training process, but will degrade afterwards and the accuracies after the final training epoch are poor~\cite{Patrini2017CVPR_Forward,Vahdat2017NIPS_CNNCRF}.

We attack the label noise problem from two aspects. First, we model the label for an image as a distribution among all possible labels~\cite{Gao2017TIP_DLDL} instead of a fixed categorical value. This \emph{probabilistic} modeling lends us the flexibility to handle noise-contaminated and noise-free labels in a \emph{unified} manner. Second, inspired by~\cite{Tanaka2018CVPR_Daiki}, we maintain and update the label distributions in both network parameter learning (in which label distributions act as labels) and label learning (in which label distributions are updated to correct noise). Unlike~\cite{Tanaka2018CVPR_Daiki} which updates labels simply by using the running average of network predictions, we correct noise and update our label distributions in a principled \emph{end-to-end} manner. The proposed framework is called PENCIL, meaning \emph{probabilistic end-to-end noise correction in labels}. The PENCIL framework only uses the noisy labels to initialize our label distributions, then iteratively correct the noisy labels by updating the label distributions, and the network loss function is computed using the label distributions rather than the noisy labels.

Our contributions are as follows.%
\begin{itemize}
\item We propose an end-to-end framework PENCIL for noisy label handling. PENCIL is independent of the backbone network structure and does not need an auxiliary clean dataset or prior information about noise, thus it is easy to apply. PENCIL utilizes back-propagation to probabilistically update and correct image labels in addition to updating the network parameters. To the best of our knowledge, PENCIL is the first method in this line.
\item We propose a variant of the DLDL method~\cite{Gao2017TIP_DLDL}, which is essential for correcting noise contained in our label distributions. PENCIL achieves state-of-the-art accuracy on datasets with both synthetic and real-world noisy labels (e.g., CIFAR-10, CIFAR-100 and Clothing1M). We also propose an attention structure and extend the PENCIL framework to handle multi-label tasks without \emph{or} with label noise.
\item Unlike DLDL, we use inverse KL-divergence in our method. And we show that inverse KL-divergence is indeed more suitable for noise correction than the original KL-divergence.
\item PENCIL is robust. It is not only robust in learning with noisy labels, but also robust enough to apply in datasets with zero or small amount of \emph{potential} label noise (e.g., CUB-200) to improve accuracy.
\end{itemize}

A preliminary version of the PENCIL framework has appeared as a conference publication~\cite{PENCIL_CVPR_2019}.
\section{Related Works}

We first briefly introduce related works that inspired this work and other noise handling methods in the literature.

Deep label distribution learning (DLDL) was introduced in \cite{Gao2017TIP_DLDL}, which was proposed to handle label \emph{uncertainty} by converting a categorical label (e.g., 25 years old) into a label distribution (e.g., a normal distribution whose mean is 25 and standard deviation is 3). The DLDL method uses constant label distributions and the Kullback-Leibler divergence to compute the network loss. In PENCIL, we use label distributions for a different purpose such that the label distributions can be updated and hence noise can be probabilistically corrected. The original DLDL method did not work in our setup and we designed a new loss function in PENCIL to overcome this difficulty.

For deep learning methods, \cite{Zhang2017ICLR_NetworkMemory} showed that a deep network with large enough capacity can memorize the training set labels even when they are randomly generated. Hence, they are particularly susceptible to noisy labels. Label noise can lead to serious overfitting and dramatically reduce network accuracy. However, \cite{Tanaka2018CVPR_Daiki} observed that when the learning rate is high, DNNs may maintain relatively high accuracy (i.e., the impact of label noise is not significant). This observation was utilized in~\cite{Tanaka2018CVPR_Daiki} to maintain an estimate of the labels using the running average of network predictions with a large learning rate. Then, these estimates were used as supervision signals to train the network. PENCIL is inspired by this observation and~\cite{Tanaka2018CVPR_Daiki}, too.

Label noise is an important issue and has long been researched~\cite{Angluin1988ML_PreviousWork,Quinlan_1986ML_PreviousWork}. There are mainly two types of label noise: symmetric noise and asymmetric noise, which are modeled in~\cite{Larsen1998ICAASSP_SN} and~\cite{Sukhbaatar2014_AN}, respectively. \cite{SurveyAboutNoise} is a survey of relatively early methods. \cite{Rolnick2017_DLRobustness} argued that deep neural networks are inherently robust to label noise to some extent. And, deep methods have achieved state-of-the-art results in recent years. Hence, we mainly focus on noise handling in deep learning models in this section.

One intuitive and easy solution is to delete all the samples which are considered as unreliable~\cite{Brodley1999JAIR_RemoveSample}. However, many difficult samples will be deleted, but these samples are important to an algorithm's accuracy~\cite{Guyon1996KDD_RemoveSampleIsNotGood}. Thus, more profound noisy label handling methods become necessary.

There are mainly two lines of attack to the the noisy label problem: constructing a special model based on noisy labels or using a robust loss function. The objective of these methods is to construct a noise-aware model which explicitly deals with noisy labels. \cite{Xiao2015CVPR_Clothing1M} constructed a model to deal with noisy labels, and tested their method on a real-world dataset collected by them. \cite{Vahdat2017NIPS_CNNCRF} proposed a framework called CNN-CRF, which combined convolutional neural networks (CNN) with conditional random fields (CRF) to characterize noisy labels. \cite{Yao2018TIP_QualityEmbedding} utilized similar ideas to determine the confidence of each label. This approach is gaining popularity in recent years (e.g., in~\cite{Lee2018CVPR_CleanNet,Ma2018ICML_LID,Wang2018CVPR_LOF}), and different techniques such as local inherent dimensionality have been brought into the noisy label learning domain.

Another effective approach is to design robust loss functions in order for a noise-tolerant model. Forward and backward methods~\cite{Patrini2017CVPR_Forward} explicitly modeled the noise transition matrix in loss computation. \cite{Ghosh2017AAAI_MSE} investigated the robustness of different loss functions, such as the mean squared loss, mean absolute loss and cross entropy loss. \cite{Zhang2018NIPS_Lq} combined advantages of the mean absolute loss and cross entropy loss to obtain a better loss function.

\cite{Tanaka2018CVPR_Daiki} did not fall in these two categories. It is special in the sense that it replaced the noisy label with their own estimate of the label (i.e., running average of the network's predictions). This approach is effective in noise handling but ad-hoc. PENCIL is partly inspired by this work, but more principled and effective.

Existing methods usually have prerequisites that are impractical, such as demanding an additional clean dataset (e.g., to curb overfitting) or a ground-truth noise transition matrix. When these prerequisites are not satisfied, they often fail to produce robust models. These methods are sometimes too complex to be deployed in real-world applications. In contrast, the proposed PENCIL method does not require additional information, and it can be easily applied to any backbone network.

\section{The Proposed PENCIL Method}

First of all, we define the notations for our study. Column vectors are denoted in bold (e.g., $\vec{x}$) and matrices in capital form (e.g., $X$). Specifically, $\vec{1}$ is a vector of all-ones. We use both hard labels and soft labels. The hard-label space is $\mathcal{H} = \{\vec{y}: \vec{y} \in \{0,1\}^c, \vec{1}^\top \vec{y} = 1 \}$, and the soft-label space is $\mathcal{S} = \{\vec{y}: \vec{y} \in [0,1]^c, \vec{1}^\top \vec{y} = 1 \}$. That is, a soft-label is a label distribution.

\subsection{Probabilistic Modeling of Noisy Labels}\label{sec:pencil_intr}

In a $c$-class classification problem, we have a training set $X = \{\vec{x}_1,\vec{x}_2,\dots,\vec{x}_n\}$. In the ideal scenario, every image $\vec{x}_i$ has a clean label $\vec{y}_i \in \mathcal{H}$, which is a one-hot vector (i.e., equivalent to an integer between 1 and $c$). In our noisy label problem, the labels might be wrong with relatively high probability and we use $\hat{\vec{y}}_i \in \mathcal{H}$ to denote labels which may contain noise. Using cross entropy, the loss function is
\begin{equation}
\mathcal{L} = -\frac{1}{n}\sum_{i=1}^n\sum_{j=1}^c \hat{y}_{ij}\log f_j(\vec{x}_i; \vec{\theta})\,,  \label{1}
\end{equation}
where $\hat{y}_{ij}$ is the $j$'th element of $\hat{\vec{y}}_i$, $f$ is a model's prediction (processed by the softmax function) and $\vec{\theta}$ is the set of network parameters.

In PENCIL, we maintain a label distribution $\vec{y}_i^d \in \mathcal{S} = \{\vec{y}: \vec{y} \in [0,1]^c, \vec{1}^\top \vec{y} = 1 \}$ for every image $\vec{x}_i$, which is our estimate of the \emph{underlying noise-free} label for $\vec{x}_i$. $\vec{y}_i^d$ is used as the pseudo-ground-truth label in our learning, which is initialized based on the noisy label $\hat{\vec{y}}_i$. It is continuously updated (i.e., the noise is gradually corrected) \emph{through back-propagation}. This probabilistic setting allows ample flexibility for noise correction. Note that our probabilistic modeling of the noisy labels is different from that in DLDL~\cite{Gao2017TIP_DLDL}. Label distributions in DLDL are fixed and cannot be updated.

In~\cite{Gao2017TIP_DLDL}, the loss function is KL-divergence:
\begin{gather}
\mathcal{L}  = \frac{1}{n}\sum_{i = 1}^n KL(\vec{y}_i^d || f(\vec{x}_i; \vec{\theta})), \text{and}  \label{2} \\
\mathrm{KL}(\vec{y}_i^d || f(\vec{x}_i; \vec{\theta})) = \sum_{j  = 1}^c y_{ij}^d \log\left(\frac{y_{ij}^d}{f_j(\vec{x}_i; \vec{\theta})}\right) \,.  \label{3}
\end{gather}
This loss is used in~\cite{Tanaka2018CVPR_Daiki}, too. However, KL-divergence is an asymmetric function. Hence, if we exchange the two operands in Eq.~\ref{2}, we obtain a new loss function
\begin{gather}
\mathcal{L}  = \frac{1}{n}\sum_{i = 1}^n KL(f(\vec{x}_i; \vec{\theta}) || \vec{y}_i^d), \text{and} \label{4}\\
\mathrm{KL}(f(\vec{x}_i; \vec{\theta}) || \vec{y}_i^d) = \sum_{j  = 1}^c f_j(\vec{x}_i; \vec{\theta}) \log\left(\frac{f_j(\vec{x}_i; \vec{\theta})}{y_{ij}^d}\right) \,.\label{5}
\end{gather}
We will soon show that Eq.~\ref{4} is more suitable for noise handling. In fact, Eq.~\ref{2} led to very poor results in our experiments and we propose to use Eq.~\ref{4} as one of the loss functions in PENCIL. More details will be discussed in Section~\ref{sec:class_loss}.

\subsection{End-to-end Noise Correction in Labels}

Our label distribution $\vec{y}^d$ models the unknown noise-free label for $\vec{x}_i$. Hence, we need to estimate these distributions in our learning process. Let $\vec{X}$ and $\vec{Y}^d$ be the union of $\vec{x}_i$ and $\vec{y}_i^d$ (for all $1 \le i \le n$), respectively. Different from \cite{Tanaka2018CVPR_Daiki}, we let $\vec{Y}^d$ be part of the parameters that are to be updated in the back-propagation process. That is, PENCIL not only updates the network parameters $\vec{\theta}$ as in traditional networks, but also updates $\vec{Y}^d$ (i.e., $\vec{y}_i^d$) in every iteration. Therefore, we optimize both network parameters and label distributions as follows:
\begin{equation}
\min\limits_{\vec{\theta}, \vec{Y}^d} \mathcal{L}(\vec{\theta}, \vec{Y}^d|\vec{X})\label{6}
\end{equation}
The overall architecture of PENCIL is shown in Fig.~\ref{fig:1}.

\begin{figure*}[t]
 \centering
 \includegraphics[width=0.6\linewidth]{framework}
  \caption{The PENCIL learning framework. We use label distributions $\vec{y}^d$ (which is the softmax transformed version of label initialization variables $\tilde{\vec{y}}$) to replace noisy labels $\hat{\vec{y}}$. The label distributions are updated in every iteration using three loss functions, among which the classification loss and compatibility loss updates $\vec{y}^d$ by requiring the label distributions produce both smooth models and not too distant from the noisy labels.}
\label{fig:1}
\end{figure*}

In the PENCIL framework, three types of ``labels'' ($\vec{y}^d$, $\hat{\vec{y}}$ and $\tilde{\vec{y}}$) are involved. Label distribution $\vec{y}^d$ is updated by back-propagation. In the end, $\vec{y}^d$ will be a good estimate of the underlying unknown noise-free label (i.e., noise corrected label). $\tilde{\vec{y}}$ is a variable that assists $\vec{y}^d$ to be normalized to a probability distribution, by
\begin{align}
\vec{y}^d = \mathrm{softmax}(\tilde{\vec{y}}) \,.
\label{7}
\end{align}
Hence, $\tilde{\vec{y}}$ is not constrained and can be updated freely using back-propagation, but $\vec{y}^d$ is always a valid distribution.

The original noisy label $\hat{\vec{y}}$ does not directly impact the parameter ($\vec{\theta}$) learning. However, it is useful because we use it to indirectly initialize our label distribution $\vec{y}^d$. At the start of PENCIL, $\tilde{\vec{y}}$ is initialized by $\hat{\vec{y}}$ as follows:
\begin{equation}
\tilde{\vec{y}} = K \hat{\vec{y}} \,,
\label{8}
\end{equation}
where $K$ is a large constant ($K=10$ in our experiments), and hence from Eq.~\ref{7} we have $\vec{y}^d \approx \hat{\vec{y}} $ after this initialization.

\subsection{Compatibility Loss}

The noisy label $\hat{\vec{y}}$ is also useful in PENCIL's loss computation. In fact, there are lots of (e.g., 80\% of) correct labels even in datasets with noisy labels. Therefore, we should not let the estimated label distribution $\vec{y}^d$ be completely different from those noisy labels $\hat{\vec{y}}$.

We define a compatibility loss $\mathcal{L}_{o}(\hat{\vec{Y}},\vec{Y}^d)$ to enforce this requirement, as
\begin{align}
\mathcal{L}_{o}(\hat{\vec{Y}},\vec{Y}^d) = -\frac{1}{n}\sum_{i=1}^n\sum_{j=1}^c \hat{y}_{ij}\log y^d_{ij} \,,
\end{align}
which is a classic cross entropy loss between label distribution and noisy label.

\subsection{Classification Loss}\label{sec:class_loss}
Selection of the classification loss is very important, because we will update the value of $\vec{y}^d$ by its gradient directly. And according to Eq.~\ref{7}, we have:
\begin{equation}
\frac{\partial y_{ik}^d}{\partial \tilde{y}_{ij}} =  y_{ik}^d(\delta_{k=j} - y_{ij}^d)
\,,
\label{10}
\end{equation}
in which $\delta_{k=j} = 1$ if $k = j$, and equals $0$ if $k \not= j$.

Assume that $\mathcal{L}_c$ is an \emph{arbitrary} classification loss function, and that we only consider one single input, we then can calculate the gradient of $\tilde{\vec{y}}$ as:
\begin{align}
\frac{\partial\mathcal{L}_c}{\partial \tilde{y}_{j}} &= \sum_{k=1}^c \frac{\partial\mathcal{L}_c}{\partial y_{k}^d} \frac{\partial y_{k}^d}{\partial \tilde{y}_{j}} \\
& = \sum_{k=1}^c \frac{\partial\mathcal{L}_c}{\partial y_{k}^d} y_{k}^d(\delta_{k=j} - y_{j}^d) \\
& = \sum_{k=1}^c \delta_{k=j}y_{k}^d\frac{\partial\mathcal{L}_c}{\partial y_{k}^d} - y_{ij}^d \sum_{k=1}^c y_{k}^d\frac{\partial\mathcal{L}_c}{\partial y_{k}^d} \\
& = y_{j}^d\frac{\partial\mathcal{L}_c}{\partial y_{j}^d} - y_{j}^d \sum_{k=1}^c y_{k}^d\frac{\partial\mathcal{L}_c}{\partial y_{k}^d}
\,.
\label{}
\end{align}

Denote $g_j = y_{j}^d\frac{\partial\mathcal{L}_c}{\partial y_{j}^d}$, then we have:
\begin{align}
\sum_{k=1}^c y_{k}^d\frac{\partial\mathcal{L}_c}{\partial y_{k}^d} &= \sum_{k=1}^c g_k = \vec{g}^T\vec{1} \,, \\
\frac{\partial\mathcal{L}_c}{\partial \tilde{y}_{j}}  &= g_j - \vec{g}^T\vec{1} y_{j}^d \,, \\
\sum_{j=1}^c\frac{\partial\mathcal{L}_c}{\partial \tilde{y}_{j}} &= \sum_{j=1}^cg_j - \vec{g}^T\vec{1} \sum_{j=1}^cy_{j}^d \\
& = \vec{g}^T\vec{1} - \vec{g}^T\vec{1} (\vec{y}^d)^T\vec{1}
\,.
\label{}
\end{align}

We know $(\vec{y}^d)^T\vec{1} = 1$ because $\vec{y}^d$ is a label distribution, thus $\sum_{j=1}^c\frac{\partial\mathcal{L}_c}{\partial \tilde{y}_{j}} = 0$. Then, we get the following proposition:
\begin{proposition}
The sum of all dimensions in the gradient of $\mathcal{L}_c$ with respect to $\vec{y}_d$ is zero.
\label{proposition:1}
\end{proposition}

In Section~\ref{sec:pencil_intr} we mentioned the difference between KL- and inverse KL-divergence, now we discuss their suitability as the classification loss function in PENCIL based on the above derivation.
\subsubsection{Case 1}
When the classification loss is the KL-divergence, we have:
\begin{align}
\frac{\partial \mathcal{L}_c}{\partial y_{j}^d} = 1 + \log\frac{y_{j}^d}{f_j(\vec{x}; \vec{\theta})}
\,.
\end{align}

Then,

\begin{align}
g_j &= y_{j}^d + y_{j}^d\log\frac{y_{j}^d}{f_j(\vec{x}; \vec{\theta})} \,,\\
\vec{g}^T\vec{1} &= \sum_{j =1}^c y_{j}^d + \sum_{j =1}^c y_{j}^d\log\frac{y_{j}^d}{f_j(\vec{x}; \vec{\theta})} \\
& = 1 + \mathcal{L}_c
\,.
\end{align}

Substituting Eq.~20 and Eq.~22 into Eq.~16 we get the following result:
\begin{align}
\frac{\partial\mathcal{L}_c}{\partial \tilde{y}_{j}} &= y_{j}^d + y_{j}^d\log\frac{y_{j}^d}{f_j(\vec{x}; \vec{\theta})} - (1 + \mathcal{L}_c)y_j^d\\
& = y^d_j\left(\log\frac{y_{j}^d}{f_j(\vec{x}; \vec{\theta})} - \mathcal{L}_c\right)
\,.
\label{24}
\end{align}
\subsubsection{Case 2}
When the classification loss is the inverse KL-divergence, we have:
\begin{align}
\frac{\partial \mathcal{L}_c}{\partial y_{j}^d} = \frac{f_j(\vec{x}; \vec{\theta})}{y_{j}^d}
\,.
\end{align}

Then,

\begin{equation}
 g_j = -y_{j}^d\frac{f_j(\vec{x}; \vec{\theta})}{y_{j}^d} = -f_j(\vec{x}; \vec{\theta})
\,.
\end{equation}

Substituting Eq.~26 into Eq.~16 we get the following result:
\begin{align}
\frac{\partial\mathcal{L}_c}{\partial \tilde{y}_{j}} &= -f_j(\vec{x}; \vec{\theta}) + f_j(\vec{x}; \vec{\theta})^T\vec{1}y^d_j\\
& = y^d_j - f_j(\vec{x}; \vec{\theta})
\,.
\label{28}
\end{align}

Next we compare Eq.~\ref{24} and Eq.~\ref{28}. We see that if  $y^d_j$ is almost zero, the value of Eq.~\ref{24} is also almost zero but the value of Eq.~\ref{28} is a negative value which depends on the prediction of the network. This difference tells us that the KL-divergence is not suitable for noise correction, but our proposed inverse KL-divergence is.

When we use the original KL-divergence, if the original noisy label is wrong, the value corresponding to the correct label $y_{correct}^d$ is almost zero, then the gradient of it is almost zero, too. Therefore we cannot correct the wrong original label (i.e., cannot successfully increase $y_{correct}^d$). Finally we also cannot get the correct label distribution. So the original KL-divergence is not suitable. But when we use the inverse KL-divergence, we can successfully update the $\vec{y}^d$ and may get the correct label distribution.

\subsection{Entropy Loss}
Obviously, when the prediction $f(\vec{x};\vec{\theta})$ is the same as the label distribution $\vec{y}^d$, the network will stop updating. However, $f(\vec{x};\vec{\theta})$ tends to approach $\vec{y}^d$ fairly quickly, because label distributions are used as the supervision signal for learning network parameters $\vec{\theta}$. Following~\cite{Tanaka2018CVPR_Daiki}, we add an additional loss (regularization) term to avoid this problem. The entropy loss can force the network to peak at only one category rather than being flat because the one-hot distribution has the smallest possible entropy value. This property is advantageous for classification problems. The entropy loss is defined as
\begin{align}
\mathcal{L}_e(f(\vec{x};\vec{\theta})) = -\frac{1}{n}\sum_{i=1}^n\sum_{j=1}^c f_j(\vec{x}; \vec{\theta})\log f_j(\vec{x}; \vec{\theta}) \,.
\end{align}
At the same time, it also helps avoid the training from being stalled in our PENCIL framework, because the label distribution is not going to be a one-hot distribution and then $f(\vec{x};\vec{\theta})$ will be different from $\vec{y}^d$.

\subsection{The Overall PENCIL Framework}

With all components ready, the PENCIL loss function is
\begin{gather}
\mathcal{L} = \frac{1}{c}\mathcal{L}_c(f(\vec{x};\vec{\theta}),\vec{Y}^d)+\alpha \mathcal{L}_{o}(\hat{\vec{Y}},\vec{Y}^d)+\frac{\beta}{c} \mathcal{L}_e(f(\vec{x};\vec{\theta})) \,, \nonumber
\label{9}
\end{gather}
in which $\alpha$ and $\beta$ are two hyperparameters. Using this loss function and the PENCIL framework's architecture in Fig.~\ref{fig:1}, we can use \emph{any} deep neural network as the backbone network in Fig.~\ref{fig:1}, and then equip it with the PENCIL component to handle learning problems with noisy labels. The relationship between variables and loss functions are clearly visualized in Fig.~\ref{fig:1} as arrows. Forward computations are visualized by red solid arrows, while back-propagation computations are visualized as blue dashed arrows. The algorithmic description of the PENCIL framework is shown in Algorithm~\ref{A1}.

\begin{algorithm}[t]
\caption{The proposed PENCIL framework}
\textbf{Input:} the noisy training set \{$\vec{x}_i$, $\hat{\vec{y}}_i$\} $(1 \le i \le n)$, and the number of training epochs $T$
\begin{algorithmic}[1]
\item initialize $\tilde{\vec{y}}_i$ $(1 \le i \le n)$ by Eq.~\ref{8}
\item $t \leftarrow 1$

\While {$t \le T$}
 \State update $\vec{\theta}$ and $\vec{y}_i^d$ ($1 \le i \le n$) by forward computation and backward propagation in the mini-batch fashion using all $n$ training examples (i.e., to finish one epoch)
 \State $t \leftarrow t+1$
\EndWhile
\end{algorithmic}
\textbf{Output:} the trained network model $\vec{\theta}$, and the noise corrected labels $\vec{y}^d_i$ $(1 \le i \le n)$.
\label{A1}
\end{algorithm}

We want to add two notes about PENCIL. First, the error back-propagation process in PENCIL is pretty straightforward. For example, it can be done automatically in deep learning packages that support automatic gradient computation. Second, after the network has been fully trained (cf. Section~\ref{sec:exp}), those PENCIL-related components in Fig.~\ref{fig:1} are \emph{not needed} at all---the backbone network alone can perform prediction for future test examples.

Similar to \cite{Tanaka2018CVPR_Daiki}, we implement our PENCIL training through 3 steps.

\textbf{Backbone learning:}  We firstly train the backbone network with a large fixed learning rate from scratch without noise handling. As aforementioned, it is observed that when the learning rate is high, a DNN often does not overfit the label noise. Therefore, in this step, we use a fixed high learning rate with only the cross-entropy loss function in Eq.~\ref{1}. The resulted DNN is the backbone network in Fig.~\ref{fig:1}.

\textbf{PENCIL learning:} Then, we use the PENCIL framework to update both network parameters and label distributions. The learning rate is still a fixed high value. Therefore, the network will not overfit label noise and the label distributions will correct noise in the original labels. At the end of this step, we obtain a label distribution vector for every image. Algorithmic details are shown in Algorithm~\ref{A1}. Note that in practice we find that updating $\tilde{\vec{y}}$ requires a learning rate that is much larger than that used for updating other parameters. Because the overall learning rate is fixed in this step, we simply use one single hyperparameters $\lambda$ to update $\tilde{\vec{y}}$ (i.e., do not use PENCIL's overall learning rate), as
\begin{equation}
 \tilde{\vec{y}} \leftarrow \tilde{\vec{y}} - \lambda \frac{\partial \mathcal{L}}{\partial \tilde{\vec{y}}} \,.
\end{equation}

\textbf{Final fine-tuning:} Lastly, we use the learned label distributions to fine-tune the network using only the classification loss $\mathcal{L}_c$ (i.e., $\alpha=\beta=0$). In this step, the label distributions will not be updated and the learning rate will be gradually reduced as in common neural network training.

\subsection{Attention Structure for Multi-label Classification}

When we extend the problem from single-label to multi-label, the complexity of the problem increased a lot. Therefore, the original network structure together with sigmoid loss function is too simple to handle this situation. Our PENCIL framework corrects noisy labels based on the predictions provided by the backbone network. Therefore, a better backbone network is very important.

We propose a simple attention structure to replace the global average pooling layer in the original backbone network. The overall framework of this attention structure is shown in Fig.~\ref{fig:2}.

\begin{figure}[t]
 \centering
 \includegraphics[width=0.9\linewidth]{attention.pdf}
 \caption{The proposed attention structure for multi-label classification. This component can replace the global average pooling layer in arbitrary backbone network.}
\label{fig:2}
\end{figure}

This structure firstly increases the feature dimensionality (e.g., from 512 to 2048) by a $1\times 1$ conv layer. This high dimensional feature can encode more information. Then, we change the feature dimensionality to the number of classes via  another $1\times 1$ conv layer. For this output (i.e., a $7\times 7 \times \mathrm{\#Classes}$ activation map), we regard one dimension as corresponding to one category. With this one-to-one correspondence, each category has a unique feature matrix (i.e., a $7\times 7$ matrix), with each of the 49 values corresponding to predictions based on different sub-regions in the original input image. We can incorporate these predictions from different sub-regions by combining these 49 values to compute one probability for the corresponding category. Therefore we use a depthwise conv layer to compute one value corresponding to one category individually for each dimension. Then we use the sigmoid function to normalize these values. This final result is our final prediction.

In this proposed component, each feature matrix only \emph{pays attention to} one specific category, hence we call this component a (spatial) \emph{attention structure}.

\subsection{Repetitive Training}
\label{sec:R2}
The proposed PENCIL framework corrects noisy labels based on the predictions of the last epoch. The initial backbone model has great impact to the PENCIL learning process. However, the initial backbone network is only trained with a fixed learning rate. The model obtained by our PENCIL framework is much better than the model obtained by the first step of the PENCIL framework. So we can use it to replace the model obtained by backbone learning. Next we process the PENCIL learning and the final fine-tuning again~\cite{R2D2_AAAI_2020}. The above process is called repetitive training. We can even repeat this process multiple times to achieve better performance.

After using repetitive training, the whole PENCIL framework includes one time of backbone learning, and multiple times of PENCIL learning and final fine-tuning. It is worth noting that different from~\cite{R2D2_AAAI_2020}, we just used the final model as the backbone model but do not use the final label distributions to initialize the $\vec{y}^d$. Because if some labels are still wrong in the final label distributions, the errors may accumulate through the repetitive training. We want to avoid the rapid propagation of impacts of these wrong labels.

\section{Experiments} \label{sec:exp}
We tested the proposed PENCIL framework on both synthetic and real-world datasets: CIFAR-100~\cite{Krizhevsky2009TR_CIFAR}, CIFAR-10~\cite{Krizhevsky2009TR_CIFAR},  CUB-200~\cite{Wah2011CIT_CUB_200_2011} and Clothing1M~\cite{Xiao2015CVPR_Clothing1M}. And we also tested it on multi-label datasets without \emph{or} with noisy label: MS-COCO~\cite{MSCOCO} and Open Images~\cite{OpenImages}. All experiments were implemented using the PyTorch framework.

\subsection{Datasets}

\textbf{CIFAR-100:} Following~\cite{Zhang2018NIPS_Lq}, we retained 10\% of the training data as the validation set, and \emph{both} train and validation sets were noise contaminated. However, we did \emph{not} use the validation set in our method, because PENCIL \emph{does not need a validation set}.

There are two types of noises: symmetric and asymmetric. Following~\cite{Zhang2018NIPS_Lq}, in the symmetric noise setup, label noise is uniformly distributed among all categories, and the label noise percentage is $r \in [0,1]$. For every example, if the correct label is $i$, then the noise-contaminated label has $1-r$ probability to remain correct, but has $r$ probability to be drawn uniformly from the $c$ labels. The asymmetric noise label was generated by flipping each class to the next class circularly with noise rate $r \in [0,1]$.

\textbf{CIFAR-10:} Following~\cite{Tanaka2018CVPR_Daiki}, we retained 10\% of the CIFAR-10 training data as the validation set and modify the original correct labels to obtain different noisy label datasets. The setting for symmetric noise is the same as that in CIFAR-100. As for asymmetric noise, following \cite{Patrini2017CVPR_Forward} the noisy labels were generated by mapping \texttt{truck} $\rightarrow$ \texttt{automobile}, \texttt{bird}$\rightarrow$ \texttt{airplane}, \texttt{deer} $\rightarrow$ \texttt{horse} and \texttt{cat} $\leftrightarrow$ \texttt{dog} with probability $r$. These noise generation methods are in coincidence with confusions that often happen in the real world.

\textbf{Clothing1M:} Clothing1M is a large-scale dataset with noisy labels. It consists of more than one million images from 14 classes with many wrong labels. Images were obtained from several online shopping websites and labels were generated by their surrounding texts. The estimated noise level is roughly 40\%~\cite{Xiao2015CVPR_Clothing1M}. This dataset is seriously imbalanced and the label mistakes mostly happen between similar classes (i.e., asymmetric). There exist additional training, validation and test sets with 50k, 14k and 10k examples whose labels are believed to be clean, respectively.

\textbf{CUB-200:} We tested the robustness of our framework in a fine-grained classification dataset CUB-200. CUB-200 contains 11788 images of 200 species of birds, which is not considered to have the noisy label difficulty. Therefore, we tested our framework on this dataset to show that PENCIL is robust.  In addition, there is probably a small percentage of noisy labels in CUB-200~\cite{Wilber2015ICCV_NoisyLabelCUB}. It is interesting to observe whether PENCIL is robust and effective in such a dataset.

\textbf{MS-COCO:} MS-COCO is originally collected for object detection tasks, and it's also widely used for the multi-label classification task. MS-COCO consists of 122218 images from 80 classes which are common in real world. It contains two subsets: a training set with 82081 images and a validation set with 40137 images. We will use it to show the effectiveness of the proposed attention structure.

\textbf{Open Images:} Open Images is a dataset collected from real world. It consists of about 9 million images annotated with image-level labels, object bounding boxes, object segmentation masks, and visual relationships. It can be used as a multi-label classification dataset with noisy labels. Each image has two kinds of labels: human-verified and machine-generated. It can be considered that the human-verified labels are almost correct and the machine-generated labels contain various levels of noise. But the number of the former is much less than that of the latter ($27.9$M human-verified vs. $78.9$M machine-generated labels). Thus, if we use this dataset, we have to make use of these machine-generated labels to ensure enough training data. We randomly selected a subset of 250 categories from this dataset for our experiments. On this dataset, we show the effectiveness of our proposed PENCIL framework and attention structure.

\subsection{Implementation Details}\label{sec:setup}
Next, we describe more implementation details for each dataset.

\textbf{CIFAR-100:} We used ResNet-34~\cite{He2016CVPR_ResNet} as the backbone network for fair comparison with existing methods. The learning rate was $0.35$, $\alpha=0.1$, $\beta=0.4$, and $\lambda=10000$. Mean subtraction, horizontal random flip and $32 \times 32$ random crops after padding 4 pixels on each side were performed as data preprocessing and augmentation. We used SGD with $0.9$ momentum, a weight decay of $10^{-4}$, and batch size of $128$. Following~\cite{Tanaka2018CVPR_Daiki}, the epoch numbers for three steps were 70, 130 and 120, respectively. In the last step, we used the learning rate of $0.2$ and divided it by 10 after 40 and 80 epochs~\cite{Tanaka2018CVPR_Daiki}. All experiments on CIFAR-100 used the same settings as described above. In fact, we can obtain better results by further tuning the hyperparameters (e.g., as what we will soon introduce for CIFAR-10). However, we choose to use the same set of hyperparameters to demonstrate the robustness of our framework.

\textbf{CIFAR-10:} We used PreAct ResNet-32~\cite{He2016ECCV_ResnetV2} as the backbone network for fair comparison with existing methods. We used the same settings as those for CIFAR-100, except the overall learning rate, $\alpha$, $\beta$ and $\lambda$ hyperparameters. On CIFAR-10, these hyperparameters are shown in Table~\ref{tb1}.

\begin{table}
\centering
\footnotesize
\caption{Hyperparameters for CIFAR-10 experiments. 3000 $\rightarrow$ 0 means that $\lambda$ decreases from 3000 to 0 linearly.}
\begin{tabular}{c|cccc}
\hline
\multicolumn{5}{c}{Symmetric Noise} \\
\hline
noise rate (\%)& 	learning rate& 	$\alpha$& 	$\beta$& 	$\lambda$\\
\hline
10	& 	0.02	& 	0.1	& 	0.8	& 	200	\\
30	& 	0.03	& 	0.1	& 	0.8	& 	300	\\
50	& 	0.04	& 	0.1	& 	0.8	& 	400	\\
70	& 	0.08	& 	0.1	& 	0.8	& 	800	\\
90	& 	0.12	& 	0.1	& 	0.4	& 	1200\\
\hline
\hline
\multicolumn{5}{c}{Asymmetric Noise}\\
\hline
noise rate (\%)& 	learning rate& 	$\alpha$& 	$\beta$& 	$\lambda$\\
\hline
10	& 	0.06	& 	0.1	& 	0.4	& 	600	\\
20	& 	0.06	& 	0.1	& 	0.4	& 	600	\\
30	& 	0.06	& 	0.1	& 	0.4	& 	600	\\
40	& 	0.03	& 	0\phantom{.0} & 	0.4	& 	3000 $\rightarrow$ 0	\\
50	& 	0.03	& 	0\phantom{.0} & 	0.4	& 	4000 $\rightarrow$ 0\\
\hline
\end{tabular}
\label{tb1}
\end{table}

As shown in Table~\ref{tb1}, the learning rate increases as the noise rate increases for symmetric noise. This is reasonable, because when noise rate gets higher, we need stronger robustness and we can increase the learning rate to prevent our network from overfitting. And, when the noise rate is very high (e.g., 50\% asymmetric), there are too many noisy labels. Hence, we can remove the effect of noisy labels by removing $\mathcal{L}_o$ (i.e., set $\alpha$ to 0). At the same time, we require a large $\lambda$ to correct these noisy labels quickly. However, after a few epochs, the noisy labels were quickly corrected to a stable state (cf. Fig.~\ref{fig:2} and Fig.~\ref{fig:3}). Hence, we need to decrease $\lambda$ linearly to prevent wrong updates in later epochs.

\textbf{CUB-200:} On this dataset, we used ResNet-50~\cite{He2016CVPR_ResNet} pre-trained on ImageNet. Data preprocessing and augmentation is also applied, including performing mean subtraction, horizontal random flip, resizing the image to $256 \times 256$ and $224 \times 224$ random crops. We used SGD with $0.9$ momentum, a weight decay of $10^{-4}$, and batch size of $16$. The number of epochs for the three steps are 35, 65 and 60, respectively. The learning rate of the first and second step is $2 \times 10^{-3}$. In the last step, the learning rate is $10^{-3}$ and divided by 10 after 20 epochs and 40 epochs. $\beta$ is 0.8 and we reported results for different values of $\alpha$ and $\lambda$ as ablation studies.

\textbf{Clothing1M:} We used ResNet-50 pre-trained on ImageNet as the backbone network for fair comparison with existing methods. Data preprocessing and augmentation are the same as those in CUB-200. We used SGD with $0.9$ momentum, a weight decay of $10^{-3}$, and batch size of $32$. The epoch numbers of the three steps are 5, 10 and 10, respectively. The first step learning rate is $1.6 \times 10^{-3}$ and the second step learning rate is $8 \times 10^{-4}$. The last step learning rate is $5 \times 10^{-4}$ and divided by 10 after 5 epochs. $\alpha = 0.08$, $\beta = 0.8$. In first 5 epochs of second step $\lambda = 3000$, and in last 5 epochs of second step $\lambda = 500$.

This dataset exists serious data imbalance. Therefore, we randomly selected a small balanced subset (using the noisy labels) to relieve the difficulty caused by imbalance. The small subset includes about 260k images and all classes have the same number of images. All our experiments on Clothing1M were done with this subset in this study. However, note that this subset is not truly balanced, because the labels are noisy.

\textbf{MS-COCO:} On this dataset, we used efficient-b0 to -b5~\cite{tan2019_efficientnet} as our backbone networks. We compared the performance of backbone networks with and without our attention structure on this dataset. The input size of different networks followed their official setting. For convenience, data preprocessing and augmentation just include mean subtraction and horizontal random flip. We used SGD with $0.9$ momentum, a weight decay of $10^{-4}$, and batch size of $24$, the number of epochs is 60. The initial learning rate is $0.01$ and divided by 10 every 20 epochs. In addition, the loss function is changed from cross entropy to binary cross entropy for multi-label classification.

\textbf{Open Images:} We randomly selected a subset of 250 categories from Open Images. Every category has about 500 training images, 100 validation images and 50 test images, respectively. On this dataset, the number of human-verified labels is much smaller than that of machine-generated ones. So we used the machine-generated labels which contain various levels of noise to ensure enough images in our training set. But our validation and test set only used the images with human-verified labels.

We used efficientnet-b0 as our backbone network. Data preprocessing and augmentation are the same as those in CUB-200. We used RMSprop following the common practice on this dataset, with $0.9$ momentum, $0.9$ alpha, a weight decay of $4 \times 10^{-5}$, and the batch size of $128$. In the baseline, the number of epochs is $90$. The learning rate is $0.1$ and multiplied by $0.94$ every $2$ epochs. In baseline with attention structure, the number of epochs is also $90$. The basic learning rate is $0.05$ and also multiplied by $0.94$ every $2$ epochs. In addition, the learning rate of our attention structure is $5$ times the basic learning rate. In PENCIL framework with attention structure, $\alpha = 0.2$, $\beta = 0.2$ and $\lambda = 300$. The epoch numbers of the three steps are 35, 65 and 60, respectively. The learning rate of first and second steps is $5 \times 10^{-3}$. In the last step, the learning rate is also $5 \times 10^{-3}$ and multiplied by $0.94$ every $2$ epochs. On this dataset, the machine-generated labels are probability values (i.e., $0, 0.1, 0.2, ..., 1$). We used them and the human-verified labels as our noisy ground-truth labels directly. All the loss functions are changed to binary version (e.g., inverse binary KL-divergence). In addition, we used both inverse binary KL-divergence and binary KL-divergence in our PENCIL framework.

\subsection{Experiments on CIFAR-100}
\begin{table*}
\centering
\footnotesize
\caption{Results on CIFAR-100. We reported the average accuracy and standard deviation of 5 trials. $\#1$ to $\#5$ are quoted from \cite{Zhang2018NIPS_Lq}. PENCIL ($\#6$) is the result of last epoch (without using the validation set). The row with a star \textbf{*} ($\#2$) did not participate in comparison for fairness.}
\begin{tabular}{@{\,}c@{\,}|c|cccc|cccc}
\hline
$\#$ & method & \multicolumn{4}{c|}{Symmetric Noise} & \multicolumn{4}{c}{Asymmetric Noise} \\
\hline
 & noise rate (\%) & 20 & 40 & 60 & 80 & 10 & 20 & 30 & 40 \\
\hline
1 & Cross Entropy Loss & 58.72$\pm$0.26& 48.20$\pm$0.65& 37.41$\pm$0.94& 18.10$\pm$0.82& 66.54$\pm$0.42& 59.20$\pm$0.18& 51.40$\pm$0.16& 42.74$\pm$0.61\\

2 & Forward $T$ \textbf{*}~\cite{Patrini2017CVPR_Forward}& {63.16$\pm$0.37}& {54.65$\pm$0.88}& {44.62$\pm$0.82}& {24.83$\pm$0.71}& {71.05$\pm$0.30}& {71.08$\pm$0.22}& {70.76$\pm$0.26}& {70.82$\pm$0.45}\\

3 & Forward $\hat{T}$~\cite{Patrini2017CVPR_Forward} & 39.19$\pm$2.61& 31.05$\pm$1.44& 19.12$\pm$1.95& \phantom{0}8.99$\pm$0.58& 45.96$\pm$1.21& 42.46$\pm$2.16& 38.13$\pm$2.97& 34.44$\pm$1.93\\

4 & $\mathcal{L}_q$~\cite{Zhang2018NIPS_Lq} & 66.81$\pm$0.42& 61.77$\pm$0.24& 53.16$\pm$0.78& 29.16$\pm$0.74& 68.36$\pm$0.42& 66.59$\pm$0.22& 61.45$\pm$0.26& 47.22$\pm$1.15\\

5 & Trunc $\mathcal{L}_q$~\cite{Zhang2018NIPS_Lq} & 67.61$\pm$0.18& 62.64$\pm$0.33& 54.04$\pm$0.56&\textbf{29.60$\pm$0.51}& 68.86$\pm$0.14& 66.59$\pm$0.23& 61.87$\pm$0.39& 47.66$\pm$0.69\\
\hline
6 & PENCIL ($last$) & \textbf{73.86$\pm$0.34}& \textbf{69.12$\pm$0.62}& \textbf{57.79$\pm$3.86} & fail & \textbf{75.93$\pm$0.20}& \textbf{74.70$\pm$0.56}& \textbf{72.52$\pm$0.38}&  \textbf{63.61$\pm$0.23}\\
\hline
\end{tabular}
\label{tb2}
\end{table*}

Firstly we tested PENCIL on  CIFAR-100. The results are shown in Table~\ref{tb2}. All dataset settings followed~\cite{Zhang2018NIPS_Lq}. The method ``Forward $T$~\cite{Patrini2017CVPR_Forward}'' used the ground-truth noise transition matrix (which is not available in real-world datasets), hence its numbers were not compared with other methods. Except for the 80\% symmetric noise case, PENCIL significantly outperformed previous methods in all symmetric and asymmetric noise cases. Even if ``Forward $T$'' used strong prior information which should not have been used, our PENCIL method still outperformed it in most cases.

As for the 80\% symmetric noise case, it revealed a \emph{failure mode} of the proposed PENCIL method. When the noise rate is too high (e.g., 80\%), the correct labels only form a minority group and they are too weak to bootstrap the noise correction process. Hence, PENCIL tends to fail in such high noise rate problems. Fortunately, we hardly deal with such high noise rate in real-world applications. For example, the large scale real-world image dataset JFT300M~\cite{Sun2017ICCV_JFT300M} only includes about 20\% noisy labels.

We have intentionally chosen the same set of hyperparameters in all experiments on this dataset, and the results demonstrate the \emph{robustness} of our PENCIL framework to these hyperparameters. We can obtain better accuracy by using different hyperparameters for different noise rate and noise type, as shown in Table~\ref{tb1} on the CIFAR-10 dataset.

\subsection{Experiments on CIFAR-10}

Next, we evaluated the performance of our PENCIL framework on CIFAR-10. All the settings have been described in Section~\ref{sec:setup}. On the original noise-free CIFAR-10 dataset, the result of our backbone network (PreAct ResNet-32) is $94.05\%$. Our setup followed that in~\cite{Tanaka2018CVPR_Daiki}. However, results in~\cite{Tanaka2018CVPR_Daiki} used a prior knowledge (i.e., all categories have the same number of noise-free training examples), which should not be used. For fair comparison, we implemented the ``Tanaka \emph{et al.}~\cite{Tanaka2018CVPR_Daiki}'' method and in our implementation we did not use this prior knowledge.

Table~\ref{tb3} lists results of symmetric noise for CIFAR-10. In Table~\ref{tb3}, ``$best$'' denotes the test accuracy of the epoch where the validation accuracy was optimal and ``$last$'' denotes the test accuracy of the last epoch. As aforementioned, when the learning rate is small, the deep neural network's accuracy will decline because the network memorizes all the (noisy) labels, i.e., the network is overfitting. As shown in row $\#1$, the traditional neural network using the classic cross entropy loss is heavily affected by this difficulty. Its $best$-epoch test accuracy was significantly better than that of the $last$-epoch one. And, as the noise rate increased, the gap was even larger because the overfitting to noise became more serious as expected. On the contrary, our method and the Tanaka~\textit{et~al.}~\cite{Tanaka2018CVPR_Daiki} did not have obvious accuracy drop between $best$- and $last$-epochs. Therefore, the proposed PENCIL method has strong robustness. As for the test set accuracy, PENCIL had a clear advantage than competing methods in Table~\ref{tb3}. The winning gap became especially apparent when the noise rate increased to larger values. For example, when the noise rate was 90\%, PENCIL obtained roughly 7\% higher accuracy than that of Tanaka~\textit{et~al.} and 10\% higher than that of cross entropy.

\begin{table}
\centering
\footnotesize
\caption{Test accuracy on CIFAR-10 with symmetric noise. We reported the average result of 5 trials. All results in this table were based on our own implementation.}
\setlength{\tabcolsep}{3.5pt}
\begin{tabular}{c|c|c|ccccc}
\hline
$\#$ &method	&		\multicolumn{6}{c}{Symmetric Noise}\\
\hline
&noise rate (\%)	& &	10	&	30	&	50	&	70	&	90	\\
\hline
\multirow{2}{*}{1}&\multirow{2}{*}{Cross Entropy Loss}	&	$best$	&	91.66	&	89.00	&	85.15	&	78.09	&	50.74	\\
&&$last$	&	88.43	&	72.78	&	53.11	&	33.32	&	16.30	\\
\hline
\multirow{2}{*}{2}&\multirow{2}{*}{Tanaka~\textit{et~al.}~\cite{Tanaka2018CVPR_Daiki}}&	$best$	&	93.23	&	91.23	&	88.50	&	84.51	&	54.36	\\
&&	$last$	&	93.23	&	91.22	&	88.51	&	84.59	&	53.49	\\
\hline
\multirow{2}{*}{3}&\multirow{2}{*}{PENCIL}&	$best$	&	{93.26}	&	{92.09}	&	{90.29}	&	87.10	&\textbf{61.21}	\\
&&	$last$	&\textbf{93.28}	&\textbf{92.24}	&\textbf{90.36}	&\textbf{87.18}	&	60.80	\\
\hline
\end{tabular}
\label{tb3}
\end{table}

Table~\ref{tb4} lists results of asymmetric noise for CIFAR-10. In terms of robustness, methods shown in row $\#1$, $\#2$ and $\#3$ had the overfitting problem and their test accuracies had large gaps between the $best$- and $last$-epochs. The Tanaka~\textit{et~al.} method experienced the same issue when the noise rate was high (50\%), but was robust in other cases. Our PENCIL method, however, remained robust throughout all the experiments.

\begin{table}
\centering
\footnotesize
\caption{Test accuracy on CIFAR-10 with asymmetric noise. We reported the average result of 5 trials. Rows $\#1$, $\#4$ and $\#5$ were based on our own implementation. Rows $\#2$ and $\#3$ were quoted from \cite{Tanaka2018CVPR_Daiki}. The methods marked with a ``*'' used additional information that should not be used, and need to be excluded in a fair comparison.}
\setlength{\tabcolsep}{3.5pt}
\begin{tabular}{c|c|c|ccccc}
\hline
$\#$ & method	&		\multicolumn{6}{c}{Asymmetric Noise}\\
\hline
&noise rate (\%)	& &	10	&	20	&	30	&	40	&	50	\\
\hline
\multirow{2}{*}{1}&\multirow{2}{*}{Cross Entropy Loss}	&$best$	&	91.09	&	89.94	&	88.78	&	87.78	&	77.79	\\
&&$last$	&	85.24	&	80.74	&	76.09	&	76.12	&	71.05	\\
\hline
\multirow{2}{*}{2}&\multirow{2}{*}{Forward $T$ *~\cite{Patrini2017CVPR_Forward}}	&$best$	&	92.4\phantom{0}	&	91.4\phantom{0}	&	91.0\phantom{0}	&	90.3\phantom{0}	&	83.8\phantom{0}	\\
&&$last$	&	91.7\phantom{0}	&	89.7\phantom{0}	&	88.0\phantom{0}	&	86.4\phantom{0}	&	80.9\phantom{0}	\\
\hline
\multirow{2}{*}{3}&\multirow{2}{*}{CNN-CRF *~\cite{Vahdat2017NIPS_CNNCRF}}	&	$best$	&	92.0\phantom{0}	&	91.5\phantom{0}	&	90.7\phantom{0}	&	89.5\phantom{0}	&	84.0\phantom{0}	\\
&&	$last$	&	90.3\phantom{0}	&	86.6\phantom{0}	&	83.6\phantom{0}	&	79.7\phantom{0}	&	76.4\phantom{0}	\\
\hline
\multirow{2}{*}{4}&\multirow{2}{*}{Tanaka~\textit{et~al.}~\cite{Tanaka2018CVPR_Daiki}}&	$best$	&	92.53	&	91.89	&	91.10	&	91.48	&	75.81	\\
&&	$last$	&	92.64	&	91.92	&	91.18	&	\textbf{91.55}	&	68.35	\\
\hline
\multirow{2}{*}{5}&\multirow{2}{*}{PENCIL}&	$best$	&	93.00	&	\textbf{92.43}	&	\textbf{91.84}	&	91.01	&	\textbf{80.51}	\\
&&	$last$	&	\textbf{93.04}	&\textbf{92.43}	&	91.80	&	91.16	&	80.06	\\
\hline
\end{tabular}
\label{tb4}
\end{table}

The Forward~\cite{Patrini2017CVPR_Forward} and CNN-CRF~\cite{Vahdat2017NIPS_CNNCRF} methods both require the ground-truth noise transition matrix, which is hardly available in applications. Our method does not require any prior information about noise labels. Table~\ref{tb4} shows that PENCIL has been robust and is the overall accuracy winner on CIFAR-10.

We recorded the number of correct labels in PENCIL's second step. In a label distribution vector, the category corresponding to the maximum value in the probability distribution was identified as the label estimated by PENCIL. If this label was the same as the noise-free ground-truth label, we say it was correct. The results for 70\% symmetric and 30\% asymmetric noise on CIFAR-10 are shown in Fig.~\ref{fig:2} and Fig.~\ref{fig:3}, respectively. We can observe that PENCIL effectively and stably estimated correct labels for most examples even with high noise rates. For example, with 70\% symmetric noise rate, originally only about 16000 labels were correct, but after PENCIL's learning process there are about 39000 correct labels.

\begin{figure}
 \centering
 \includegraphics[width=0.9\linewidth]{SN70}
 \caption{Correct labels on CIFAR-10 with 70\% symmetric noise.}
 \label{fig:3}
\end{figure}

\begin{figure}
 \centering
 \includegraphics[width=0.9\linewidth]{AN30}
 \caption{Correct labels on CIFAR-10 with 30\% asymmetric noise.}
 \label{fig:4}
\end{figure}

\subsection{Analyzing the Two Classification Losses}
In this section, we compared the performance of KL- and inverse KL-divergence in PENCIL using the CIFAR-10 dataset. We observed the value and the gradient of $\vec{y}^d$ with original and inverse KL-divergence respectively and analysed the trends of them.

First of all, we observed the value of $\vec{y}_d$ directly. We turn the vector $\vec{y}_d$ into a label by finding its maximum value.  As shown in Figure~\ref{fig:3} and Figure~\ref{fig:4}, we can see that the labels are corrected successfully when we used the inverse KL-divergence. However, when we used the original KL-divergence following the same setting, we see that the number of correct labels is unchanged and the curve of it is a horizontal line (figures not shown). Therefore, the original KL-divergence is not suitable for noise correction in PENCIL.

Next, we randomly selected some images from CIFAR-10 with 30\% symmetric noise to observe the gradient of $\vec{y}^d$. We consider two cases: when the original label is wrong (Figure~\ref{fig:5}) or correct (Figure~\ref{fig:6}).

\subsubsection{When the Original Label Is Wrong}

In Figure~\ref{fig:5}, the blue curve represents the component in the gradient of $\vec{y}_d$ corresponding to the original label (which is incorrect), and the orange curve is for the correct label. The left two figures ($a$) and ($c$) show the results of two different input images with inverse KL-divergence. In these figures, we can see the blue curves are almost positive while the orange curves are almost negative. Therefore the labels are continuously corrected in PENCIL. We can see that both the correct and the original labels are successfully updated towards their desired values during training. In figures ($b$) and ($d$), which used the original KL-divergence, the magnitudes of the gradient (about $10^{-6}$) are much smaller than those in the left two figures (about $10^{-3}$). It is too small to correct the noisy labels. It's obvious that the original KL-divergence is not suitable in correcting label noise.
\begin{figure}[t]
 \centering
 \includegraphics[width=0.975\linewidth]{SN30_clsF_noise3.pdf}
 \caption{The component in the gradient of $\vec{y}^d$ corresponding to the correct label (solid orange curve) and the original label (blue dashed curve) when the original label is wrong. Figure ($a$) and ($c$) used the inverse KL-divergence. Figure ($b$) and ($d$) used the original KL-divergence. The top and bottom rows represent two different input images.}
\label{fig:5}
\end{figure}

\subsubsection{When the Original Label Is Correct}

In this case we want the labels to remain unchanged. In other words, we want the gradient of $\vec{y}^d$ as small as possible. As shown in Figure~\ref{fig:6}, we can see the magnitudes of the left two figures are much smaller than those in the right two figures. Therefore the performance of the inverse KL-divergence is better than the original KL-divergence again.

Through the above two cases, we can conclude that the original KL-divergence is not suitable in our PENCIL framework, but our proposed inverse KL-divergence is.

\begin{figure}[t]
 \centering
 \includegraphics[width=0.975\linewidth]{SN30_clsF_clean3.pdf}
 \caption{The component in the gradient of $\vec{y}^d$ corresponding to the correct (orignial) label. Figure ($a$) and ($c$) used the inverse KL-divergence. Figure ($b$) and ($d$) used the original KL-divergence. The top and bottom rows represent two different images.}
\label{fig:6}
\end{figure}

\subsection{Repetitive Training on CIFAR-10}
We tested PENCIL with repetitive training on CIFAR-10 and reported results on the teest set of each iteration in the repetitive training process. All the settings of baseline is the same as the description in Section~\ref{sec:setup}. In repetitive training, the learning rate and the $\lambda$ will be reduced slightly according to the number of iterations.

The results are shown in Table~\ref{tb5}. All the results are the \emph{last}-epoch accuracy. We just run the whole repetitive training process once, therefore there are some small differences on the values of accuracy between Table~\ref{tb5}, Table~\ref{tb3} and Table~\ref{tb4}.

As shown in Table~\ref{tb5}, we can see the accuracy always achieve best performance after repeating twice or thrice. The accuracy can be sorted as: baseline $<$ repeat once $<$ repeat twice $\approx$ repeat thrice. In addition, the performance of repetitive training with high level noise is close to or even better than baseline with low level noise (e.g., the accuracy of repeating once with 40\% asymmetric noise is $91.68\%$, which is better than the baseline with 30\% asymmetric noise of $91.46\%$).

\begin{table}
\centering
\footnotesize
\caption{Test set accuracy on CIFAR-10 with symmetric and asymmetric noise. We reported results of each iteration in the repetitive training process.}
\begin{tabular}{c|c|cccc}
\hline
\multicolumn{6}{c}{Symmetric Noise} \\
\hline
$\#$ &noise rate (\%)& 	10 & 	30& 	50& 	70\\
\hline
1 &baseline                   & 93.50	   & 	92.13	& 	90.46	&  86.95\\
2 &repeat once              & 	93.73	& 	92.37	& 	91.46	& 	87.98\\
3 &repeat twice              & 	\textbf{94.08}	& 	93.04	& 	91.64	& 	88.91\\
4 & repeat thrice           & 	93.78	& 	\textbf{93.28}	& 	\textbf{92.21}	& 	\textbf{89.40}\\
\hline
\hline
\multicolumn{6}{c}{Asymmetric Noise}\\
\hline
$\#$ & noise rate (\%)& 	10& 	20& 	30& 	40\\
\hline
6 &baseline                   & 93.50	   & 	92.64	& 	91.46	&  91.02\\
7 &repeat once              & 	93.22	& 	92.90	& 	91.42	& 	\textbf{91.68}\\
8 &repeat twice              & 	\textbf{93.58}	& 	92.97	& 	\textbf{92.15}	& 	91.64\\
9 &repeat thrice            & 	93.55	& 	\textbf{93.35}	& 	92.01	& 	91.53\\
\hline
\end{tabular}
\label{tb5}
\end{table}

\subsection{Experiments on CUB-200}

We performed additional experiments on CUB-200 with different hyperparameters $\alpha$ and $\lambda$. This dataset is generally considered to contain no or only few noisy labels. Therefore, we use it to further test the robustness of PENCIL on problems not affected by noisy labels.

The results are listed in Table~\ref{tb6}. Row $\#1$ is the baseline (classic method) and rows $\#2$ to $\#7$ are PENCIL results. For a wide range of $\alpha$ and $\lambda$ values, PENCIL consistently exhibited competitive results (i.e., without obvious degradation). Furthermore, we observed the final label distributions, and the maximum values of all label distributions are correct (i.e., same as the correct labels). This observation shows that PENCIL works robustly in clean datasets, too.

In the settings of rows $\#4$ to $\#7$, PENCIL achieved higher accuracy than the baseline. In particular, row $\#4$ is 0.71\% higher. A small percentage of label noise may exist in this dataset~\cite{Wilber2015ICCV_NoisyLabelCUB}. Our hypothesis is that by replacing the original one-hot label with probabilistic modeling in PENCIL, we obtained better robustness and consequently a small edge in accuracy.

\begin{table}
\footnotesize
\centering
\caption{Test accuracy on CUB-200 with different hyperparameters. The accuracy of PENCIL does not decline in standard datasets with clean labels.}
\begin{tabular}{c|p{1.5cm}<{\centering}p{1.5cm}<{\centering}|c}
\hline
$\#$ & \multicolumn{2}{c|}{method} & Test Accuracy (\%)\\
\hline
1& \multicolumn{2}{c|}{Cross Entropy Loss} & 81.93\\
\hline
\hline
&\multicolumn{2}{c|}{PENCIL}&\\
\hline
& $\lambda$& $\alpha$ & \\
\hline
2&1000&0\phantom{.0}&81.91\\
\hline
3&2000&0\phantom{.0}&81.84\\
\hline
4&3000&0\phantom{.0}&\textbf{82.64}\\
\hline
5&1000&0.1&82.09\\
\hline
6&2000&0.1&82.21\\
\hline
7&3000&0.1&82.22\\
\hline
\end{tabular}
\label{tb6}
\end{table}

\subsection{Experiments on MS-COCO}

Next, we show the effectiveness of our proposed attention structure in networks with different sizes on the MS-COCO dataset. We compared the performance of networks with and without the proposed attention structure. Efficientnet is suitable for our evaluations because it has versions with different sizes to test our proposed attention structure comprehensively.

The results are shown in Table~\ref{tb7}. In all the rows, the performance of networks with the proposed attention structure outperforms those without it. Even on efficientnet-b5 which is a large scale network, our attention structure still achieved 6.56 percentage points mAP higher than baseline network.

In Table~\ref{tb7}, the result of the network with attention structure in row $\#1$ ($68.06$) is better than the networks without our attention structure in rows $\#2$ ($62.48$) and $\#3$ ($65.61$), and even close to the network without the attention structure in row $\#4$ ($68.62$). However, the input image's size of efficientnet-b0 (224) is smaller than efficientnet-b1 (240), b2 (260) and b3 (300). The same holds for their model sizes. That means the computing cost of the former one is much less than the latter three. In comparison, the extra computing cost of our attention structure is small.

\begin{table}
\caption{Performance of networks with different sizes with or without the proposed attention structure on the MS-COCO dataset. We selected efficientnet-b0 to -b5~\cite{tan2019_efficientnet} as our backbone networks to show the effectiveness of our attention structure comprehensively.}
\footnotesize
\centering
\begin{tabular}{c|c|c|c}
\hline
$\#$&backbone network & \multicolumn{2}{c}{mAP}\\
\hline
 &  & w/o attention & w/ attention \\
\hline
1& efficientnet-b0 & 60.26 & \textbf{68.06}\\
\hline
2& efficientnet-b1 & 62.48 & \textbf{69.56}\\
\hline
3& efficientnet-b2 & 65.61 & \textbf{71.09}\\
\hline
4& efficientnet-b3 & 68.62 & \textbf{73.48}\\
\hline
5& efficientnet-b4 & 70.97 & \textbf{76.33}\\
\hline
6& efficientnet-b5 & 72.21 & \textbf{78.77}\\
\hline
\end{tabular}
\label{tb7}
\end{table}

\subsection{Experiments on a Subset of Open Images}
Then we tested our PENCIL framework on a subset of Open Images, which is a large scale real-world multi-label dataset with noisy labels. Because PENCIL corrects the noisy labels based on the backbone network, we need a backbone network with a reasonable starting point. Therefore we need to combine both PENCIL and the attention structure.

The results are shown in Table~\ref{tb8}. Same as the results on MS-COCO, the performance of baseline with attention structure is much better than baseline without attention structure. Therefore our attention structure is still effective on the multi-label dataset with noisy labels. In row $\#3$, when  combining PENCIL and the attention structure, we obtain the best performance. That means our proposed PENCIL framework is also effective on multi-label tasks.

\begin{table}
\caption{Performance of backbone network with or without the proposed attention structure and PENCIL with attention structure on a subset of the Open Images dataset.}
\footnotesize
\centering
\begin{tabular}{c|c|c}
\hline
$\#$&method & mAP\\
\hline
1& Baseline & 62.55\\
\hline
2& Baseline w/ attention structure & 75.11\\
\hline
3& PENCIL w/ attention structure & \textbf{77.13}\\
\hline
\end{tabular}
\label{tb8}
\end{table}

\subsection{Experiments on Clothing1M}
Finally, we tested PENCIL on Clothing1M, which is a real-world noisy label dataset. It includes a lot of unknown structure (asymmetric) noise.

The results are shown in Table~\ref{tb9}. All results are $best$ test accuracy. Rows $\#1$ and $\#2$ were quoted from \cite{Patrini2017CVPR_Forward}, and row $\#3$ was reported in \cite{Tanaka2018CVPR_Daiki}. Although these baseline models were trained on the whole Clothing1M training set, our PENCIL used a randomly sampled pseudo-balanced subset, including about 260k images. The backbone network was ResNet-50 for all methods.

\begin{table}
\caption{Test accuracy on the Clothing1M dataset. Rows $\#1$ and $\#2$ were quoted from \cite{{Patrini2017CVPR_Forward}} and $\#3$ was quoted from \cite{Tanaka2018CVPR_Daiki}. These baseline methods used the complete Clothing1M training data, but our method only used a small pseudo-balanced subset (i.e., balanced in terms of noisy labels). Our method achieved state-of-the-art result in this real-world dataset.}
\footnotesize
\centering
\begin{tabular}{c|c|c}
\hline
$\#$&method & Test Accuracy (\%)\\
\hline
1& Cross Entropy Loss & 68.94\\
\hline
2& Forward~\cite{Patrini2017CVPR_Forward} & 69.84\\
\hline
3& Tanaka~\textit{et~al.}~\cite{Tanaka2018CVPR_Daiki} & 72.16\\
\hline
4& PENCIL & \textbf{73.49}\\
\hline
\end{tabular}
\label{tb9}
\end{table}

In Table~\ref{tb9}, only noisy labeled examples were used (i.e., without using the clean training subset). The Forward~\cite{Patrini2017CVPR_Forward} method required the ground-truth noise transition matrix, which is not available. Hence, it used an estimated matrix instead. The Tanaka~\textit{et~al.}~\cite{Tanaka2018CVPR_Daiki} method used the distribution of noisy labels to relieve the imbalanced problem. In our PENCIL method, we did not use any extra prior information. PENCIL achieved 1.33\% higher accuracy than that of Tanaka~\textit{et~al.}~\cite{Tanaka2018CVPR_Daiki}, 3.65\% higher than Forward~\cite{Patrini2017CVPR_Forward} and 4.55\% than cross entropy.

\section{Conclusion}
We proposed a framework named PENCIL to solve the noisy label recognition problem in deep leaning. PENCIL adopted label probability distributions to supervise network learning and to update these distributions through back-propagation end-to-end in every epoch. We proposed an inverse KL-loss, which is different from previous methods but is robust for noisy label handling, then we show that the inverse KL-loss is indeed more suitable than the original KL-loss. The proposed PENCIL framework is end-to-end and independent of the backbone network structure, thus it is easy to deploy. Then we find that repetitive training of PENCIL can achieve better performance. Finally we extend PENCIL to multi-label classification tasks with our proposed attention structure.

We tested PENCIL with synthetic label noise on CIFAR-100 and CIFAR-10 with different noise types and noise rates, and outperformed current state-of-the-art methods by large margins. On CIFAR-10, we also show the effectiveness of repetitive training and the inverse KL-loss is more suitable than the original KL-loss. We also experimented on CUB-200, which is considered to be noise free. The results show that PENCIL is robust for different datasets and hyperparameters. Then we evaluated our proposed attention structure on multi-label dataset MS-COCO with different networks sizes, which show our attention structure is effective in multi-label classification. Next, We tested PENCIL with attention structure on a subset of Open Images which is a large scale real-world multi-label dataset with noisy labels. The results show the effectiveness of our PENCIL framework with attention structure on multi-label dataset with noisy labels. Lastly, we tested PENCIL on the real-world large scale single label noise dataset Clothing1M. On this dataset, we achieved 1.33\% higher accuracy than previous state-of-the-art.

\ifCLASSOPTIONcaptionsoff
  \newpage
\fi

\bibliographystyle{IEEEtran}

redpajama_set_name: RedPajamaArXiv


Controls signal outputs when TRCITCTRL.IME is set.
The following figure shows the TRCITMISCOUT bit assignments.
The following table shows the TRCITMISCOUT bit assignments.
[11:8] ETMEXTOUT[3:0] Drives the EXTOUT[3:0] outputs.
The TRCITMISCOUT can be accessed through the internal memory-mapped interface and the external debug interface, offset 0xEDC.


text:
<!DOCTYPE html>
<html>
<head lang="en">
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>ExpirySelect Directive</title>
  <script src="../bower_components/angular/angular.js"></script>
  <script src="../date-elements.min.js"></script>
  <script>
    var app = angular.module('app', ['leodido.dateElements']);
    app.controller('ExpirySelectController', function ($scope) {
      $scope.name = 'abc';
    });
  </script>
</head>
<body ng-app="app" ng-controller="ExpirySelectController">
  <div
    name="{{name}}"
    data-expiry-select
    data-ng-model="exp"
    data-options="{month: {emptyOption: 'Month', selection: 'Feb', format: 'MMM', class: 'myCssClass'}, year: {name: 'y', class: 'myCssClass2', selection: 2015, emptyOption: 'Year', format: 'yy', last: '2018'}}"
    ></div>
    <!-- NOTE: first year ignored; it is always current year -->
</body>
</html>

redpajama_set_name: RedPajamaGithub


El Paso is seeing a beak in the 20 degree weather with a gorgeous 70 degree, sunny day and I intend to take full advantage of it by heading out back with a Illusione Cigares Prive paired with a bottle of Bottle Logic's Number crunch.
The Cigares Prive line used to be a limited production run that Illusione had converted over to a full-production line last year. The line comes with two different wrapper variations, Corojo and Mexican San Andres. Today we focus on the Corojo. Produced at the Tabacos Valle de Jalapa S.A. (TABSA) in Nicaragua the cigars starts with a Corojo wrapper, Nicaraguan binder and Nicaraguan filler. The Cigares Prive comes in 3 different sizes: Robusto (5 x 52), Prensado Suave (5 1/2 x 56), and the Presado 660 (6 x 60). Each format comes packaged in boxes of 25 and run between $7.90 and $8.90 a stick. Big thanks for Fox Cigar Bar for sending a few of these my way. You can order them directly from them here.
The Illusione Cigares Prive starts out with a very worn, rustic looking, beautiful wrapper. I say beautiful cause I love it when cigars look natural. The wrapper features a medium brown color with some scattered splotches of darker brown areas along with some orangish areas mixed it. The texture is semi-toothy with a good amount of oil and some medium-sized veins running through the cigar's body leading up to the round triple-wrapped cap. The cigar is firm and the only soft areas are located around the foot of the cigar. The cigar is polished off with a dark blue version of Illusione's beautifully elegant bands which simply states "Cigares Prive" across the front in white.
The wrapper on the Cigares Prive gives off some bold spice and natural tobacco aromas while the foot carries more mossy, sweet aromas. The cap cut clean and easily using my Palio double bladder cutter. The cold draw produces lots of spice, sweetness, cocoa, and natural tobacco.
The Illusione Cigares Prive starts out with a quick blast of black pepper which quickly fades to release bold spice, cocoa, vanilla, oak, and a bit of moss/dirt on the tail end of the taste. The cigar is starting out fantastically. The draw is effortless and releases a great amount of thick, white smoke which each puff. The burn line is a bit wavy, but not too concerning as it leaves behind a trail of compacted white and dark gray ash which held for about 3/4 of a inch before falling off.
Into the second third of the Illusione Cigares Prive and an interesting pine flavor has really come on in the flavor profile backed by the bold spice, cocoa and vanilla with some light oak and natural mossy flavors. The retrohale brings out a lot of the pepper that I haven't really been picking up in the experience. I close out the second third with nothing really in terms of nicotine.
Into the final third of the Illusione Cigars Prive and the flavors have changed up quite a bit. The main flavor is now the oak and a mixture of oaty/grain with a good amount of vanilla and pine backed by soft cocoa, a bit of light coffee and spice. It took me almost two hours to take this cigar down to the nub and I'm surprised with how slow it burned. I experienced no harshness, nor any extended heat and only a little bit of nicotine.
Again, I am a big Illusione fan and I absolutely loved the Cigares Prive. The flavor profile is complex, bold, and extremely enjoyable while the construction was flawless. Add in the ease of finding these and the low cost per cigar and it's easy to see why these will find their way into my everyday rotation. This was probably one of the more full-bodied cigars I have had in their portfolio and I am still amazed with just how smooth it went down.
Number Crunch is a Bourbon Barrel aged Imperial stout finished with a variety of cacao nibs and vanilla brewed at the Bottle Logic brewery in Anaheim, California. Coming in at a whopping 13.6% ABV this beer packs a punch. The beer starts out very sweet with a whole ton of chocolate and vanilla. It almost tastes like ice cream. It's sticky with very heavy bourbon notes, caramel, and roasted malt with a heavy mouthfeel before finishing even sweeter with more cocoa and vanilla pouring on. The alcohol is completely hidden in this beer. The vanilla and cocoa is what drew my attention to this pairing as the Cigares Prive are heavy on both flavors as well. The beer is BOLD and will overpower most cigars but the Cigares Prive really held its own and never once got overpowered by the body of the beer.


Along with its four-star SKYTRAX rating, China Southern Airlines has been voted the best airline in China by various international sources. Don't take our word for it – see for yourself!
Flying aboard the new long-haul fleet of Airbus A330, A380 and Boeing 787 aircrafts offers you a huge variety of movies, TV series, documentaries, games, music and audio books, while the friendly cabin crew ensure a comfortable journey.
Frequent business flyers can benefit from SkyTeam's flexible advantages on international travel with China Southern Airlines, offering free access to 516 lounges all over the world and an extensive worldwide network of destinations.
China Southern Airlines is the largest airline in Asia and the world's sixth-largest airline measured by passengers carried. The airline has held its Four-Star status from SKYTRAX, the world's most prestigious evaluation organization, since 2011. China Southern Airlines have been honoured with the Diamond Flight Safety Award in China, thus becoming the leading Chinese carrier to maintain the highest safety records.
Book China Southern Airlines flights to Beijing, Guangzhou, Bangkok and other exciting places!
Free baggage allowances vary depending on the destination flown to/from.
First Class: 3 bags - 32kg each.
Business Class: 2 bags - 32kg each.
Premium Economy: 2 bags - 23kg each.
First Class: 2 bags - 5kg each.
Other classes: 1 bag - 5kg.
For the complete baggage allowance guide please check the airline's website.
The overall dimensions of each checked-in bag shall not exceed 158cm, while it shall not exceed 115cm for hand baggage.
You can get your boarding pass and choose your seat by checking in online from the comfort of your home, from 24 hours and up to 1 hour before your flight, depending on the destination.
You can also use airport self-service kiosks to check in, choose your preferred seat and print your boarding pass yourself, where available.
Alternatively, you can check in at the regular counters, which open 3 hours before departure, closing 60 minutes prior to departure at most airports.
Offering basic seats with comfortable legroom and entertainment options depending on the aircraft type.
Offering more space and legroom and better cabin service.
Featuring seats reclining into flat beds, with more private space, wider screens and high-class amenities and services.
Featuring private suites with fully-flat beds, wider screens and the finest services available to premium passengers.
Offering 2000 daily flights to 207 destinations in 40 countries and regions across the world, China Southern offers access to 1024 destinations in 187 countries across the world with its SkyTeam member airlines.


text:
Over the almost four decades he has been working in fluid dynamics, Werner Dahm has yet to see a lull in its upward trajectory as a springboard for progress in science, engineering, technology and the growth of industry.
The vigor of the field as a driver of innovation in developing potential solutions to some of the world's most critical challenges was recently on display at the 74th Annual Meeting of the American Physical Society's Division of Fluid Dynamics, or APS DFD, in Phoenix.
"This was one of the best APS DFD meetings I've attended in the past 40 years," says Dahm, the Arizona State University Foundation Professor of Aerospace and Mechanical Engineering, leader of the Laboratory for Turbulence, Combustion, and Propulsion, and founding director and chief scientist for ASU's Security and Defense Systems Initiative.

redpajama_set_name: RedPajamaCommonCrawl


text:
Biomimetic Propulsion System Powers Record-Breaking Submarine
The IMarEST's European International Submarine Races 2018, held July 3-13 at QinetiQ's Ocean Basin in Gosport, U.K., have concluded, with the Canadian École de Technologie Supérieure team taking the winning trophy with their submarine, Omer 11, and breaking the world speed record. The runners-up came from universities in the Netherlands and the U.S.
The overall winner of the eISR must combine excellence in engineering design, manufacture, performance, reliability and reporting. In submarine racing, the name Omer is synonymous with all of these. The students from Montreal brought innovation to a new level this year, combining several iterations of racing success into a single world record-breaking submersible speedster.
Omer 11 is the first human-powered submarine to be equipped with interchangeable propeller and non-propeller propulsion systems. The team started the week with a standard two-bladed propeller, then moved to an even more efficient single-bladed prop to clinch the top speed of the competition by a male pilot. Finally they swapped in their biomimetic propulsion system, which was inspired by the wings of a penguin or sea turtle, and with a speed of 4.94 kts, beat the world speed record, which had stood since 2009, in the non-propeller division. View a full list of teams and participants at IMarEST.
tagged with biomimetic propulsion system, Canadian École de Technologie Supérieure, engineering design, European International Submarine Races, Gosport, IMarEST, interchangeable propulsion systems, Montreal, Netherlands, non-propeller propulsion system, Omer 11, penguin, QinetiQ, sea turtle, submersible speedster, U.K., U.S., world record, world speed record


Washer-Dryer Features Explained - Which?
Get to grips with the main programs and features you'll find when you're shopping for a washer-dryer.
Below, we explain the key features to consider when choosing a washer-dryer, plus a range of helpful extras that some machines offer and you may like to look out for.
Once you've decided what type of washer-dryer is right for you, go straight to our washer-dryer reviews to find our expert impartial verdict on more than 100 of the latest models.
All washer-dryers offer a suite of standard washing programs, which includes settings for cottons, man-made materials and delicate fabrics at a range of different temperatures.
Most models have additional options, such as pre-wash, spin and extra rinse, and some have special cycles designed for specific scenarios, such as hand-wash, curtains, sportswear or even trainers.
Washer-dryers are all of a very similar size on the outside, but their washing capacities vary between 4.5kg and 7kg of laundry, and they can dry 2.25kg to 5kg. An extra kilogram equals around five T-shirts or one large sheet.
Bigger isn't necessarily better when it comes to capacity. If you don't fill up the drum, the larger washer-dryers can be less efficient, because they use so much water in each wash. If you wash or dry with a full load in a larger machine, you use less energy, water and time per item.
All washer-dryers have a filter, which catches fluff whipped up when drying, as well as bits of debris. They need to be cleaned regularly, which can be a pain on some models because they are often close to the ground and can be fiddly to open.
The condenser in a washer-dryer cools moist, hot air inside the machine, producing water which pours down the drain.
This process uses a lot of water itself, making some washer-dryers a poor choice for people with a water meter.
Washer-dryers are rated between A+ and G for energy efficiency, with A+ being the most efficient. Although the ratings officially go down to G, in practice you won't see new machines below C in the shops.
We assess the energy and water use of every washer-dryer we test in the Which? test lab, and give each one a star rating for energy efficiency, as well as an estimate of how much each will cost to run.
To compare washer-dryers for energy use and running costs, go to our washer-dryer reviews.
Most machines have a quick cycle of 50 to 70 minutes (standard washes are usually nearer two hours) which work best for clothes that aren't too dirty.
This is a gentler program for cleaning wool-mixture clothes, which can become bobbly on normal programs.
Only items marked as 'pure new wool' and 'washable', 'pre-shrunk' or 'non-matting' should be washed using the wool program.
You should wash all other types of wool with a handwash program or by hand.
We've found that there can be little difference between a load washed on an easy-iron program and one washed on a standard cycle.
Using lower spin speeds can help to reduce creases.
This is intended for garments made of delicate materials such as cashmere, angora and silk.
Like the delicates program, it washes clothes more gently than the normal setting, so they don't snag, matt or shrink.
Many washer-dryers have 'fuzzy-logic' sensors that automatically vary the wash program according to how big or dirty the load is.
Bioprofile, biostage or protein eliminators are just some of the names different companies call these stain-removing features. Aquatronic wash, 3D AquaSpa, Showerwize, Power Jet and other similar terms refer to features that spray water to soak clothes and distribute detergent more effectively.
All these features vary in how effective they are. One fuzzy-logic system might be much more sophisticated than another. So the safest option is just to check out our washer-dryer reviews to see how each one scores in our tough cleaning tests.
Many washer-dryers are poor at rinsing, so the ability to add an extra rinse can sometimes ensure more detergent is removed from laundry. This is useful for people with eczema, sensitive skin or allergies.
If you have a problem with undissolved detergents, consider switching to liquid detergents, particularly for quick-wash programs.
This lets you vary the spin speed of the machine to protect woollens, handwash-only clothes and delicate items from damage.
It's useful if the default spin speed for your machine is higher than 1,000rpm.
This allows you to set the washer-dryer to automatically start in a few hours time, which is useful if you want the machine to work while you are out of the house, or overnight. Some washer-dryers with this feature let you set the machine up to 24 hours in advice, while others only allow smaller delays.
If you use delayed start at night, fit a smoke alarm nearby.
This helpful feature lets you check which stage of washing or drying a program has reached and how long it has left to run.


In this Odoo Technical Tutorial, I would like to explain how to add colors in tree view odoo.
Sometime, it's required to show the odoo tree view lines record in different colours to easy identification.like green colour for indicate approved record, red colour for expectation record. We will show how can we change list view lines colours based on condition.
It means using decoration with success will return green color and decoration with danger will return red color.


Nikki is a woman on an incredible mission to save dogs that few others would. Based in San Diego, California, Road Dogs & Rescue works together with their 501c3 partner For Pete Sake Foundation to rescue dogs from the shelter and neglectful situations in the greater Los Angeles area. Visit them on FaceBook and Instagram to see the pups they save and donate if you feel that little tug at your heart to do so – you're helping a dog on the down and out find his/her FURever home.