text:
Join our team Investors en
Leadership -	Photobox Group - 05 May 2015
Photobox Group strengthens Board with major appointment
Douglas McCallum appointed as Non-Executive Chairman
London – Tuesday 5th May 2015 – Photobox Group, parent of the Photobox, Moonpig, PaperShaker, Sticky9 and Hofmann brands, is pleased to announce the appointment of the former Senior Vice President and General Manager of eBay Europe, Douglas McCallum, as Non-Executive Chairman of the Board. This appointment strengthen the organisation and reflect the Group's ambitious growth plans as it seeks to consolidate its foothold in the personalisation market, enter new markets and broaden its innovative product offering.
Commenting on the appointments, Photobox Group President & Chief Executive Officer, Stan Laurent said: "An international digital pioneer, Douglas will play a critical role in strengthening our governance and our Board experience. His track record in developing digital businesses organically as well as through international M&A matches perfectly our Group strategy. I have no doubt that his extensive experience, at the very highest level, will prove to be invaluable as we strive towards joining the billion-dollar club."
Douglas brings more than 20 years of experience as an investor and global digital business leader. He is currently Chairman of trainline.com and is also a non-executive director of Ocado Plc, where he chairs the company's remuneration committee. Douglas scaled eBay in the UK from 350,000 users to 15 million in just a few short years. In 2008, he took on EMEA, which had 50% of eBay's global revenues, where he devised and implemented a turnaround strategy that enabled the business to win back market share and grow eBay Europe to a $16bn business. Douglas is also a member of the Cabinet Office's Digital Advisory Board and is President and Founder of eBay for Charity.
Douglas commented:"I have always had something of a passion for personalisation and in many ways have made my career in the internet for over twenty years now leading the charge to making affordable personalisation a reality for consumers. Today it is a large and fast growing market, but one in which only the most innovative businesses thrive.The Photobox Group of companies are all ambitious innovators, continually developing new and exciting products that their customers, including me, love to buy and share, so I'm truly delighted to have been asked to join as Chairman. Photobox is already the market leader in several countries across Europe, but there remains huge potential for it to grow further and faster, innovating and disrupting its way into becoming a very large business; I am greatly looking forward to helping it succeed in this mission."
Brand: PhotoBox
Enter your email address to stay up-to-date on our news
Trafficking and Slavery Statement
© PhotoBox Group 2015


The star made the announcement through her friend, journalist Jessica Yellin.
The actress and comedian posted a photo on Instagram showing her and husband Chris Fischer's heads edited on to the bodies of the Duke and Duchess of Sussex, who is also expecting, with a message directing fans to the page of journalist Jessica Yellin.
She wrote: "About to announce some exciting news on @jessicayellin insta page. Please follow her for up to the minute #newsnotnoise she breaks down what's really going on.
Schumer's friend Yellin, of the site NewsNotNoise.org, said in a video on her own Instagram page: "I wanted to share some news from our community, maybe it's noise but it's happy noise.
"So I'm not in the business of making voting recommendations, but these are the recommendations of Amy Schumer, one of the most consistent and earliest supporters of News Not Noise.
She then shared a screen grab which included a note from Schumer saying "I'm pregnant" on a list of her candidate recommendations for voters for the upcoming midterm elections in the US.
Schumer, 37, married chef Fischer in February.


text:
\section{Introduction}
Label Smoothing (LS) is a commonly used output distribution regularization technique to improve the generalization performance of deep learning models~\cite{Chorowski2016TowardsBD,Huang2019GPipeET,structurels,Real2019RegularizedEF,Szegedy2016RethinkingTI,Vaswani2017AttentionIA,8579005}.
Instead of training with data associated with one-hot labels, models with label
smoothing  are trained on samples with  soft targets, where each target is
a weighted mixture of the ground truth one-hot label  with the uniform distribution of the classes.
This penalizes overconfident output distributions, resulting in improved model generalization~\cite{Lukasik2020DoesLS,MullerKH19,DBLP:conf/iclr/PereyraTCKH17,yuan2019revisit,DBLP:conf/aaai/ZhuJZGHSZ20}.

\iffalse
\begin{figure} [h]
\caption{Illustration of label smoothing (left) and pairwise label smoothing (right) on how the distribution mass of the one-hot ground truth label(s) (in blue bars) are distributed over the non- ground truth targets (in red pattern bars) for a training sample; orange  solid bars are the resulting smoothed ground truth targets.
 Y-axis indicates the distribution mass of the  four target classes (X-axis) of the training sample.
 }
\label{fig:ill}
\centering
\includegraphics[width=3.316in]{mass2.png}
\end{figure}
\fi

\begin{figure}[h]
\caption{
Average  distribution mass (Y-axis) of PLS and Mixup (from sample pairs with  different labels) and ULS on Cifar10's 10 classes (X-axis).
 PLS' class-wise training signals  are  spread out,  with  gold labels probabilities  close to 0.35, which has  larger  uncertainty than that of 0.5 and 0.9 from Mixup and ULS, receptivity.
 }
\label{fig:trainsignal}
\centering
\includegraphics[
width=4.25972i
]{trainsignal2.png}
\end{figure}

\begin{figure}[h]
\caption{
Histograms of the winning softmax scores on Cifar10 validation data, generated by  ULS  (right Y-axis) and by Mixup and PLS (left  Y-axis) where the X-axis depicts the softmax score. The PLS produces very low softmax winning scores; the differences versus the ULS and Mixup  are striking: the PLS  models are extremely sparse  in the high confidence region at the right end.}
\label{fig:pilot}
\centering
\includegraphics
width=3.972i
]{pilot8.png}
\end{figure}

When smoothing the one-hot training signal, existing LS methods, however, only consider the distance between the only gold label and the  non-ground truth targets.
This motivates our Pairwise Label Smoothing (PLS) strategy, which takes  a pair of samples as input.
In a nutshell, the PLS first creates a midpoint sample by averaging the inputs and labels of a sample pair,
and then distributes the  distribution mass of the two ground truth  targets  of the new midpoint sample over its non-ground truth classes.
Smoothing with a pair  of ground truth labels  enables  PLS to  preserve  the relative distance between the two truth labels  while being able to further soften that between the truth labels  and the other class targets.
Also, unlike current LS methods, which typically require to  find a global smoothing distribution mass through cross-validation search, PLS  automatically  learns the   distribution mass for each input pair during training.
Hence, it effectively eliminates the   turning efforts for searching the right level of smoothing strength
for different applications.

Also, smoothing with a pair of labels empowers PLS   to be trained with low  distribution mass (i.e., high uncertainty)  for the ground truth  targets.
Figure~\ref{fig:trainsignal} depicts the average  values  (over all the midpoint  samples created from sample  pairs with different labels) of the 10 target classes in Cifar10 used by PLS for training. These ground truth targets are smaller than 0.35, which represents  much larger  uncertain than the 0.5 and 0.9 in Mixup and the  uniform label smoothing (denoted as ULS), respectively.
Owing to such   high uncertainty ground truth  target training signals,
PLS  produces extremely low winning softmax scores during testing, resulting in very conservative classification decisions.  Figure~\ref{fig:pilot} depicts the histograms of the winning  softmax scores of Mixup, ULS
and PLS on the Cifar10 validation data.
Unlike Mixup and ULS,  PLS  produced  very conservative  softmax scores,  with extremely sparse distribution in  the high confidence region.
Such conservative classifications  are produced not only  on in-distribution data but also on out-of-distribution samples.

We empirically show that
 PLS  significantly
outperforms  LS and  Mixup~\cite{Mixup17},
with up to 30\% of relative classification error reduction.
We  also visualize that PLS produces extremely conservative predictions  in testing time,
with many of its  winning predicted softmax scores  slightly over 0.5 for both in and out of distribution samples.

\section{Related Work}
Label smoothing has shown to  provide consistent model generalization gains across many tasks~\cite{Szegedy2016RethinkingTI,DBLP:conf/iclr/PereyraTCKH17,7780883,MullerKH19,Lukasik2020DoesLS,structurels}.
Unlike the above methods which apply  label smoothing to  each single input, our smoothing strategy leverages a pair of inputs for label smoothing. Furthermore,
unlike existing LS approaches which deploy one static and uniform smoothing distribution for all the training samples, our strategy  automatically learns a dynamic distribution mass for each sample during training.

Mixup methods~\cite{DBLP:journals/corr/abs-1906-06875,DBLP:conf/aaai/Guo20,DBLP:journals/corr/abs-1905-08941,adamixup,DBLP:conf/iclr/TokozumeUH18,cutmix19,Mixup1
}
 create  {\em a large number} of synthetic samples with various features from a  sample pair,  through  interpolating   the pair's   both features and labels  with mixing coefficients randomly sampling  between [0,1]. In contrast,
our approach   creates {\em only the midpoint} sample of a sample pair (equivalent to  Mixup with a fixed  mixing ratio of 0.5).
More importantly, our method adds a label smoothing component on the midpoint samples. That is, we
adaptively learn  a  distribution mass to smooth the pair of ground truth  labels of each midpoint sample,
 with the aims of  deploying   high uncertainty training targets (much higher than that of Mixup) for output distribution regularization. Such  high uncertain training labels  result in PLS producing very conservative classification decisions and superior  accuracy than  Mixup.

The generation of the  smoothing distribution  in our method is also related to  self-distillation~\cite{ahn2019variational,Furlanello2018BornAN,hinton2015distilling,mobahi2020selfdistillation,DBLP:conf/aaai/YangXQY19}. However, these  approaches treat the final predictions as target labels for a new round of training, and  the teacher and student architectures are identical~\cite{mobahi2020selfdistillation}.  In our method, the classification   and  the smoothing distribution  generator   have different network architectures, and the training targets for the classification  are a mixture of the outputs of the two networks.

\section{Label Smoothing over Midpoint Samples}
Our proposed  method PLS leverages a pair of samples,  randomly selected from the training data set, to conduct label smoothing. It first creates a midpoint sample for each sample pair, and then adaptively learns a smoothing distribution for each midpoint sample. These midpoint samples are then used for training to regularize the learning of the networks.
\subsection{Preliminaries}
We consider a standard classification setting with a given training data set $(X; Y)$ associated with $K$ candidate classes $\{1,2,\cdots,K\}$.
For an example $x_i$ from the training dataset $(X; Y)$, we denote the ground truth distribution $q$ over the labels as $q(y|x_i)$ ( $\sum_{y=1}^{K} q(y|x_i) = 1$). Also, we denote  a neural network  model to be trained as $f_{\theta}$ (parameterized with $\theta$), and it produces
a conditional  label distribution over the $K$ classes as $p_{\theta}(y|x_i)$:
\begin{equation}
p_{\theta}(y|x_i) = \frac{\exp(z_{y})}{\sum_{k=1}^{K}\exp(z_{y_{k}})},
\end{equation}
with
 $\sum_{y=1}^{K} p_{\theta}(y|x_i) = 1$, and $z$
is noted as the logit of the model $f_{\theta}$. The logits are generated with two steps:
 the model $f_{\theta}$ first  constructs the $m$-dimensional input embedding $S_{i} \in R^{m}$ for the given input $x_{i}$, and then passes it
  through
a linear   fully connected layer $W_{l} \in R^{K \times m}$:
\begin{equation}

\label{logit}
S_{i} = f_{\theta}(x_{i}),
z=  W_{l}S_{i}.

\end{equation}
During learning, the model $f_{\theta}$ is trained to optimize the parameter $\theta$ using the $n$ examples from  $(X; Y)$ by  minimizing the cross-entropy loss:
\begin{equation}
\ell =- \sum_{i=1}^{n} H_i(q,p_{\theta}).
\end{equation}
Instead of using one-hot encoded vector for each example $x_{i}$ in $(X; Y)$, label smoothing (LS) adds a smoothed label distribution  (i.e., the prior distribution) $u(y|x_{i})$ to each example $x_{i}$, forming a new target label, namely  soft label:
\begin{equation}
\label{oriequ}
q^{\prime}(y|x_i) = (1-\alpha) q(y|x_i) + \alpha u(y|x_i),
\end{equation}
where hyper-parameter $\alpha$ is a weight factor ($\alpha \in [0,1]$) needed to be tuned to indicate the smoothing strength for the one-hot label. This modification results in a new loss:
\begin{equation}
\label{lsloss}
\ell^{\prime}
= - \sum_{i=1}^{n} \Big[ (1-\alpha) H_i(q,p_{\theta}) + \alpha H_i(u,p_{\theta}) \Big].
\end{equation}
Usually, the $u(y|x_{i})$ is a uniform distribution, independent of data $x_{i}$, as $u(y|x_{i}) = 1/K$, and hyper-parameter $\alpha$ is tuned with cross-validation.

\subsection{Midpoint Generation}
\label{plsmethod}
For a sample, denoted as $x_{i}$, from the provided training  set $(X; Y)$ for training, PLS first randomly selects~\footnote{For efficiency purpose, we implement this by randomly selecting a sample from the same mini-batch during training.} another training sample $x_{j}$. For the  pair of samples $(x_{i}; y_{i})$ and $(x_{j} ; y_{j})$, where $x$ is the
input and $y$ the one-hot encoding of the corresponding class, PLS then generates a synthetic sample  through element-wisely averaging both the input features and the labels, respectively, as follows:
\begin{equation}
x_{ij} =  (x_{i} +  x_{j})/2,
\end{equation}
\begin{equation}
\label{equ:2}
q(y|x_{ij}) = ( y_{i} +  y_{j})/2.
\end{equation}
These synthetic samples can be considered as the  {\em midpoint samples} of the original sample pairs. It is also worth noting that, the midpoint samples here are equivalent to fixing the linear interpolation mixing ratios as 0.5 in the Mixup~\cite{Mixup17} method.
Doing so, for the midpoint sample $x_{ij}$ we have the ground truth distribution $q$ over the labels as $q(y|x_{ij})$ ($\sum_{y=1}^{K} q(y|x_{ij}) = 1$).
The newly resulting  midpoint  $x_{ij}$ will  then be  used for label smoothing (will be discussed in detail in Section~\ref{evovinglabel}) before feeding into the networks for training.
In other words, the predicted logits as defined in Equation~\ref{logit} is computed by first generating the  $m$-dimensional input  embedding $S_{ij} \in R^{m}$ for  $x_{ij}$ and then passing through the fully connected linear layer to construct the logit $z$:
\begin{equation}
S_{ij} = f_{\theta}(x_{ij}),
\end{equation}
\begin{equation}
\label{mmix22}
z= W_{l}S_{ij}.
\end{equation}
Hence, the predicted conditional  label distribution over the $K$ classes
produced by the networks is as follows
\begin{equation}
p_{\theta}(y|x_{ij}) = \frac{\exp(z_{y})}{\sum_{k=1}^{K}\exp(z_{y_{k}})}. \end{equation}

\subsection
{Learning Smoothing Distribution for Midpoints}\label{evovinglabel}
PLS leverages a learned distribution,  which depends on the input $x$, to dynamically generate the smoothing distribution mass for midpoint samples to distribute their ground truth target distribution to the non-target classes.
To this end, the PLS implements this by adding a  fully connected layer  to the network $f_{\theta}$. That is,   the
$f_{\theta}$ produces two projections from the {\em penultimate} layer representations of the network:  one for the logits as the original network (Equation~\ref{mmix22}), and another for generating the smoothing distribution as follows.

In specific, an additional  fully connected
layer $W_{t} \in R^{K \times m}$ is added to the original networks $f_{\theta}$ to produce the smoothing distribution over the $K$ classification classes.
That is, for the given
input image $x_{ij}$,  its smoothing  distributions over the $K$ classification  targets, denoted as  $u^{\prime}_{\theta}(y|x_{ij})$, are computed as follows:
\begin{equation}
\label{prelabel}
u^{\prime}_{\theta}(y|x_{ij}) =
\frac{\exp(v_{y})}{\sum_{k=1}^{K}\exp(v_{y_{k}})},
\end{equation}
   \begin{equation}
   \label{sha}
   v = \sigma (W_{t}S_{ij}),
   \end{equation}
where $\sigma$ denotes the Sigmoid function, and
 $S_{ij}$ is the same input  embedding as that in Equation~\ref{mmix22}. In other words, the two predictions (i.e., Equations~\ref{mmix22} and~\ref{sha}) share the same networks except the last fully connected layer.
That is, the only difference between PLS and the original networks is the added  fully connected layer $W_{t}$.
The added Sigmoid function here aims to  squash the smoothing distributions learned for different targets to the same range of  [0, 1].

After having the smoothing distributions $u^{\prime}_{\theta}(y|x_{ij})$, PLS then uses them to smooth the ground truth labels  $q(y|x_{ij})$ as described in Equation~\ref{equ:2}, with   average:
\begin{equation}
\label{merge}
q^{\prime}(y|x_{ij}) = ( q(y|x_{ij}) +  u^{\prime}_{\theta}(y|x_{ij}))/2.
\end{equation}

The loss function of PLS thus becomes the follows:
\begin{equation}
\ell^{\prime}
= - \sum_{i=1}^{n} \Big[ 0.5 \cdot  H_i(q(y|x_{ij}),p_{\theta}(y|x_{ij}))
+  0.5\cdot H_i(u^{\prime}_{\theta}(y|x_{ij}),p_{\theta}(y|x_{ij}))
\Big].
\end{equation}

Coefficient 0.5 here  helps prevent the network from over-distributing its label distribution  to the non-ground truth targets. Over-smoothing  degrades the model performance, as will be shown in the experiments.
Also,
0.5 here   causes the resulting training signals to have high  uncertainty regarding the ground truth targets.
Such high uncertainty  helps train  the models to make conservative predictions.

\subsection{Optimization}
For training, PLS minimizes, with gradient descent on mini-batch, the  loss  $\ell^{\prime}$.
One more issue needs to be addressed for the training. That is, the midpoint samples used for training, namely $(x_{ij}; y)$, may lack  information on the original training samples  $(x_{i}; y)$ due to the average operation in midpoint generation. To compensate this,
we alternatively feed inputs to the networks with either a mini-batch from the original inputs, i.e., ${x}_{i}$ or ${x}_{ii}$,  or a mini-batch from the midpoint samples, i.e., ${x}_{ij}$.
Note that, when training with the former, the networks still  need to learn to assign the smoothing distribution $ u^{\prime}_{\theta}(y|x_{ii})$ to form the  soft  targets $q^{\prime}(y|x_{ii})$ for the sample ${x}_{ii}$.
As will be shown in the experiment  section, this training strategy is   important to PLS'  regularization effect.

\begin{table*}[h]
  \centering
\scalebox{0.9}{
\begin{tabular}{l|c|c|c|c|c|c}\hline
Methods& MNIST&Fashion&SVHN&Cifar10&Cifar100&Tiny-ImageNet\\ \hline
PreAct ResNet-18 &0.62	$\pm$0.05&4.78$\pm$0.19&3.64$\pm$	0.42&5.19$\pm$0.30&24.19$\pm$	1.27&39.71$\pm$0.08\\
ULS-0.1 &0.63$\pm$0.02&4.81$\pm$0.07&3.20 $\pm$0.06&4.95$\pm$0.15&21.62$\pm$0.29&38.85$\pm$0.56\\
ULS-0.2&0.62$\pm$0.02&4.57$\pm$0.05&3.14$\pm$0.11&4.89$\pm$0.11&21.51$\pm$0.25&38.54$\pm$0.32\\
ULS-0.3&0.60$\pm$0.01&4.60$\pm$0.06&3.12$\pm$0.03&5.02$\pm$0.12&21.64$\pm$0.27&38.32$\pm$0.37\\
\hline
Mixup  &0.56$\pm$0.01&4.18$\pm$	0.02&3.37$\pm$0.49&3.88$\pm$	0.32&21.10$\pm$0.21&38.06$\pm$0.29\\
Mixup-ULS0.1 &0.53$\pm$0.02&4.13$\pm$0.10&2.96$\pm$0.39&4.00$\pm$0.17&21.51$\pm$0.51&37.23$\pm$0.48\\
Mixup-ULS0.2 &0.53$\pm$0.03	&4.18$\pm$0.09	&3.02$\pm$0.42&3.95$\pm$0.13&21.41$\pm$0.55&38.21$\pm$0.38\\
Mixup-ULS0.3  &0.50$\pm$0.02	&4.15$\pm$0.06	&2.88$\pm$0.31&4.06$\pm$0.04&$20.94\pm$0.49&38.93$\pm$0.43\\
\hline
PLS &\textbf{0.47 $\pm$0.03}&\textbf{3.96$\pm$0.05}&\textbf{2.68 $\pm$0.09}&\textbf{3.63$\pm$0.10}&\textbf{19.14$\pm$0.20}&\textbf{35.26$\pm$0.10}\\
\hline
\end{tabular}
}
\caption{Error rate (\%) of the testing methods with PreAct ResNet-18~\cite{DBLP:journals/corr/HeZR016} as baseline.  We report mean scores over 5 runs with standard deviations (denoted $\pm$).
Best results are  in \textbf{Bold}.
  }    \label{tab:accuracy:resnet18}
\end{table*}

\begin{table*}[h]
  \centering
\scalebox{0.92}{
\begin{tabular}{l|c|c|c|c|c|c}\hline
Methods& MNIST&Fashion&SVHN&Cifar10&Cifar100&Tiny-ImageNet\\ \hline
ResNet-50 &0.61$\pm$0.05
	&4.55$\pm$0.14	&3.22$\pm$0.05&4.83$\pm$0.30&23.10$\pm$0.62	&35.67$\pm$0.50\\
ULS-0.1 &0.63$\pm$0.02&4.58$\pm$0.16&2.98$\pm$0.02&4.98$\pm$0.25&23.90$\pm$0.99&35.02$\pm$0.39\\
ULS-0.2 &0.62$\pm$0.03&4.52$\pm$0.04&3.08$\pm$0.03&5.00$\pm$0.35&23.88$\pm$0.73&36.19$\pm$0.66\\
ULS-0.3 &0.65$\pm$0.03&4.51$\pm$0.15&3.04$\pm$0.07&5.16$\pm$0.16&23.17$\pm$0.50&36.14$\pm$0.06\\
\hline
Mixup  &0.57$\pm$	0.03&4.31$\pm$0.05	&2.85$\pm$0.07&4.29$\pm$0.28&19.48$\pm$0.48&32.36$\pm$0.53\\
Mixup-ULS0.1  &0.60$\pm$0.04	&4.28$\pm$0.12	&2.90$\pm$0.09&4.02$\pm$0.27&21.58$\pm$0.86&32.11$\pm$0.09\\
Mixup-ULS0.2  &0.58$\pm$0.02	&4.33$\pm$0.09	&2.89$\pm$0.07&4.09$\pm$0.10&20.87$\pm$0.51&32.81$\pm$0.48\\
Mixup-ULS0.3  &0.57$\pm$0.04	&4.29$\pm$0.11	&2.84$\pm$0.10&4.19$\pm$0.18&21.64$\pm$0.41&33.94$\pm$0.56\\
\hline
PLS &\textbf{0.51$\pm$
0.02}&\textbf{4.15$\pm$0.09}&\textbf{2.36$\pm$0.03}&\textbf{3.60$\pm$0.18}&\textbf{18.65$\pm$1.08}&\textbf{30.73$\pm$0.20}\\
\hline
\end{tabular}
}
\caption{Error rate (\%) of the testing methods with  ResNet-50~\cite{DBLP:journals/corr/HeZR016} as baseline.  We report mean scores over 5 runs with standard deviations (denoted $\pm$).
Best results are  in \textbf{Bold}.
  }    \label{tab:accuracy:resnet50}
\end{table*}

\section{Experiments}
We first  show our  method's superior  accuracy, and then visualize its high uncertainty training labels and  conservative classifications.
\subsection{Datasets, Baselines, and Settings}

We use
six benchmark image classification tasks.
\noindent \textbf{MNIST} is a digit (1-10)  recognition dataset with 60,000 training and 10,000 test 28x28-dimensional gray-level images.
\noindent
\textbf {Fashion} is an image recognition dataset with the same scale as
MNIST, containing 10 classes of fashion product pictures.
\noindent
\textbf{SVHN} is the Google street view house numbers recognition data set. It has 73,257 digits, 32x32 color images for training, 26,032 for testing, and 531,131 additional, easier samples. Following literature, we did not use the additional images.
\noindent
\textbf{Cifar10} is an image classification task with  10 classes,
  50,000 training  and 10,000 test samples.
\noindent
\textbf{Cifar100} is similar to Cifar10 but with 100 classes and 600 images each.
\textbf{Tiny-ImageNet}~\cite{ChrabaszczLH17}
has
200 classes, each with 500 training
and 50 test 64x64x3 images.

We conduct experiments  using the popular benchmarking networks  PreAct ResNet-18 and ResNet-50~\cite{DBLP:journals/corr/HeZR016}.
We  compare with the
label smoothing methods~\cite{MullerKH19,Lukasik2020DoesLS} (denoted as ULS) with various smoothing coefficients (i.e., $\alpha$  in Equation~\ref{lsloss}), where ULS-0.1, ULS-0.2, and ULS-0.3  denote the smoothing coefficient of 0.1, 0.2, and 0.3, respectively.
We also compare  with the input-pair based data augmentation method  Mixup~\cite{Mixup17}.
We further compare with methods that stacking the ULS on top of Mixup, denoted  Mixup-ULS0.1, Mixup-ULS0.2, and Mixup-ULS0.3.

For Mixup, we use the authors' code at~\footnote{https://github.com/facebookresearch/mixup-cifar10} and the uniformly selected mixing coefficients between [0,1].
For PreAct ResNet-18 and ResNet-50, we use the  implementation  from Facebook~\footnote{https://github.com/facebookresearch/mixup-cifar10/models/}. For PLS, the added fully connected layer is the same as  the last fully connected layer of the baseline network with a Sigmoid function on the top.
All models  are trained using mini-batched (128 examples) backprop,
 with the  {\em exact settings} as in the Facebook codes,     for 400 epochs.   Each reported  value (accuracy or error rate) is the mean of   five runs  on a NVIDIA GTX TitanX GPU with 12GB memory.

\subsection{Predictive Accuracy}
The  error rates obtained by  ULS, Mixup, Mixup-ULS, and PLS using    ResNet-18  as baseline on the six test datasets are presented in Table~\ref{tab:accuracy:resnet18}. The  results with ResNet-50 as baselines  are provided in Table~\ref{tab:accuracy:resnet50}.

Table~\ref{tab:accuracy:resnet18} shows that PLS outperforms, in terms of predictive error, the  ResNet-18, the label smoothing models (ULS-0.1, ULS-0.2, ULS-0.3),  Mixup, stacking ULS on top of Mixup (Mixup-ULS0.1, Mixup-ULS0.2, Mixup-ULS0.3) on  all the six datasets.
For example, the relative improvement of PLS over ResNet-18   on  Cifar10 and MNIST
are over 30\% and 24\%, respectively.
When considering PLS and the average error obtained by the three ULS models,
on both  Cifar10 and MNIST, the relative improvement is over 23\%.
It is also promising to see that on  Tiny-ImageNet,  PLS  reduced the absolute error rates over  Mixup, Mixup-ULS, and ULS from about 38\% to 35\%.

For the cases with ResNet-50 as baselines,
Table~\ref{tab:accuracy:resnet50}  indicates that similar error reductions are obtained by PLS. Again, on all the  testing datasets, PLS outperforms all the comparison baselines. For example, for  Cifar10, the relative improvement achieved by PLS over   ResNet-50 and the average of the three label smoothing strategies (i.e. ULS) are 25.47\% and 28.67\%, respectively.
For  Cifar100 and Tiny-ImageNet, PLS reduced the absolute error rates of ULS from  23\% and 36\% to 18.65\% and 30.73\% respectively

An important observation here is that,
 stacking label smoothing on top of Mixup (i.e., Mixup-ULS) did not improve, or even degraded, Mixup's  accuracy. For example, for Cifar10, Cifar100, and Tiny-ImageNet (the last three columns in  Tables~\ref{tab:accuracy:resnet18} and~\ref{tab:accuracy:resnet50}),     Mixup-ULS obtained similar or slightly higher error rate than  Mixup. The reason here is that  Mixup creates samples with soft labels through linearly interpolating between [0, 1], which is a form of label smoothing regularization~\cite{carratino2020mixup}. Consequently,  stacking  another label smoothing regularizer on   top of Mixup can easily mess up the soft training targets,  resulting in underfitting.
  Promisingly,  PLS  was able to improve over Mixup and Mixup-ULS. For example, on  Tiny-ImageNet PLS outperformed Mixup and Mixup-ULS by reducing the error  from 38.06\% to 35.26\% and from 32.36\% to 30.73\%, respectively, when using  ResNet-18 and ResNet-50. Similar error reduction can be observed on  Cifar10 and Cifar100.

\subsection{Ablation Studies}
\textbf{Impact of Learned Distribution and Original  Samples}
\label{ablation}
We   evaluate the impact of the key components in  PLS using  ResNet-18 and ResNet-50 on Cifar100. Results are in Table~\ref{aaaablation}. The key components include  1) removing the  learned smoothing distribution mass in Equation~\ref{merge},  2) excluding the original training inputs as discussed in the method section, and 3) replacing the learned smoothing distribution mass with uniform distribution with weight coefficients of 0.1, 0.2, and 0.3 (denoted  UD-0.1, UD-0.2 and UD-0.3).

The error rates obtained in  Table~\ref{aaaablation} show that,
  both  the learned smoothing distribution  and  the original training samples are critical for  PLS.
In particular, when excluding the original  samples  from  training, the predictive error of PLS  dramatically increased from about 19\% to nearly 24\% for both  ResNet-18 and ResNet-50. The reason,  as discussed in the method section,
 is that,
without the original training samples, the networks are trained with midpoint samples only, thus may lack  information on the  validation samples with one-hot labels.

\begin{table}[h]
  \centering
  \begin{tabular}{l|r|r}\hline
 \multirow{2}{*}{PLS }& ResNet-18&ResNet-50 \\
  &19.14&18.65 \\ \hline
----  no learned distribution  &21.06&19.35\\
----  no original  samples  &23.84&24.42\\
 \hline
----  UD 0.1&19.50&18.91\\
----  UD 0.2&19.25&18.81\\
----  UD 0.3&19.31&18.89
\\
\hline
\end{tabular}
  \caption{Error rates (\%) on Cifar100 by PLS  while varying its key components: no learned  distribution, no original  samples, replacing learned smoothing distribution with uniform  distribution.
\label{aaaablation}}

\end{table}

Also,  Table~\ref{aaaablation} indicates that,  replacing the learned smoothing distribution  in PLS with manually tuned Uniform distribution  (i.e., UD) obtained slightly larger errors. This  indicates that  PLS  is able to  learn the  distribution  to smooth the two target labels in the midpoint samples, resulting in superior  accuracy and excluding the need for the  coefficient search  for different  applications.

\textbf{Learning Smoothing Coefficient}
We also conducted experiments of  learning the smoothing coefficient, instead of learning the smoothing distribution, for each midpoint example. That is,  we replace Equation~\ref{sha}  with $\sigma(W_{t} S_{ij})$ where $W_{t}$ is $R^{1 \times m}$ instead of $R^{K \times m}$. Table~\ref{coefficientpredictin} presents the error rates (\%) obtained using PreAct ResNet-18 and ResNet-50 on Cifar100 and Cifar10.
These results suggest that learning to predict smoothing distribution significantly better than predicting the  smoothing coefficient in PLS.

\begin{table}[h]
  \centering
  \begin{tabular}{l|r|r}\hline
 \multirow{2}{*}{PLS }& ResNet-18&ResNet-50 \\
  & pred. coeff. / PLS& pred. coeff. / PLS \\ \hline
Cifar100  &21.60 / 19.14&19.21 / 18.65\\
Cifar10 &4.67 / 3.63&4.06 / 3.60\\
\hline
\end{tabular}
  \caption{Error rates (\%) on Cifar100 and Cifar10 obtained by PLS  while predicting the smoothing coefficient (pred. coeff.) instead of learning the smoothing distribution (PLS) for each sample pair.
\label{coefficientpredictin}}

\end{table}

\begin{figure}[h]
\centering
    \includegraphics[width=3.20972in, height=1.3in
    ]{reweight2.png}
  \caption{
  Error rates (\%, Y-axis) on Cifar100 obtained by
  varying the weight factor in PLS from 0.1 to 0.9 (X-axis). }
  \label{weighting}
\end{figure}

\textbf{Re-weight Smoothing Strength}
PLS distributes half of a midpoint sample's  ground truth distribution mass over the non-ground truth targets (Equation~\ref{merge}).
We here  evaluate the impact of different weight factors between the ground truth and non-ground truth targets, by varying it from 0.1 to 0.9 (0.5 equals to the average used by PLS). The results obtained by PLS  using  ResNet-18 on Cifar100 are in Figure~\ref{weighting}. The error rates  in Figure~\ref{weighting} suggest
that average used by  midpoints  in  PLS
provides better accuracy than other weighting ratios. This is expected as discussed in Section~\ref{evovinglabel}.

\subsection{ Uncertainty  Label and Conservative Classification}
\label{confidence}
This section will show  that  PLS utilizes  high uncertainty  labels from midpoint samples for training and produces very conservative classification decisions in testing.
\subsubsection{High Uncertainty  Labels in Training}
 We visualize, in Figure~\ref{fig:top3}, the soft target labels used for training by  PLS  with ResNet-18 on  Cifar100  (top) and Cifar10 (bottom). Figure~\ref{fig:top3} depicts the   soft label values of the training samples for both the ground truth targets (in green) and their top 5 largest  non-ground truth classes (in orange). The figure presents the average values over  all the training samples resulting from sample pairs with two different one-hot true labels.  Here,   X-axis depicts the training targets, and   Y-axis  the corresponding distribution mass.

Results in Figure~\ref{fig:top3} indicate that, PLS uses much smaller target values for the ground truth labels during training, when compared to the one-hot representation label used by the baselines and the soft targets used by label smoothing ULS and Mixup.
For example, the largest ground truth  targets for PLS are around  0.25 and 0.35  (green bars), respectively, for Cifar100 and Cifar10. These values are much smaller than 1.0 used by the baselines and  0.9 and 0.5 used by ULS-0.1 and Mixup respectively.
Consequently, in PLS, the distance between  the  ground truth targets and the non-ground truth targets are much smaller than that in the baseline, ULS-0.1 and Mixup. In addition, the distance between the two ground truth targets in PLS (green bars) is very small, when compared to the distance between the ground truth and non-ground truth target values (green vs. orange bars).

\begin{figure}[h]
\caption{
Average distribution  of ground truth targets
(green bars)
and the top 5 largest  non-ground truth targets (orange bars) used by PLS in  training  with  ResNet-18 on Cifar100 (top) and Cifar10 (bottom).
X-axis is the classes and Y-axis the  distribution mass.
 }
\label{fig:top3}
\centering
\includegraphics
width=3.50972i
]{top99.png

\end{figure}

These results indicate that
the training samples in PLS have smoother training targets across  classes, and  those  training signals are far from 1.0 (thus with much higher uncertainty), which in turn impacts how PLS makes its classification decisions as will be discussed next.

\subsubsection{Low Predicted Softmax Scores in Testing}
One effect of the high uncertainty  training signals (far from 1.0) as discussed above was reflected on the model's prediction scores made in test time.
Figure~\ref{fig:hist}  visualizes the predicted winning softmax scores  made by  ResNet-18 (top row), ULS-0.1 (second row), Mixup (third row) and PLS (bottom row) on all the 10K test data samples in Cifar100 (left column) and Cifar10 (right column). To have better visualization, we have removed the predictive scores less than 0.1 for all the  methods since all models obtained similar results for confidences smaller than 0.1.

For  Cifar100, results on the left of  Figure~\ref{fig:hist} indicate that    ResNet-18 produced very confident predictions (top-left subfigure), namely skewing large mass of its predicted softmax scores on 1.0 (i.e., 100\% confidence). On the other hand,   ULS and Mixup  were able to
decrease its prediction confidence at test time (middle rows/left). They  spread their predictive confidences to the two ends, namely moving most of the predicted  scores into two bins [0.1-0.2] and [0.8-1.0].
In contrast,  PLS  produced very conservative predicted softmax scores, by distributing many of its predicted scores to the middle, namely 0.5, and with sparse distribution for scores  larger than 0.7 (bottom left subfigure).

For Cifar10, results  on the right of Figure~\ref{fig:hist} again indicate that  ResNet-18 (top) produced very confident predictions,  putting large mass of its predicted softmax scores  near 1.0.
For  ULS and Mixup  (middle rows), the predicted softmax scores were also distributed near the 1.0, but they are much less than that of  ResNet-18. In contrast,  PLS  (bottom right) again generated very conservative predicted softmax scores. Most of them distributes near the middle  of the  softmax score range, namely 0.5, with a very few larger than 0.6.

These results suggest that, resulting from high uncertainty  training  signals across classes, PLS becomes extremely conservative when generating predicted scores in test time, producing low  winning softmax scores for  classification.

\begin{figure}[h]
\caption{ Histograms of predicted softmax scores on the validation data   by  ResNet-18 (top row), ULS-0.1 (second row), Mixup (third row), and PLS (bottom row) on Cifar100 (left) and Cifar10 (right).
 }
	\centering
\includegraphics[width=4.3814i
]{hist_v3.png}
	\label{fig:hist}
\end{figure}

\subsubsection{Impact on  Model Calibration}
\label{tempscaling}
The  conservative predictions generated by PLS improve the model's accuracy, but  how such low softmax scores  affect the calibration of the output probabilities?
In this section, we report the Expected Calibration Error (ECE)~\cite{10.5555/3305381.3305518} obtained by the baseline PreAct ResNet-18, ULS-0.1, Mixup and PLS on the test set with 15 bins as used in~\cite{MullerKH19} for both  Cifar100 and Cifar10.

Results in Figure~\ref{fig:ece} indicate that ULS (dark curve) is able to reduce the miscalibration error ECE on the Cifar100 data set (left subfigure), but for the Cifar10 dataset (right subfigure), ULS  has larger ECE error after 100 epochs of training than the baseline model. Also, Mixup has higher ECE error than ULS on both Cifar100 and Cifar10. However, the ECE errors obtained by the PLS methods for both the Cifar100 and Cifar10 are much larger than   the baseline,  ULS and Mixup.
Note that, although the authors in~\cite{10.5555/3305381.3305518} state that the Batch Normalization (BN) strategy~\cite{DBLP:conf/nips/Ioffe17} also increases the miscalibration ECE errors for unknown reasons, we doubt that  the PLS will have the same reason as the BN approach.
 This is because the main characteristic of the PLS model is that it produces extremely conservative winning softmax scores which is not the case for the BN strategy.
We here suspect that the high ECE score of the PLS method may be caused by the fact that ECE is an evenly spaced binning metrics but the PLS produces sparse dispersion of the softmax scores  across the range.

\begin{figure}[h]
\caption{ ECE curves on the   validation data of Cifar100 (left) and Cifar10 (right)  from  ResNet-18, ULS-0.1, Mixup, PLS and PLS with TS. X-axis is the training epoch and Y-axis  the ECE value.
	\centering
\includegraphics[width=4.3in]{ece_curv3.png}
	\label{fig:ece}
\end{figure}

To verify the above hypothesis, we further investigate the Temperature Scaling (TS) method~\cite{10.5555/3305381.3305518}, which enables redistributing the distribution dispersion after training with no impact on the testing accuracy.
During testing, TS  multiplies the logits by a scalar before applying the softmax operator.
 We apply this TS technique to PLS, and present the results in Figure~\ref{fig:ece}, depicting by  red curve in the left and right subfigures for the Cifar100 and Cifar10, respectively. The TS factors was 0.5 and 0.2 respectively for Cifar100 and Cifar10, which were found by a search with 10 percentage of the training data.
Figure~\ref{fig:ece} indicates that  TS can significantly improve the  calibration of  PLS for both cases. The ECE errors obtained by PLS-TS for both  Cifar100 and Cifar10 (red curves) are lower than that of the baseline,  ULS, and Mixup

\subsection{Testing on Out-of-Distribution Data}
Section~\ref{confidence} shows that PLS produces very conservative classification decisions for  in-distribution samples. We here
 explore the effect of PLS on out-of-distribution data.

In this experiment, we first train a  ResNet-18 or ResNet-50 network (denoted as vanilla models) with   in-distribution data (using either Cifar10 or Cifar100) and
then let the trained networks to predict on the testing samples from the SVHN dataset (i.e., out-of-distribution samples).
We compare our method with the vanilla baseline (ResNet-18 or ResNet-50), Mixup, and ULS-0.1. The winning predicted softmax scores on the SVHN testing samples  are presented in  Figure~\ref{fig:ood}, where the top and bottom rows depict the results for ResNet-18 and ResNet-50, respectively.

Figure~\ref{fig:ood} shows that,
PLS again produces low winning predicted softmax scores (namely less confident) against the samples from the SVHN dataset when  training with either Cifar10 or Cifar100 data, when compared to  Mixup, ULS, and the vanilla models.
For example, when being trained with Cifar10 and tested on the SVHN data with ResNet-18 (the top-left subfigure), PLS  produced  (orange curve) a spike of score distribution  near the midpoint 0.5, with extremely spare distribution in the regions of high confidence.
In contrast, while Mixup and ULS (yellow/green curves) are more conservative than
the vanilla model on out-of-distribution data, it is noticeably  more overconfident than the PLS strategy by producing a  spike of prediction with 90\% confidence at the right end of the figure.

\begin{figure}[h]
\caption{
Winning softmax scores produced by PLS, Mixup, ULS, and baseline models, when being trained with Cifar10 (left) and Cifar100 (right) and then testing on SVHN with PreAct ResNet-18 (top) and ResNet-50 (bottom).
}%
	\centering
\includegraphics[width=4.3in]{ood6.png}
	\label{fig:ood}
\end{figure}

\section{Conclusion and Future Work}
We proposed a novel output distribution  regularization technique, coined PLS, which learns smoothing distribution for midpoint samples that average random sample pairs.
 We empirically showed   PLS'  superior   accuracy over label smoothing and Mixup models.
 We  visualized   the high uncertainty training labels of  the  midpoint samples, which cause PLS to produce  very  low winning softmax scores  for unseen  in and out of distribution samples.

Our studies here suggest some  interesting directions for future investigation. For example, what are the
other  benefits arising from  high uncertainty training labels and  conservative classification?
Would such uncertain predictions help with  beam search or ranking process for some downstream applications?
Does the proposed method benefit from mixing  more than two samples for smoothing?
Another interesting research direction would be providing theoretical explanation  on
 why the method works considering that the synthetic images are not realistic.
We are also interested in applying our strategy to other domains  beyond image.

\bibliographystyle{splncs04}

\section{Introduction}
Label Smoothing (LS) is a commonly used output distribution regularization technique to improve the generalization performance of deep learning models~\cite{
Szegedy2016RethinkingTI,Chorowski2016TowardsBD,Vaswani2017AttentionIA,8579005,Real2019RegularizedEF,Huang2019GPipeET,structurels}.
Instead of training with data associated with one-hot labels, models with label
smoothing  are trained on samples with  soft targets, where each target is
a weighted mixture of the ground-truth one-hot label  with the uniform distribution of the classes. Such  regularization approaches prevent  overfitting of a model by penalizing overconfident output distributions, resulting in improved model generalization and model calibration~\cite{Szegedy2016RethinkingTI,DBLP:conf/iclr/PereyraTCKH17,yuan2019revisit,MullerKH19,DBLP:conf/aaai/ZhuJZGHSZ20,Lukasik2020DoesLS}. Owing to its simplicity and effectiveness, label  smoothing  has been successfully deployed to improve the accuracy of deep models across a range of applications, including  image classification~\cite{Szegedy2016RethinkingTI}, speech recognition~\cite{Vaswani2017AttentionIA}, and machine translation~\cite{Chorowski2016TowardsBD}.

When distributing the distribution mass of the one-hot  training signal over the non ground-truth classes,   current smoothing strategies, however, only consider the distance between the only gold label and the other non-ground-truth targets. Our observations suggest that providing an additional  ground-truth label to constrain the smoothing strength can significantly improve the smoothing effectiveness. The additional ground-truth label  here acts as a  consistency constraint between the pair of ground-truth labels while distancing them from the other targets. Motivated by this observation, in this paper, we introduce a novel label smoothing technique that takes a pair of samples as input.  We term it Pairwise Label Smoothing (denoted as PLS).

\begin{figure} [h]
\caption{Illustration of label smoothing (left) and pairwise label smoothing (right) on how the distribution mass of the one-hot ground-truth label(s) (in blue bars) are distributed over the non ground-truth targets (in red pattern bars) for a training sample; orange  solid bars are the resulting smoothed ground-truth targets.
 Y-axis indicates the distribution mass of the  four target classes (X-axis) of the training sample.
 }
\label{fig:ill}
\centering
\includegraphics[width=3.316in]{mass2.png}

\end{figure}

In a nutshell, the PLS first averages the inputs and labels of a pair of samples to form a new sample,  and then distributes the  distribution mass of the two ground-truth  targets  of the new sample over its non ground-truth classes. As illustrated in Figure~\ref{fig:ill}, smoothing with a pair  of ground-truth labels  enables  PLS to  preserve  the relative distance between the two truth labels  while being able to further soften that between the truth labels  and the other class targets. Also,   PLS  automatically  learns the  smoothing distribution mass for each input pair during training. Consequently, it effectively eliminates the   turning efforts for searching the right level of smoothing strength when applying  to different data and network architectures.

We empirically show that
the PLS strategy significantly
outperforms  LS  and the baseline models,
with up to 30\% of relative classification error reduction.
In addition,  we  visually demonstrate that PLS produces extremely conservative predictions  in testing time, resulting in that many of its  winning predicted softmax scores are slightly over 0.5. These low softmax scores come with  large Expected Calibration Error (ECE)~\cite{10.5555/3305381.3305518}, but such miscalibration error can be effectively reduced  by
the post-training calibration  step Temperature Scaling~\cite{10.5555/3305381.3305518}.

 Our contributions are as follows.
\begin{itemize}
   \item
We introduce a novel label smoothing method PLS, which leverages a pair of  ground-truth labels to constrain the smoothing strength and embraces  dynamic smoothing distribution.
    \item
We experimentally demonstrate that the  PLS method   significantly improves the predictive accuracy
    of deep classification networks.
We also visually show that when achieving such  accuracy gains the PLS strategy tends to produce very low winning softmax scores.
\end{itemize}

\section{Pairwise Label Smoothing}
\subsection{Preliminaries}
We consider a standard classification setting with a given training data set $(X; Y)$ associated with $K$ candidate classes $\{1,2,\cdots,K\}$.
For an example $x_i$ from the training dataset $(X; Y)$, we denote the ground truth distribution $q$ over the labels as $q(y|x_i)$ ( $\sum_{y=1}^{K} q(y|x_i) = 1$). Also, we denote  a neural network  model to be trained as $f_{\theta}$ (parameterized with $\theta$), and it produces
a conditional  label distribution over the $K$ classes as $p_{\theta}(y|x_i)$:
\begin{equation}
p_{\theta}(y|x_i) = \frac{\exp(z_{y})}{\sum_{k=1}^{K}\exp(z_{y_{k}})},
\end{equation}
with
 $\sum_{y=1}^{K} p_{\theta}(y|x_i) = 1$, and $z$
is noted as the logit of the model $f_{\theta}$. The logits are generated with two steps:
 the model $f_{\theta}$ first  constructs the $m$-dimensional input embedding $S_{i} \in R^{m}$ for the given input $x_{i}$, and then passes it
  through
a linear   fullyconnected layer $W_{l} \in R^{K \times m}$:
\begin{equation}
\label{logit}
S_{i} = f_{\theta}(x_{i}),
\end{equation}
\begin{equation}
\label{mmix}
z=  W_{l}S_{i}.
\end{equation}

During learning, the model $f_{\theta}$ is trained to optimize the parameter $\theta$ using the $n$ examples from  $(X; Y)$ by  minimizing the cross-entropy loss:
\begin{equation}
\ell =- \sum_{i=1}^{n} H_i(q,p_{\theta}).
\end{equation}
Instead of using one-hot encoded vector for each example $x_{i}$ in $(X; Y)$, label smoothing (LS) adds a smoothed label distribution  (i.e., the prior distribution) $u(y|x_{i})$ to each example $x_{i}$, forming a new target label, namely  soft label:
\begin{equation}
\label{oriequ}
q^{\prime}(y|x_i) = (1-\alpha) q(y|x_i) + \alpha u(y|x_i),
\end{equation}
where hyper-parameter $\alpha$ is a weight factor ($\alpha \in [0,1]$) needed to be tuned to indicate the smoothing strength for the one-hot label. This modification results in a new loss function:
\begin{equation}
\label{lsloss}
\ell^{\prime}
= - \sum_{i=1}^{n} \Big[ (1-\alpha) H_i(q,p_{\theta}) + \alpha H_i(u,p_{\theta}) \Big].
\end{equation}
Usually, the $u(y|x_{i})$ is an uniform distribution, independent of data $x_{i}$, as $u(y|x_{i}) = 1/K$, and hyper-parameter $\alpha$ is tuned with cross-validation.
\iffalse
With such  uniform prior, label smoothing is equivalent to adding the KL divergence loss between the uniform distribution $u$ and the network's predicted distribution $q_{\theta}$ due to the fact that $H(u)$ is a constant:
\begin{equation}
\ell^{\prime}
&= - \sum_{i=1}^{n} \Big[ (1-\alpha) H_i(q,p_{\theta}) + \alpha H_i(u,p_{\theta}) \Big] \\
&=  - \sum_{i=1}^{n} \Big[ (1-\alpha) H_i(q,p_{\theta}) + \alpha D_{KL}(u||p_{\theta}(y|x_{i})) \Big],
\end{equation}
where the second measures how dissimilar the predicted distribution $p_{\theta}$ is to uniform distribution over the $K$ target classes.
In this sense, label smoothing is equivalent to replacing a single cross-entropy loss $H(q,p_{\theta})$ with a pair of such losses $H(q,p_{\theta})$ and $H(u,p_{\theta})$. The latter loss  penalizes the deviation of predicted label distribution $p_{\theta}$ from the prior distribution $u$ with the relative weight $\alpha$, which decides how aggressive one can smooth the one-hot labels in the  first loss term.
\fi

\subsection{Pairwise Label Smoothing}
\label{plsmethod}
Our proposed pairwise label smoothing method PLS leverages a pair of samples,  randomly selected from  $(X; Y)$, to conduct label smoothing.

In detail, for a sample $x_{i}$ from the provided training  set $(X; Y)$ for training, PLS first randomly selects~\footnote{For efficiency purpose, we implement this by randomly selecting a sample from the same mini-batch during training.} another training sample $x_{j}$. For the  pair of samples $(x_{i}; y_{i})$ and $(x_{j} ; y_{j})$, where $x$ is the
input and $y$ the one-hot encoding of the corresponding class, PLS then generates a synthetic sample  through element-wisely averaging both the input features and the labels, respectively, as follows:
\begin{equation}
x_{ij} =  (x_{i} +  x_{j})/2,
\end{equation}
\begin{equation}
\label{equ:2}
q(y|x_{ij}) = ( y_{i} +  y_{j})/2.
\end{equation}
In this way, for the sample $x_{ij}$ we have the ground truth distribution $q$ over the labels as $q(y|x_{ij})$ ($\sum_{y=1}^{K} q(y|x_{ij}) = 1$).
The newly resulting  sample $x_{ij}$ will  then be  used for label smoothing (will be discussed in detail later) before feeding into the networks for training.
In other words, the logits as defined in Equation~\ref{mmix} is computed by first generating the  $m$-dimensional image embedding $S_{ij} \in R^{m}$ for the input $x_{ij}$ and then passing through the fully-connected linear layer to construct the logit $z$:
\begin{equation}
S_{ij} = f_{\theta}(x_{ij}),
\end{equation}
\begin{equation}
\label{mmix22}
z= W_{l}S_{ij}.
\end{equation}
Similarly, the conditional  label distribution over the $K$ classes is computed as
\begin{equation}
p_{\theta}(y|x_{ij}) = \frac{\exp(z_{y})}{\sum_{k=1}^{K}\exp(z_{y_{k}})}. \end{equation}

\subsubsection
{Dynamic Smoothing Distribution}\label{evovinglabel}
PLS leverages a learned distribution,  which depends on the input $x$, to dynamically generate the smoothing distribution mass for distributing the ground-truth target distribution to the non-target classes.
To this end, the PLS implements this by adding a  fullyconnected layer  to the network $f_{\theta}$. That is,   the
$f_{\theta}$ produces two projections from the penultimate layer representations of the network:  one for the logits as the original network (Equation~\ref{mmix22}), and another for generating the smoothing distribution as follows.

In specific, an additional  fullyconnected
layer $W_{t} \in R^{K \times m}$ is added to the original networks $f_{\theta}$ to produce the smoothing distribution over the $K$ classification classes.
That is, for the given
input image $x_{ij}$,  its smoothing  distributions over the $K$ classification  targets, denoted as  $u^{\prime}_{\theta}(y|x_{ij})$, are computed as follows:
\begin{equation}
\label{prelabel}
u^{\prime}_{\theta}(y|x_{ij}) =
\frac{\exp(v_{y})}{\sum_{k=1}^{K}\exp(v_{y_{k}})},
\end{equation}
   \begin{equation}
   \label{sha}
   v = \sigma (W_{t}S_{ij}),
   \end{equation}
where $\sigma$ denotes the Sigmoid function, and
 $S_{ij}$ is the same image embedding as that in Equation~\ref{mmix22}. In other words, the two predictions (i.e., Equations~\ref{mmix22} and~\ref{sha}) share the same networks except the last fully connected layer.
That is, the only difference between PLS and the original networks is the added  fullyconnected layer $W_{t}$.

After having the smoothing distributions $u^{\prime}_{\theta}(y|x_{ij})$, PLS then uses them to smooth the ground-truth labels  $q(y|x_{ij})$ as described in Equation~\ref{equ:2}, with an  average:
\begin{equation}
\label{merge}
q^{\prime}(y|x_{ij}) = ( q(y|x_{ij}) +  u^{\prime}_{\theta}(y|x_{ij}))/2.
\end{equation}

The loss function of PLS thus becomes the follows:
\begin{equation}
\begin{split}
\ell^{\prime}
=& - \sum_{i=1}^{n} \Big[ 0.5 \cdot  H_i(q(y|x_{ij}),p_{\theta}(y|x_{ij})) \\
&+  0.5\cdot H_i(u^{\prime}_{\theta}(y|x_{ij}),p_{\theta}(y|x_{ij}))
\Big].
\end{split}
\end{equation}
For training, PLS minimizes, with gradient descent on mini-batch, the  loss  $\ell^{\prime}$.
One more issue needed to be addressed for the training. That is, the data samples used for training, namely $(x_{ij}; y)$ may lack  information on the original training samples  $(x_{i}; y)$ due to the average operation in the PLS. To compensate this fact,
we alternatively feed inputs to the networks with either a mini-batch from the original images, i.e., ${x}_{i}$ or ${x}_{ii}$,  or a mini-batch from the averaged images, i.e., ${x}_{ij}$.
Note that, when training with the former, the networks still  need to learn to assign the smoothing distribution $ u^{\prime}_{\theta}(y|x_{ii})$ to form the  soft  targets $q^{\prime}(y|x_{ii})$ for the sample ${x}_{ii}$.
As will be shown in the experiment  section, this training strategy is   important to PLS'  regularization effect.

\section{Experiments}
\subsection{Datasets, Baselines, and Settings}

We evaluate our proposed  method PLS with the following five benchmark image classification tasks.
\noindent \textbf{MNIST} is a digit (1-10)  recognition dataset with 60,000 training and 10,000 test 28x28-dimensional gray-level images.
\noindent
\textbf {Fashion} is an image recognition dataset with the same scale as
MNIST, containing 10 classes of fashion product pictures.
\noindent
\textbf{SVHN} is the Google street view house numbers recognition data set. It has 73,257 digits, 32x32 color images for training, 26,032 for testing, and 531,131 additional, easier samples. Following literature, we did not use the additional images.
\noindent
\textbf{Cifar10} is an image classification task with  10 classes. It has
  50,000 training  and 10,000 test samples.
\noindent
\textbf{Cifar100} is similar to Cifar10 but with 100 classes and 600 images each.

We conduct experiments  using the popular benchmarking networks  PreAct ResNet-18~\cite{DBLP:journals/corr/HeZR016} and ResNet-50~\cite{DBLP:journals/corr/HeZR016}. We also perform ablation evaluations using deeper and wider networks
 WideResNet-28-10~\cite{DBLP:conf/bmvc/ZagoruykoK16} and
  DenseNet-121~\cite{densenet}.
We  compare with the state-of-the-art label smoothing methods~\cite{MullerKH19,Lukasik2020DoesLS} (denoted as ULS) with various smoothing coefficients (i.e., $\alpha$ as defined in Equation~\ref{lsloss}), where ULS-0.1, ULS-0.2, and ULS-0.3  denote the smoothing coefficient of 0.1, 0.2, and 0.3, respectively.
We also compare our method with the input-pair based data augmentation method  Mixup~\cite{Mixup17}, which learns from synthetic samples generated from a pair of inputs through interpolations on both the inputs and labels of the input pair. For Mixup, we use the authors' code at~\footnote{https://github.com/facebookresearch/mixup-cifar10} and the uniformly selected mixing coefficients between [0,1].
For PreAct ResNet-18, ResNet-50, and DenseNet-121, we use the PyTorch implementation  from Facebook~\footnote{https://github.com/facebookresearch/mixup-cifar10/blob/master/models/}. For PLS, the added fullyconnected layer is the same as  the last fullyconnected layer of the baseline network with a Sigmoid function on the top.
All models  are trained using mini-batched (128 examples) backprop,
 with the  exact settings as in the Facebook codes,     for 400 epochs.   Each reported  value (accuracy or error rate) is the mean of   five runs. All our experiments were run on a NVIDIA GTX TitanX GPU with 12GB memory.

\begin{table*}[h]
  \centering
\scalebox{0.9913}{
\begin{tabular}{l|c|c|c|c|c}\hline
Methods& MNIST&Fashion&SVHN&Cifar10&Cifar100\\ \hline
PreAct ResNet-18 &0.62	$\pm$0.05&4.78$\pm$	0.19&3.64$\pm$	0.42&5.19$\pm$	0.30&	24.19$\pm$	1.27\\%&55.80$\pm$0.34\\
ULS-0.1 &0.63$\pm$0.02&4.81$\pm$0.07&3.20 $\pm$ 0.06&4.95$\pm$0.15&21.62$\pm$0.29\\%&57.20$\pm$0.44\\
ULS-0.2 &0.62$\pm$0.02&4.57$\pm$0.05&3.14 $\pm$ 0.11&4.89$\pm$0.11&21.51$\pm$0.25\\%&56.37$\pm$0.22\\
ULS-0.3 &0.60$\pm$0.01&4.60$\pm$0.06&3.12 $\pm$ 0.03&5.02$\pm$0.12&21.64$\pm$0.27\\%&55.48$\pm$0.14\\
Mixup  &0.56$\pm$	0.01&4.18$\pm$	0.02&3.37$\pm$	0.49&3.88$\pm$	0.32&	21.10$\pm$	0.21\\%&54.35$\pm$0.20\\
\hline
PLS &\textbf{0.47 $\pm$0.03}&\textbf{3.96$\pm$0.05}&\textbf{2.68 $\pm$0.09}&\textbf{3.63$\pm$0.10}&\textbf{19.14$\pm$0.20}\\%&\textbf{51.12$\pm$0.12}\\
Rel. Imp. over Ave. ULS (\%)&23.78&15.02&15.01&26.72&11.35\\%&9.28\\
Rel. Imp. over Baseline  (\%)&24.19&17.15&26.37&30.06&20.88\\%&8.39\\
\hline
\end{tabular}
}
\caption{Error rate (\%) of the testing methods with PreAct ResNet-18~\cite{DBLP:journals/corr/HeZR016} as baseline.  We report mean scores over 5 runs with standard deviations (denoted $\pm$). The relative improvement of PLS over the baseline  and the average of the label smoothing ULS with various smoothing coefficients are  provided in the last two rows of the table. Best results are  in \textbf{Bold}.
  }    \label{tab:accuracy:resnet18}
\end{table*}

\begin{table*}[h]
  \centering
\scalebox{0.9937}{
\begin{tabular}{l|c|c|c|c|c}\hline
Methods& MNIST&Fashion&SVHN&Cifar10&Cifar100\\ \hline
ResNet-50 &0.61$\pm$0.05
	&4.55$\pm$0.14	&3.22$\pm$0.05&4.83$\pm$0.30&23.10$\pm$0.62	\\
ULS-0.1 &0.63$\pm$0.02&4.58$\pm$0.16&2.98$\pm$0.02&4.98$\pm$0.25&23.90$\pm$0.99\\
ULS-0.2 &0.62$\pm$0.03&4.52$\pm$0.04&3.08$\pm$0.03&5.00$\pm$0.35&23.88$\pm$0.73\\
ULS-0.3 &0.65$\pm$0.03&4.51$\pm$0.15&3.04$\pm$0.07&5.16$\pm$0.16&23.17$\pm$0.50\\
Mixup  &0.57$\pm$	0.03&4.31$\pm$0.05	&2.85$\pm$0.07&4.29$\pm$0.28&19.48$\pm$0.48\\
\hline
PLS &\textbf{0.51$\pm$
0.02}&\textbf{4.15$\pm$0.09}&\textbf{2.36$\pm$0.03}&\textbf{3.60$\pm$0.18}&\textbf{18.65$\pm$1.08}\\
Rel. Imp. over ULS (\%)&19.47&	8.52&	22.20&	28.67&	21.14\\
Rel. Imp. over Baseline  (\%)&16.39&8.79&	26.71&	25.47&	19.26\\

\hline
\end{tabular}
}
\caption{Error rate (\%) of the testing methods with  ResNet-50~\cite{DBLP:journals/corr/HeZR016} as baseline.  We report mean scores over 5 runs with standard deviations (denoted $\pm$). The relative improvement of PLS over the baseline model and the average of the label smoothing (ULS) with various smoothing coefficients are  provided in the last two rows of the table. Best results are  in \textbf{Bold}.
  }    \label{tab:accuracy:resnet50}
\end{table*}

\subsection{Predictive Accuracy}
The predictive error rates obtained by label smoothing with various smoothing coefficients (i.e., ULS), Mixup, and PLS with   PreAct ResNet-18  as baseline on the five test datasets are presented in Table~\ref{tab:accuracy:resnet18}, where the relative improvement of PLS over the baseline model PreAct ResNet-18 and the average of the three  ULS models (i.e., ULS-0.1, ULS-0.2, ULS-0.3) are in the last two rows of the table. The  results with ResNet-50 as baselines for the testing methods are provided in Table~\ref{tab:accuracy:resnet50}.

The results in Table~\ref{tab:accuracy:resnet18} show that PLS outperforms, in terms of predictive error, the PreAct ResNet-18 baseline, the label smoothing models (ULS-0.1, ULS-0.2, ULS-0.3), and Mixup  on  all the five datasets.
For example, the relative improvement of PLS over the baseline model  on the Cifar10 and MNIST datasets, as depicted in the  last row of the table, are over 30\% and 24\%, respectively.
When considering PLS and the average error obtained by the three ULS models, as depicted in the second last row of the table, the relative error reduction on all the five datasets is at least 11\%; on both the Cifar10 and MNIST tasks, the relative improvement is over 23\%.

When considering the cases with ResNet-50 as baselines, results as shown in
Table~\ref{tab:accuracy:resnet50}  indicate that similar error reductions are obtained by PLS. Again, on all the five testing datasets, PLS outperforms all the comparison baselines, namely ResNet-50, label smoothing with three different coefficient settings, and Mixup. Also, the relative improvement is large in some cases. For example, for the Cifar10 dataset, the relative improvement achieved by PLS over the baseline ResNet-50 and the average of the three label smoothing strategies (i.e. ULS) are 25.47\% and 28.67\%, respectively.

These results suggest that the PLS method is able to significantly reduce the predictive error of the baseline models with or without label smoothing (ULS) applied.

\subsubsection{Accuracy on Deeper and Wider Networks}
We also evaluate PLS using deeper and wider networks:  WideResNet-28-10~\cite{DBLP:conf/bmvc/ZagoruykoK16}, and  DenseNet-121~\cite{densenet}.
The results are in Table~\ref{tab:accuracy:wideresnet}.

Similar to that of using PreAct ResNet-18 and ResNet-50 as presented in Tables~\ref{tab:accuracy:resnet18} and~\ref{tab:accuracy:resnet50}, PLS  performed consistently better on the two additional network architectures, i.e.,  the much wider network WideResNet-28-10  and the 121 layers network DenseNet-121. For example, as shown in the table, PLS significantly outperformed the  baseline and the label smoothing strategy ULS with different smoothing coefficients.

\iffalse
\begin{table*}[h]
  \centering
\scalebox{1}{
 \begin{tabular}{l|c|c|c|c|c}\hline
method&Baseline&ULS-0.1&ULS-0.2&ULS-0.3&PLS\\\hline
WideResNet-28-10 &19.90	$\pm$ 0.41&19.59$\pm$0.17&20.05$\pm$0.43&20.41$\pm$0.28&\textbf{17.12$\pm$0.26}\\
DenseNet-121&19.50$\pm$	0.34&19.42$\pm$0.10&19.37$\pm$0.15&19.45$\pm$0.09&\textbf{16.90$\pm$0.42}\\
\hline
\end{tabular}
}
\caption{Error rates (\%) of testing methods withWideResNet28-10 and DenseNet-121 as baselines. We report mean scores over 5 runs with standard deviations (denoted $\pm$).  Best results are in \textbf{Bold}.
  }
  \label{tab:accuracy:wideresnet}
\end{table*}
\fi

\begin{table}[h]
  \centering
\scalebox{0.97}{
 \begin{tabular}{l|c|c}\hline
Networks&WideResNet-28-10&DenseNet-121\\ \hline
Baseline &19.90	$\pm$ 0.41&19.50$\pm$	0.34\\
ULS-0.1 &19.59$\pm$0.17&19.42$\pm$0.10\\
ULS-0.2 &20.05$\pm$0.43&19.37$\pm$0.15\\
ULS-0.3 &20.41$\pm$0.28&19.45$\pm$0.09\\
Mixup &	18.11$\pm$	0.30&17.92$\pm$0.12	\\
\hline
PLS &\textbf{17.12$\pm$0.26}&\textbf{16.90$\pm$0.22}\\
\hline
\end{tabular}
}
\caption{Error rates (\%) of testing methods with WideResNet28-10 and DenseNet-121 as baselines. We report mean scores over 5 runs with standard deviations (denoted $\pm$).  Best results are in \textbf{Bold}.
  }
  \label{tab:accuracy:wideresnet}
\end{table}

\subsubsection{Training Characteristics}
To better understand the training characteristics of the PLS method, in Figure~\ref{fig:converge} we  plot the training loss and validation error rate across the 400 training epochs of  PLS, ULS-0.1, and  PreAct ResNet-18  on the  Cifar100 dataset.

 Figure~\ref{fig:converge} shows that  the baseline mode's training loss goes to zero after 100 epochs, providing no gradient after that.  On the contrary, the  training loss  of PLS (green curve in the right subfigure) maintains a relatively higher  level than the baseline and the ULS methods,  allowing the PLS model to keep tuning the networks. Promisingly, as shown by the  validation error curve (green curve in the left subfigure),  PLS  is not overfitting the training set with the high training loss but keep decreasing the  validation error.

\begin{figure}[h]
\caption{Training loss (right) and validation error (left) of PreAct ResNet-18 on  Cifar100.
 }
\label{fig:converge}
\centering
\includegraphics[width=3.3in]{converge.pdf}

\end{figure}

\subsubsection{Impact of Key Components}
\label{ablation}
We  also evaluate the impact of the key components in  PLS using PreAct ResNet-18 and ResNet-50 on Cifar100. Results are in Table~\ref{aaaablation}. The key components include  1) removing the  learned smoothing distribution mass in Equation~\ref{merge},  2) excluding the use of the original training inputs as discussed in the method section, and 3) replacing the learned smoothing distribution mass with uniform distribution with weighting coefficients of 0.1, 0.2, and 0.3 (denoted as UD-0.1, UD-0.2 and UD-0.3).

The error rates obtained in  Table~\ref{aaaablation} show that,
  both  the learned smoothing distribution mass and  the original training samples are critical for the PLS model.
In particular, when excluding the original  samples  from  training, the predictive error of PLS  dramatically increased from about 19\% to nearly 24\% for both the PreAct ResNet-18 and ResNet-50. The reason,  as discussed in the method section,
 is that,
without the original training samples, the networks may lack  information on the validation samples.

\begin{table}[h]
  \centering
  \begin{tabular}{l|r|r}\hline
 \multirow{2}{*}{PLS }& ResNet-18&ResNet-50 \\
  &19.14&18.65 \\ \hline
----  no learned distribution  &21.06&19.35\\
----  no original  images &23.84&24.42\\
 \hline
----  UD 0.1&19.50&18.91\\
----  UD 0.2&19.25&18.81\\
----  UD 0.3&19.31&18.89
\\
\hline
\end{tabular}
  \caption{Error rates (\%) on Cifar100 obtained by PLS  while varying its key components.
\label{aaaablation}}

\end{table}

Also, results  in Table~\ref{aaaablation} indicate that,  replacing the learned smoothing distribution mass in PLS with manually tuned Uniform distribution mass (i.e., UD) obtained slightly larger errors. Theses results indicate that the PLS method is able to adaptively learn the  distribution mass for smoothing the two target labels, resulting in superior  accuracy and excluding the need for the  coefficient search  for different  applications.
These results here also further confirm the benefits of conducting label smoothing on sample pairs.

\begin{figure}[h]
\centering
    \includegraphics[width=2.20972in, height=1.002in]{reweight.png}
  \caption{
  Error rates on Cifar100 obtained by
  varying the weight factor in PLS. }
  \label{weighting}
\end{figure}

\subsubsection{Re-weight Smoothing Strength}
PLS distributes half of its ground-truth distribution mass over the non ground-truth targets as discussed in Equation~\ref{merge}.
We here  evaluate the impact of different weight factors between the ground-truth and non ground-truth targets, by varying it from 0.1 to 0.9. Note that, 0.5 equals to the average used by PLS. The results obtained by PLS  using PreAct ResNet-18 on Cifar100 are in Figure~\ref{weighting}. The error rates obtained in Figure~\ref{weighting} suggest
that average as deployed by PLS
provides better accuracy than that of other weighting ratios.
Why average works better is not obvious to us, so we would like to leave it to future work.

\iffalse
\subsection{Class Activation Mapping (CAM) Visualization}
After  the 400 training epochs, we also, in
Figure~\ref{cam},  depict the CAM maps~\cite{cvpr2016_zhou} of ten classes  using  testing samples from Cifar10 obtained by the baseline, LS-0.1 and PLS, on the top, middle, and bottom rows respectively.
Figure~\ref{cam} that, LaMix tends to focus on   narrower regions of the images  for salient features than the baseline and the LS-0.1 models.

\begin{figure}[h
\caption{ Left: two classes of  Cifar10 testing images; The right three   depict the visualized CAMs of LaMix, Mixup, and CutMix, respectively..  }
	\centering
\includegraphics[width=4in]{cam.pdf}
	\label{cam}
\end{figure}
\fi

\begin{figure}[h]
\caption{
Average distribution mass of ground-truth targets
(green bar)
and the top 5 largest  non ground-truth targets (red bar) used by PLS in  training  with PreAct ResNet-18 on Cifar100 (left) and Cifar10 (right).
X-axis depicts the targets, and Y-axis indicates their distribution mass.
 }
\label{fig:top3}
\centering
\includegraphics[width=3.3in]{top55.png

\end{figure}

\vspace{-2mm}

\subsection{Why and How   PLS Benefits from Label Pair?}
\label{confidence}

\subsubsection{Soft Labels Used in Training}
To peek into how the additional ground-truth label in PLS constrains the smoothing distribution mass,
 we visualize, in Figure~\ref{fig:top3}, the soft target labels used for training by  PLS  with PreAct ResNet-18 on  Cifar100 (left) and Cifar10 (right). Figure~\ref{fig:top3} depict the   soft label values of the training samples for both the ground-truth targets (in green) and their top 5 largest  non ground-truth classes (in red). The figure presents the average values over  all the training samples resulting from sample pairs with two different one-hot true labels.  In the figure, the X-axis depicts the training targets, and the  Y-axis indicates the corresponding distribution mass.

Results in Figure~\ref{fig:top3} indicate that, PLS uses much smaller target values for the ground-truth labels during training, when compared to the one-hot representation label used by the baseline models and the soft targets used by label smoothing ULS.
For example, the largest ground-truth training targets for PLS are around  0.25 and 0.35  (green bars), respectively, for Cifar100 and Cifar10. These values are much smaller than the 1.0 used by the baseline models and the 0.9 used by the ULS-0.1 models.
Consequently, in PLS, the distance between  the  ground-truth targets and the non ground-truth targets are much smaller than that in the baseline and the ULS-0.1 models. In addition, the distance between the two ground-truth targets in PLS (green bars) is very small, when compared to the distance between the ground-truth and non ground-truth target values (green vs. red bars).

Theses results indicate that smoothing with a pair  of ground-truth labels  enables  PLS to  preserve  the relative distance between the two truth labels  while being able to further soften that between the truth labels  and the other class targets, for both  Cifar100 and Cifar10.
In other words, the training samples in PLS have smoother training targets across all classes, and  those  training signals are far from 1.0, which in turn impacts how PLS makes its classification decisions as will be discussed next.

\begin{figure}[h]
\caption{ Histograms of predicted softmax scores on the validation data   by PreAct ResNet-18 (top), ULS-0.1 (middle), and PLS (bottom) on Cifar100 (left) and Cifar10 (right).
 }
	\centering
\includegraphics[width=3.4in,height=2.8in]{hist_v.png}
	\label{fig:hist}
\end{figure}

\subsubsection{Predicted Softmax Scores in Testing}
We observed that a direct effect of the low training signals (far from 1.0) as discussed above was reflected on the model's prediction scores made in test time.
In this section, we present the PLS strategy's predicted softmax scores made in test time  in order to get further idea about why and how the PLS methods benefit from a pair of  samples when conducting label smoothing.

In Figure~\ref{fig:hist}, we visualize the predicted softmax scores  made by PreAct ResNet-18 (top), ULS-0.1 (middle), and PLS (bottom) on all the 10K test data samples in Cifar100 (left column) and Cifar10 (right column). To have better visualization, we have removed the predictive scores less than 0.1 for all the testing methods since all models obtained similar results for confidences smaller than 0.1.

For the Cifar100 dataset, results on the left of  Figure~\ref{fig:hist} indicate that  the baseline model PreAct ResNet-18 produced very confident predictions (top), namely skewing large mass of its predicted softmax scores on 1.0 (i.e., 100\% confidence). On the other hand,  the ULS method was able to
decrease its prediction confidence at test time (middle). It  spreads its predictive confidences to the two ends, namely moving most of the predicted softmax scores into two bins [0.1-0.2] and [0.9-1.0].
Interestingly, the PLS strategy produced very conservative predicted softmax scores, by distributing many of its predicted scores to the middle, namely 0.5, and with sparse distribution for scores  larger than 0.7 (bottom subfigure).

When considering the Cifar10 data, results as shown on the right of Figure~\ref{fig:hist} again indicate that the baseline model PreAct ResNet-18 (top) produced very confident predictions, namely put large mass of its predicted softmax scores  near 1.0.
For the ULS model (middle), the predicted softmax scores were also distributed near the 1.0, but it is much less than that of PreAct ResNet-18. On the other hand, the PLS method (bottom) again generated very conservative predicted softmax scores. The predicted scores mostly distribute near the middle point of the  softmax score range, namely 0.5, with a very few larger than 0.7.

These results suggest that, resulting from the further smoothed  training target signals across classes, PLS becomes extremely conservative when generating predicted scores in test time, producing small winning softmax scores when making classification decisions.

\subsubsection{Model Calibration}
To further tease out the  conservative predictions preferred by the PLS method, we  evaluate how such conservative softmax scores  affect the calibration of the output probabilities. In this section, we report the Expected Calibration Error (ECE)~\cite{10.5555/3305381.3305518} obtained by the baseline PreAct ResNet-18, ULS-0.1, and PLS on the test set with 15 bins as used in~\cite{MullerKH19} for both  Cifar100 and Cifar10.

Results in Figure~\ref{fig:ece} indicate that ULS (dark curve) is able to reduce the miscalibration error ECE on the Cifar100 data set (left subfigure), but for the Cifar10 dataset (right subfigure), ULS  has larger ECE error after 100 epochs of training than the baseline model. However, the ECE errors obtained by the PLS methods for both the Cifar100 and Cifar10 are much larger than  both the baseline and the ULS models.
Note that, although the authors in~\cite{10.5555/3305381.3305518} state that the Batch Normalization (BN) strategy~\cite{DBLP:conf/nips/Ioffe17} also increases the miscalibration ECE errors for unknown reasons, we doubt that  the PLS will have the same reason as the BN approach.
 This is because the main characteristic of the PLS model is that it produces extremely conservative winning softmax scores which is not the case for the BN strategy.
We here suspect that the high ECE score of the PLS method may be caused by the fact that ECE is an evenly spaced binning metrics but the PLS produces sparse dispersion of the softmax scores  across the range.
As previously shown in Figure~\ref{fig:hist}, PLS produces a very sparsely populated region
 at the rightward end.

\begin{figure}[h]
\caption{ ECE on  validation data of Cifar100 (left) and Cifar10 (right)  by  ResNet-18, ULS-0.1 and PLS.
	\centering
\includegraphics[width=3.3in]{ece_curv.pdf}
	\label{fig:ece}
\end{figure}

To verify the above hypothesis, we further investigate the Temperature Scaling (TS) method introduced  in~\cite{10.5555/3305381.3305518}, which allows us to redistribute the distribution dispersion after training with no impact on the testing accuracy.
During testing, TS  multiplies the logits by a scalar before applying the softmax operator.
 We apply this TS technique to PLS, and present the results in Figure~\ref{fig:ece}, depicting by  red curve in the left and right subfigures for the Cifar100 and Cifar10, respectively. The TS factors was 0.5 and 0.2 respectively for Cifar100 and Cifar10, which were found by a search with 10 percentage of the training data.
Results in Figure~\ref{fig:ece} indicate that the TS can significantly improve the  calibration of the PLS for both cases. The ECE errors obtained by PLS-TS for both the Cifar100 and Cifar10 cases (red curves) are lower than both the baseline (blue curves) and the label smoothing models ULS (black curves), and did not show any upward trend during training.

\vspace{-2mm}
\begin{figure}[h]
\caption{
Numbers of samples (red bar) and ECE errors (green curve) in different bins used by ECE for the PLS (left) and PLS + Temperature Scaling (right) on Cifar100 (top row) and Cifar10 (bottom row).
Note that the pre-set bin number is 15, but not all bins have samples.

	\centering
\includegraphics[width=3.3in]{redist4.png}
	\label{fig:temperaturescaling}
\end{figure}

\vspace{-2mm}

How does TS help PLS in terms of ECE error?
 Figure~\ref{fig:temperaturescaling} further depicts  the number of samples (right y-axis) for each of the 15 pre-set bins~\footnote{Note that not all bins have samples.} and its corresponding ECE error (left y-axis) without  and with  TS applied to PLS at the left and right subfigure respectively. We present the results for   Cifar100 and Cifar10 at the top and bottom rows,
  respectively.
As shown in Figure~\ref{fig:temperaturescaling},
 samples in the densely populated distribution regions in the middle (around 0.5), as depicted at the left subfigures (red bars), were  effectively redistributed by the TS to the sparsely populated area at the right end near 1.0, resulting in the redistributed mass in the right subfigures.
As a result, the ECE error for each bin got smaller (green curve) after the redistribution by the TS. This effect resulted in smaller overall miscalibration error than the baseline and the ULS methods as presented previously.
It is worth noting that, unlike typical networks, the TS factor is usually larger than 1.0 to avoid over-confident predictions~\cite{10.5555/3305381.3305518}, for PLS the scaling factor needs to be less than 1.0 in order to re-distribute
the densely populated distribution regions in the middle to the sparsely populated area at the right end.

\section{Related Work}
Label smoothing has shown to  provide consistent accuracy gains across many tasks~\cite{Szegedy2016RethinkingTI,DBLP:conf/iclr/PereyraTCKH17}.
Müller et al.~\cite{MullerKH19}  explicitly show that label smoothing not only  improves model generalization but also enhances model calibration, empowering the models to generate less over-confident predictions. Li et al.~\cite{Lukasik2020DoesLS}  demonstrate that label smoothing can also  mitigate label noise. In~\cite{structurels}, Li et al.
identify a quantifiable bias of the Bayes error rate in the uniform label smoothing, and propose a cluster-based strategy to fix that.
Label smoothing also relates to DisturbLabel~\cite{7780883}, which can be seen as a marginalized version of label dropout.
Unlike the above methods which apply label smoothing to  each single input, our smoothing strategy leverages a pair of inputs for label smoothing. Also, the smoothing distribution mass of our approach is not uniform but  dynamic, which is automatically computed based on the pair of inputs.

Our work is also related to methods that leverage a pair of samples for learning such as Mixup~\cite{Mixup17} and  variants~\cite{DBLP:conf/iclr/TokozumeUH18,adamixup,cutmix19,li2020feature,attributemix,supermix,DBLP:conf/aaai/Guo20,attentioncutmix,DBLP:journals/corr/abs-1905-08941,DBLP:journals/corr/abs-1906-06875}. These methods focus on augmenting the size of the training data. They  interpolate both the features and labels through random mixing coefficients sampling  between [0,1].
Our work here does not create a set of synthetic inputs from a pair of inputs.
 Instead, our method focuses on distributing the distribution mass of the two one-hot  labels of an input pair to the other non ground-truth targets of the tasks.

The generation of the dynamic smoothing distribution mass in PLS is also related to how the training targets generated by self-distillation methods~\cite{hinton2015distilling,Furlanello2018BornAN,DBLP:conf/aaai/YangXQY19,ahn2019variational,mobahi2020selfdistillation}. However, these self-distillation approaches treat the final predictions as target labels for a new round of training, and  the teacher and student architectures are identical~\cite{mobahi2020selfdistillation}.  In PLS, the target  prediction network and the smoothing distribution mass generation network have different architectures, and the training targets for the classification model are a mix of the outputs of the two networks.

\section{Conclusion and Future Work}
We contributed a novel label smoothing strategy PLS. The PLS leverages a pair of inputs to constrain the distributing of the distribution mass of the ground-truth labels over  other non ground-truth targets.
 We empirically showed that  PLS  significantly outperforms, in terms of predictive accuracy, the baseline models with or without uniform label smoothing applied.
 We  visually showed that when achieving such accuracy gains the PLS strategy tends to produce very low winning softmax scores.
We  also empirically demonstrated that these conservative predictions come with large  miscalibration error ECE, but  such ECE error can be effectively reduced by the simple post training process step Temperature Scaling.

Our studies here suggest some  interesting directions for future investigation. For example, what are the other benefits for very conservative predicted softmax scores?
Another interesting research direction would be providing theoretical explanation  on why the PLS and Batch Normalization strategies  improve the predictive accuracy but decrease the model miscalibration score ECE.

\bibliographystyle{labelsmoothing}

redpajama_set_name: RedPajamaArXiv


"One of the things I love about being a Dramatists Guild member (in addition to all the great programs and services) is being a part of the DG community. I was really inspired by the recent issue of The Dramatist, The Community Issue. It included so many great examples of how we can build a community of playwrights and our local theatre community, as well as a community with our neighbors in all walks of life.
Emma is reachable at epalzere-rae@dramatistsguild.com.
We would like to invite you to participate in a regional membership survey to help Emma gain perspective about how to best proceed with assisting membership in your area, since each region is unique and might require or desire different types of programming. To take the survey, please CLICK HERE.
For further questions or concerns about the Regional Rep program, please contact Jordan Stovall, Manager of the Regional Rep program.


how often they will be accessed.
Storing records in appropriate environmental conditions will prevent deterioration and is a central factor in ensuring their long-term access.
If your agency is using a storage provider it is important to remember that it is still the agency's responsibility to make sure that they are kept in appropriate conditions and can be accessed when required.
Records that are sentenced 'retain as national archives' under a records authority or general disposal authority may then be transferred to the National Archives. Once this occurs the agency is no longer responsible for the storage of those records.
With more and more digital information being produced and stored by Australian Government agencies we understand that there needs to be flexibility in arrangements to meet the needs of agencies, while making sure that records of continuing value are kept as national archives (RNA) and are available for public use.
In 2016 the Archives published its Distributed Custody Policy. Under this policy the default position is still for RNA records in all formats to be transferred to the care and custody of the Archives. However, the it also acknowledges that sometimes it may be more efficient for these records to remain with the agency or to be transferred to another organisation.
This distributed custody model recognises the complexity of systems, and in particular digital systems, which store information, the nature of digital records, and the need for agencies to sometimes keep information beyond the period when it would normally be transferred to the Archives. This could be for a variety of reasons including for ongoing business needs, because of the specialist knowledge that is needed to manage or interpret the information, or to deliver enhanced access to the records.
Under the Policy the Archives and the custodial organisation set up a distributed custody arrangement under section 64 of the Archives Act. This ensures the records remain in the custody of the agency or organisation where the records are held with the appropriate care, preservation and access arrangements to enable the Archives to meet its obligations under the Act.
Please contact us through the Agency Service Centre if you have any questions about the Distributed Custody Policy or for more advice about your circumstances.


The closer we get to Thanksgiving, the closer we are to the new year. From a business planning standpoint, if you haven't already started thinking about the who, what, where, why and when (5Ws) of your business, it's time to start. The most basic form of a business plan should include your objective, strategies, the tactics to deliver on each strategy and finally the financial implications of what you expect to get back in return (ROI). We can easily get caught up in the day-to-day, "executional" activities of our business and suddenly find ourselves with no new plans for growth in the new year. Between now and January 1st here are some simple steps to help you focus and create a plan for the new year.
1. Set aside specific days and times to work on your b-plan for the next 6 weeks; 2 hour blocks of time are best.
2. Start by reviewing last year's plan – what worked, what didn't work and why; best done with your management team.
3. Establish your objectives and goals for 2014; need to grow the business by x%.
4. Develop the Marketing, Sales, and Operational strategies to achieve your goals.
5. Line out the tactics needed to make each of the strategies happen.
6. Financially calculate the cost of the tactics and the expected ROI; repeat this step as necessary to meet your budget.
A good resource for helping map out your business plan is the book Business Model Generation by Osterwalder and Pigneur. Let us know if you have other good resources you'd recommend for helping plan for the new year.
‹ How do you – To Do?
1 Pings/Trackbacks for "New Year Planning Starts Now"


In its response to the Treasury consultation on the proposed 3% levy for subsequent property purchases, the Residential Landlords Association (RLA) accused the government of showing favour to larger investors, many of whom are likely to be from overseas.
The trade body has called for all new-build properties contributing to a net increase in the UK's housing stock to be exempt from the Stamp Duty hike. It cited research of 1,100 landlords that found 30% would be more likely to buy new properties if this were the case.
Responses to the consultation are due by 1 February. However, the government is expecting to implement its proposals on 1 April, just weeks after the final policy design is announced during the Budget on 16 March.
As a result, the RLA is urging the government to give more time to consider the implications of the policy, highlighting the potential ramifications on housing supply and rent levels.
RLA chairman Alan Ward described Chancellor George Osborne's decision to limit opportunities for individual UK landlords as 'astonishing'.
"This additional assault on private landlords coming on top of changes to the taxation of rental income will only lead to reduced supply and higher rents," he said.
The RLA is also taking advice over whether the government's decision to limit mortgage interest relief to the basic rate for landlords is a breach of the Human Rights Act and European Union Law on free movement of capital.
Mortgage intermediary confidence about the prospects for the mortgage market recorded a new high, according to a sentiment tracking index.


text:
Ecovillages. Co-housing. Intentional Communities.
Ecovillage Networks
Global Ecovillage Network Directory devoted to sustainable living and settlements. Search for communities worldwide.
Ecovillage Network of the Americas Western Hemisphere representative of the Global Ecovillage Network.
Fellowship of Intentional Communities
Network for a New Culture
Here are a few examples. See these links pages for more.
Earthaven Black Mountain, NC. "a growing neo-tribal ecovillage, is dedicated to caring for people and the Earth by learning, living and demonstrating holistic, sustainable culture. Culture's Edge, our non-profit education center, offers hands-on workshops and programs throughout the year which are integral to the overall vision of Earthaven as a living demonstration ecovillage.
TwinOaks "...an intentional community in rural central Virginia, made up of around 85 adult members and 15 children. Since the community's beginning in 1967, our way of life has reflected our values of cooperation, sharing, nonviolence, equality, and ecology. We welcome you to schedule a visit." News page with lots of photos gives you an idea of life at TwinOaks.
ZEGG is a German intentional community. "After many years of preparation in a forerunner community, ZEGG was founded in 1991on a 15-hectare site some 80km southwest of Berlin. ZEGG's aim is to establish an international conference and research centre working on draft models for socially and ecologically sustainable ways of living. In addition, ZEGG has become a diverse cultural centre, a meeting place for the region and a networking hub for communities and activists searching for new solutions within the fields of politics, ecology and communal living."
Ecovillages and Eco-Friendly Resources
FIC's List of Intentional Communities is an ongoing project of the Fellowship of Intentional Communities, and is probably the largest list in existence. Welcomes additions.
EcoVillage at Findhorn maintains a juicy list featuring the many facets of Eco-friendly community living.
EcoBusiness Links
How EcoVillagers Earn Their Livelihood
Earthaven's members are involved in a variety of independent enterprises which sustain body, mind and soul... including healing, wildcrafting herbs, magazine publishing, book publishing, building eco-friendly homes...and more.
The Farm is one of the earliest intentional communities. Located in rural Tennessee, they have an extensive and diverse means of livelihood: mushroom culture (indoor morels), book publishing, soy products, mail order catalog, inn, midwifery classes and more.
Twin Oaks earns their income with a variety of sustainable and eco-friendly endeavors.
Twin Oaks Indexing is a collective of experienced indexers offering high quality back-of-the-book indexing promptly, dependably, and at reasonable rates.
Twin Oaks members earn a sizeable portion of their income via a mail order catalog of hammocks and hammock furniture which they manufacture.
They also make and sell tofu, raise and sell herbs (and do workshops teaching herb culture), and write and publish books. Is it Utopia Yet is in its 3 printing.
Foundations and Organizations
Findhorn Foundation features the Ecovillage at Findhorn. A founding member of the Global Ecovillage Network.
The Earth Charter is "the product of a decade long, worldwide, cross-cultural conversation about common goals and shared values. The drafting of the Earth Charter has involved the most open and participatory consultation process ever conducted in connection with an international document. Thousands of individuals and hundreds of organizations from all regions of the world, different cultures, and diverse sectors of society have participated. The Charter has been shaped by both experts and representatives of grassroots communities. It is a people's treaty that sets forth an important expression of the hopes and aspirations of the emerging global civil society."
Co-housing Association of the United States
Grants for Ecovillages
Gaia Trust Danish-based charitable association founded in 1987 on the initiative of Ross and Hildur Jackson, with the intention of supporting the transition to a sustainable and more spiritual future society through grants and proactive initiatives. No longer accepts applications due to low funds, but provides a wealth of information and support nonetheless.
Articles About Ecovillages and Intentional Communities
What's True About Intentional Communities: Dispelling the Myths

redpajama_set_name: RedPajamaCommonCrawl


64 Bit Notebook Coming? : Appigo Inc.
Has my favorite Notebook App been abandoned? Don't see anything much on the website and I guess iOS 11 won't support 32bit apps. Please say it's coming! Thanks!
The Notebook App is no longer owned or supported by Appigo, regrettably the new owner has also pulled the app from the App Store.