text:
\section{Conclusion}
In this paper, we have proposed three novel approaches to solve the optimization problem for quantizing the network with both low-precision weights and activations.
Firstly, we have proposed the progressive quantization approach which includes two schemes.
Specifically, we have proposed a two-step training scheme, where we use the real-valued activations as an intermediate step. We have also observed that continuously quantizing from high-precision to low-precision is also beneficial to the final performance.
Moreover, we have proposed a stochastic precision strategy to significantly reduce the training complexity of progressive quantization while still improving the performance.

Furthermore, we have presented to improve the accuracy of low-precision networks with knowledge distillation.
In particular, to better take advange of the knowledge from the full-precision model, we have proposed to jointly learn the low-precision model and its full-precision counterpart. We have explored various distillation schemes and all observed improvements over the baseline.
Finally, we have combined the three training approaches to further boost the performance. We have conducted extensive experiments to justify the effectiveness of the proposed approaches on the image classification task.

\section{Experiments}  \label{exp:experiment}

To investigate the performance of the proposed methods, we conduct experiments on CIFAR-100~\cite{krizhevsky2009learning} and ImageNet~\cite{russakovsky2015imagenet}. We employ ResNet~\cite{he2016deep}, PreResNet~\cite{he2016identity} and AlexNet~\cite{krizhevsky2012imagenet} for experiments.
We use a variant of the AlexNet structure by removing dropout layers and add batch normalization after each convolutional layer and fully-connected layer. This structure is widely used in previous works~\cite{zhou2016dorefa, zhu2016trained}.

We explore the effect of the joint knowledge distillation approach in Sec.~\ref{exp:mutual}, the progressive quantization strategies in Sec.~\ref{exp:relaxed} and the stochastic precision in Sec.~\ref{exp:stochastic} in details.

To justify the robustness of the proposed approaches, we conduct experiments on various representative quantization approaches, including uniform fixed-point approach DoReFa-Net~\cite{zhou2016dorefa}, non-uniform fixed-point method LQ-Net~\cite{zhang2018lq}, as well as binary neural network approaches BiReal-Net~\cite{Liu_2018_ECCV} and Group-Net~\cite{zhuang2019structured}.
We define ``TS'', ``PP'', ``SP'' and ``KD'' to represent two-step optimization in Sec.~\ref{sec:two-stage}, progressive precision in Sec.~\ref{sec:precision}, stochastic precision in Sec.~\ref{sec:stochastic} and joint knowledge distillation in Sec.~\ref{sec:mutual}, respectively.

\subsection{Implementation details}
\noindent As in \cite{rastegari2016xnor, Cai_2017_CVPR, zhou2016dorefa, zhuang2018towards, zhuang2019structured}, we quantize the weights and activations of all convolutional layers except that the first and the last layer are kept to be in full-precision.
However, we also quantize all the layers so that the model contains complete fixed-point operations.
We label this case with a * symbol
where we also quantize the input image to 8-bit.
In all ImageNet experiments, training images are resized to $256 \times 256$, and a $224 \times 224$ crop is randomly sampled from an image or its horizontal flip, with the per-pixel mean subtracted. We do not use any further data augmentation in our implementation. We use a simple single-crop testing for standard evaluation. No bias term is utilized. More details are provided in specific sections. Our implementation is based on Pytorch.

\subsection{Effect of the distillation on quantization} \label{exp:mutual}

To investigate the effect of the joint knowledge distillation approach explained in Sec.~\ref{sec:mutual}, we explore four different training schemes to obtain a low-precision student network.

\subsubsection{Joint fine-tuning of low-precision student and full-precision teacher}  \label{exp:joint}

In this scheme, both the networks are primed with full-precision pretrained weights as initialization and are jointly optimized. We explore two network structures, including PreResNet and ResNet. When using a certain student network $M_{low}$, we use the teacher network $M_{full}$ to have either the same or larger depth. The results are reported in Table~\ref{tab:joint_ResNet} and Table~\ref{tab:joint_PreResNet}. For all these cases, we use the posterior-based training strategy as the guidance regularizer. The initial learning rate for $M_{low}$ and $M_{full}$ are set to 0.005 and 0.001, respectively. We train a maximum 30 epochs, and decay the learning rate by 10 at the 15-th and 25-th epoch. We use SGD for optimization, with batch size 256, momentum 0.9 and weight decay 1e-4. The balancing parameter $\beta$ is set to 0.5.

\noindent\textbf{Discussion}: From the results, we can observe that all our low-precision models surpass the corresponding baselines. It justifies that $M_{full}$ can provide useful auxiliary supervision to assist the convergence of $M_{low}$. Moreover, the relative improvement with ResNet is larger than that with PreResNet. To highlight, the relative Top-1 improvement w.r.t.\  2-bit ResNet-50 is 1.2\% while the PreResNet-50 counterpart is 0.5\%.
This phenomenon can be attributed that quantized ResNet is more difficult to be optimized since the skip connections are also quantized which blocks layers later in the network to access information gained in earlier layers. In this scenario, $M_{full}$ can effectively ease the training of $M_{low}$ by adapting knowledge to each other. We can also justify that keeping the skip connections to high-precision is important to maintain the performance of the low-precision network similar to~\cite{Liu_2018_ECCV, park2018precision}.

Moreover, we can come to an assumption that distillation process becomes more effective when the low-precision network is more difficult to train.
This assumption can be further proved by the experiments in Sec.~\ref{exp:scratch}.

In Table~\ref{tab:joint_PreResNet}, we experiment with PreResNet-18 which is paired with various teacher networks but with deeper layers. However, the benefit from using a deeper network saturates at some points. For example, the final trained accuracy of 2-bit PreResNet-18 model paired with PreResNet-50 is 0.3\% worse than that obtained by pairing the PreResNet-34 network.

With the simple DoReFa-Net uniform quantization strategy, we can achieve  comparable or even higher accuracy compared with the full-precision model using 4-bit precision. It means that we can deploy the 4-bit model in hardware devices with no loss of accuracy which would greatly save memory bandwidth and power consumption.

Interestingly, we also observe that the full-precision teacher can also be improved by learning together with the student. We plot the convergence curves in Figure~\ref{fig:kl_finetune}. We can observe that the teacher's performance drops at the beginning epochs due to inaccurate gradient from the student. During optimization, the student network serves as a regularizer for the teacher network which can even surpass the pretrained baseline.

\begin{table*}[tb]
	\centering
	\caption{The accuracy of the quantized ResNet using joint training approach and finetuning. W and A refer to the bitwidth of weights and activations, respectively.}
	\scalebox{1.0}
	{
	\begin{tabular}{cccccccc}
		\toprule
		Precision & & \tabincell{c}{ResNet-18 \\ Baseline} & \tabincell{c} {ResNet-18 \\ with ResNet-18}  & \tabincell{c} {ResNet-34 \\ Baseline} &
		\tabincell{c} {ResNet-34 \\ with ResNet-34}  &\tabincell{c} {ResNet-50 \\ Baseline} & \tabincell{c} {ResNet-50 \\ with ResNet-50}  \\ \midrule
	    \multirow{2}{*}{32W, 32A} &Top-1\% &69.7  &- &73.2 &- &75.6 &-  \\
	     &Top-5\% &89.0  &- &91.4 &- &92.2 &- \\ \midrule
	    \multirow{2}{*}{4W, 4A} &Top-1\% &69.4 &\bf{70.0}  &71.3 &\bf{72.9} &74.5 &\bf{75.3}  \\
	     &Top-5\% &88.9  &\bf{89.9}  &90.0  &\bf{91.3} &91.5 &\bf{91.7} \\ \midrule
	    \multirow{2}{*}{2W, 2A} &Top-1\% &64.7 &\bf{65.6}  &68.2  &\bf{69.0} &70.2 &\bf{71.4} \\
	     &Top-5\% &86.0 &\bf{86.3} &88.1 &\bf{88.6} &89.1 &\bf{90.0} \\
		\bottomrule
	\end{tabular}}
	\label{tab:joint_ResNet}
\end{table*}

\begin{table*}[ht]
	\centering
	\caption{The accuracy of the quantized PreResNet using joint training approach and finetuning.}
	\scalebox{0.8}
	{
	\begin{tabular}{cccccccccc}
		\toprule
		Precision & &\tabincell{c}{PreResNet-18 \\ Baseline} & \tabincell{c} {PreResNet-18 \\ with PreResNet-18} & \tabincell{c} {PreResNet-18 \\ with PreResNet-34}  & \tabincell{c} {PreResNet-18 \\ with PreResNet-50} &\tabincell{c} {PreResNet-34 \\ Baseline} &
		\tabincell{c} {PreResNet-34 \\ with PreResNet-34}  & \tabincell{c} {PreResNet-50 \\ Baseline}
		& \tabincell{c} {PreResNet-50 \\ with PreResNet-50}  \\ \midrule
	    \multirow{2}{*}{32W, 32A} &Top-1\% &70.0  &- &-  &- &73.5 &- &76.1 &-  \\
	     &Top-5\% &89.2 &-  &-  &- &91.3 &- &92.8  &- \\ \midrule
	    \multirow{2}{*}{4W, 4A} &Top-1\% &69.8  &\bf{70.0}  &- &- &73.6 &\bf{73.8}  &75.9  &\bf{76.4}  \\
	     &Top-5\% &89.1 &\bf{89.4} &- &- &91.3 &\bf{91.4} &92.8  &\bf{93.0} \\ \midrule
	    \multirow{2}{*}{2W, 2A} &Top-1\% &64.5  &65.2  &\bf{65.4} &65.3 &69.3 &\bf{70.0}  &71.2  &\bf{71.7}  \\
	     &Top-5\% &85.9 &86.2 &\bf{86.4}  &86.4  &89.0  &\bf{89.4}  &90.1  &\bf{90.5} \\
		\bottomrule
	\end{tabular}}
	\label{tab:joint_PreResNet}
\end{table*}

\begin{figure}[!htb]
	\centering
	\resizebox{0.8\linewidth}{!}
	{
		\begin{tabular}{c}
			\includegraphics{pdf/kl_finetune.pdf}
		\end{tabular}
	}
	\caption{Both student and teacher are fine-tuned from the pretrained models. We use PreResNet-34 as illustration.}
	\label{fig:kl_finetune}
\end{figure}

\subsubsection{Learning from scratch vs.\ fine-tuning} \label{exp:scratch}

In this scheme, we train a low-precision student from scratch given a pretrained full-precision teacher network. During training, both of the models are mutually updated. The initial learning rates for student and teacher are set to 0.1 and 0.001, respectively. We train a maximum 80 epochs with SGD, and the learning rate is decayed by 10 at 30, 50, 60 and 70 epochs. We adopt the batch size of 256.

The results are reported in Table~\ref{tab:scratch}. From the results, we can summarize two instructive statements.
\begin{itemize}
    \item
The relative improvement of KD is more apparent than those that are from fine-tuning. For instance, with 2-bit representations, the relative improvement for PreResNet-50 is 2.2\% while the fine-tuning counterpart is 0.5\% in Table~\ref{tab:joint_PreResNet}.
This is reasonable since learning from scratch is more challenging than fine-tuning and the auxiliary guidance from the teacher has more affects.
\item
Fine-tuning performs steadily better than learning from scratch. It shows that the pretrained full-precision model serves as an important initial point.

\end{itemize}
We also plot the convergence curves in Figure~\ref{fig:scratch_kl}.

\begin{table*}[!tb]
	\centering
	\caption{The accuracy of the quantized PreResNet using the joint training approach, which is learnt from scratch.}
	\scalebox{1.0}
	{
	\begin{tabular}{cccccccc}
		\toprule
		Precision & & \tabincell{c}{PreResNet-18 \\ Baseline} & \tabincell{c} {PreResNet-18 \\ with PreResNet-18} & \tabincell{c} {PreResNet-34 \\ Baseline}  & \tabincell{c} {PreResNet-34 \\ with PreResNet-34} &
		\tabincell{c} {PreResNet-50 \\ Baseline}  & \tabincell{c} {PreResNet-50 \\ with PreResNet-50} \\ \midrule
	    \multirow{2}{*}{32W, 32A} &Top-1\% &70.0  &-  &73.5  &- &76.1 &-  \\
	     &Top-5\% &89.2 &-  &91.3 &-  &92.8 &- \\ \midrule
	    \multirow{2}{*}{4W, 4A} &Top-1\%  &67.9  &\bf{69.2}  &71.5 &\bf{72.9} &73.9 &\bf{75.4}  \\
	     &Top-5\% &88.2 &\bf{88.7}  &90.1 &\bf{90.9} &91.6 &\bf{92.6} \\ \midrule
	    \multirow{2}{*}{2W, 2A} &Top-1\% &62.6 &\bf{64.8}  &66.5  &\bf{68.5} &67.2 &\bf{69.4}  \\
	     &Top-5\% &84.5 &\bf{86.0}  &87.2 &\bf{88.4} &87.7 &\bf{89.3} \\
		\bottomrule
	\end{tabular}}
	\label{tab:scratch}
\end{table*}

\begin{figure}[!htb]
	\centering
	\resizebox{0.8\linewidth}{!}
	{
		\begin{tabular}{c}
			\includegraphics{pdf/scratch_kl.pdf}
		\end{tabular}
	}
	\caption{Student is learnt from scratch while teacher is fine-tuned. PreResNet-50 is used here.}
	\label{fig:scratch_kl}
\end{figure}

\subsubsection{Learning from the fixed teacher} \label{exp:fixed}

In this section, we fix the pretrained teacher network and only fine-tune the student network. This is the scheme used by~\cite{hinton2015distilling} to train their student network.
The training details for the student network are the same as those described in Sec.~\ref{exp:joint}.

From Table~\ref{tab:fixed_teacher}, we can observe that the improvement is relatively lower than that with pretrained teachers in Table~\ref{tab:joint_PreResNet}. This proves that directly transferring the knowledge from the pretrained teacher may not be optimal or not be suitable for quantization.

Both $M_{low}$ and $M_{full}$ should be jointly optimized to adapt to each other.
However, this scheme has an advantage that one can pre-compute and store the guidance signals and access them during training $M_{low}$, which can save the forward and backward pass computations w.r.t.\  $M_{full}$.
For better understanding, we further show the convergence curves for AlexNet* on ImageNet in Figure~\ref{fig:jointornot}.

\begin{table*}[!htb]
	\centering
	\small
	\caption{The accuracy of the quantized PreResNet using fixed full-precision teacher.}
	\scalebox{1.0}
	{
	\begin{tabular}{cccccc}
		\toprule
		Precision & & \tabincell{c}{PreResNet-18 \\ Baseline} & \tabincell{c} {PreResNet-18 \\ with PreResNet-18} & \tabincell{c} {PreResNet-34 \\ Baseline}  & \tabincell{c} {PreResNet-34 \\ with PreResNet-34}  \\ \midrule
	    \multirow{2}{*}{4W, 4A} &Top-1\% &69.8  &\bf{70.0}  &73.6  &\bf{73.8}  \\
	     &Top-5\% &89.1  &\bf{89.3} &91.3 &\bf{91.5} \\ \midrule
	    \multirow{2}{*}{2W, 2A} &Top-1\% &64.5  &\bf{64.9}  &69.3  &\bf{69.7}  \\
	     &Top-5\% &85.9  &\bf{86.2}  &89.0  &\bf{89.2} \\
		\bottomrule
	\end{tabular}}
	\label{tab:fixed_teacher}
\end{table*}

\begin{figure}[!htb]
	\centering
	\resizebox{0.8\linewidth}{!}
	{
		\begin{tabular}{c}
			\includegraphics{pdf/jointornot.pdf}
		\end{tabular}
	}
	\caption{Joint training vs. fixed teacher using AlexNet* on ImageNet.}
	\vspace{-1em}
	\label{fig:jointornot}
\end{figure}

\subsubsection{Hint-based vs.\  posterior-based distillation}
\label{exp:hintvsposterior}
In this scheme, we further explore the effect of different knowledge distillation strategies as introduced in Sec.~\ref{sec:mutual}. The results are reported in Table~\ref{tab:hintvsposterior}.
We analyze the sensitivity of the MSE loss position to the final performance. We add the hints at the end of the last block or last two blocks.
For hint-based distillation, we set the optimization details the same as those for posterior-based distillation in Sec.~\ref{exp:joint} except that the balancing parameter $\lambda$ is set to 0.1.
For hint-based training, we observe that the position of the hints can affect the performance to some extent. For example, adding two hints outperforms the one-hint counterpart by 0.7\%.

\begin{table}[!htb]
	\centering
	\small
	\caption{Hint-based vs. posterior-based with ResNet-18 on ImageNet.}
	\scalebox{0.8}
	{
	\begin{tabular}{cccccc}
		\toprule
		Precision & & \tabincell{c}{ResNet-18 \\ Baseline} & \tabincell{c} {ResNet-18 \\ posterior} & \tabincell{c} {ResNet-18  \\ two hints}  & \tabincell{c} {ResNet-18 \\ one hint}  \\ \midrule
	    \multirow{2}{*}{2W, 2A} &Top-1\% &64.7  &65.6  &\bf{65.7}  &65.0  \\
	     &Top-5\% &86.0  &86.3  &\bf{86.4}  &86.1 \\
		\bottomrule
	\end{tabular}}
	\label{tab:hintvsposterior}
\end{table}

\subsection{Effect of progressive quantization} \label{exp:relaxed}
In this part, we explore the effect of the proposed progressive quantization methods.

\subsubsection{Effect of the two-step optimization} \label{exp:two-step}
We further analyze the effect of each stage in the two-step approach in Figure~\ref{fig:two_step}.
We take the 2-bit ResNet-50 on ImageNet as example. In Figure~\ref{fig:two_step}, step-1 has the minimal loss of accuracy. As for the step-2, although it incurs apparent accuracy decrease in comparison with that of the step-1, its accuracy is consistently better than the results of baseline in every epoch. This illustrates that progressively seeking for the local minimum point is crucial for final better convergence, which proves the effectiveness of this simple mechanism.

\begin{figure}[ht]
	\centering
	\resizebox{0.8\linewidth}{!}
	{
		\begin{tabular}{c}
			\includegraphics{pdf/two_step.pdf}
		\end{tabular}
	}
	\caption{The two-step training approach on ResNet-50.}
	\label{fig:two_step}
\end{figure}

\begin{table}[!htb]
	\centering
	\caption{Accuracy (\%) of different comparing methods on the ImageNet validation set. All the cases keep skip connections during testing.}
	\scalebox{1.0}
	{
		\begin{tabu}{c c | c | c c}
		  Precision & model &	method &top-1 acc. &top-5 acc. \\
			\tabucline[1pt]{-}
		\multirow{4}{*}{2W, 2A} &\multirow{4}{*}{ResNet-50} &Baseline  &70.2  &89.1  \\
		  &	& Baseline + TS  &70.9  &90.0  \\
		  &	& Baseline + PP &70.8  &90.0 \\
		  & & Baseline + TS + PP &\bf{71.1}  &\bf{90.1}  \\
		  \hline
	     \multirow{4}{*}{4W, 4A} &\multirow{4}{*}{ResNet-50*} &Baseline  &75.1  &75.7  \\
		  &	& Baseline + TS  &75.3  &91.9  \\
		  &	& Baseline + PP &75.4  &91.8 \\
		  & & Baseline + TS + PP &\bf{75.5}  &\bf{92.0} \\
		  \hline
		\multirow{4}{*}{2W, 2A} &\multirow{4}{*}{ResNet-50*} &Baseline  &67.7  &70.0  \\
		  &	& Baseline + TS  &69.2  &87.0  \\
		  &	& Baseline + PP &68.8  &86.9 \\
		  & & Baseline + TS + PP &\bf{69.4}  &\bf{87.0} \\
		  \hline
		 \multirow{4}{*}{4W, 4A} &  \multirow{4}{*}{AlexNet*} &Baseline  &56.2  &79.4 \\
		  &	&Baseline + TS &57.7  &81.0 \\
		  &	&Baseline + PP &57.5  &80.8 \\
		  & &Baseline + TS + PP &\bf{57.8}  &\bf{80.8} \\
		  \hline
		 \multirow{4}{*}{2W, 2A} &  \multirow{4}{*}{AlexNet*} &Baseline  &48.3  &71.6  \\
		  &	&Baseline + TS &50.7  &74.9 \\
		  &	&Baseline + PP &50.3  &74.8 \\
		  & &Baseline + TS + PP &\bf{50.9}  &\bf{74.9} \\
	\end{tabu}}
	\vspace{-1.0em}
	\label{tab:exp_twostep_progressive}
\end{table}

\subsubsection{Effect of the progressive precision strategy}  \label{exp:progressive}
What is more, we also separately explore the progressive precision effect on the final performance.
In this experiment, we apply AlexNet and ResNet-50 on the ImageNet dataset.
We continuously quantize both weights and activations simultaneously from 32-bit$\to$8-bit$\to$4-bit$\to$2-bit and explicitly illustrate the accuracy change process for each precision in Figure~\ref{fig:progressive}. The quantitative results are also reported in Table~\ref{tab:exp_twostep_progressive}. From the figure we can find that for the 8-bit and 4-bit, the low-bit model has no accuracy loss with respect to the full precision model. However, when quantizing from 4-bit to 2-bit, we can observe significant accuracy drop.
Despite this, we still observe $2.0\%$ relative improvement by comparing the Top-1 accuracy over the 2-bit baseline, which proves the effectiveness of the proposed strategy. It is worth noting that the accuracy curves become more unstable when quantizing to lower bit. This phenomenon is reasonable since the precision becomes lower, the quantized value will change more frequently during training.

\begin{figure}[!htb]
	\centering
	\resizebox{0.8\linewidth}{!}
	{
		\begin{tabular}{c}
			\includegraphics{pdf/progressive.pdf}
		\end{tabular}
	}
	\caption{The progressive training approach on AlexNet*.}
	\label{fig:progressive}
	\vspace{-1.0em}
\end{figure}

\subsection{Effect of the stochastic precision}   \label{exp:stochastic}
In this subsection, we further explore the effect of the stochastic precision strategy on general quantization approaches. The default structure of the fragment $f$ is a residual block and we stochastically quantize weights and activations in all cases unless special explanations.
The results are reported in Table~\ref{tab:exp_sp}. By combining the baseline methods with \emph{SP}, we find apparent performance increase compared with the baselines in all cases. During training, we stochastically keep a portion of network to full-precision and update by the standard gradient-based method. This strategy shares the similar spirit with the progressive quantization to relax the discrete quantizer effectively. Moreover, the proposed stochastic strategy only requires one training stage without fine-tuning the model in many training rounds.

\begin{table}[!htb]
	\centering
	\caption{Accuracy (\%) of different comparing methods with SP on the ImageNet validation set.}
	\scalebox{1.0}
	{
		\begin{tabu}{c | c | c c}
		    model &	method &top-1 acc. &top-5 acc. \\
			\tabucline[1pt]{-}
			\multirow{2}{*}{ResNet-50} & DoReFa-Net (2-bit)  &70.2  &89.1  \\
			& DoReFa-Net + SP  &\bf{72.2}  & \bf{90.8}  \\
			\hline
		    \multirow{2}{*}{ResNet-50} & LQ-Net (3-bit)  &74.2  &91.6 \\
			& LQ-Net + SP &\bf{75.1}  &\bf{92.3} \\
			\hline
			\multirow{2}{*}{ResNet-18} & BiReal-Net &56.4  &79.5 \\
			& BiReal-Net + SP & \bf{58.8}  &\bf{81.2} \\
			\hline
			\multirow{2}{*}{ResNet-18} & GroupNet (5 bases) &64.8  &85.7 \\
			& GroupNet + SP &\bf{65.9}  &\bf{86.3} \\

	\end{tabu}}
	\label{tab:exp_sp}
\end{table}

\begin{figure}[!htb]
	\centering
	\resizebox{0.8\linewidth}{!}
	{
		\begin{tabular}{c}
			\includegraphics{pdf/SP.pdf}
		\end{tabular}
	}
	\caption{The stochastic precision training approach on
	ResNet-50.}
	\label{fig:SP}
\end{figure}

\subsubsection{Effect of different SP policies}   \label{exp:different_SP}
We further explore the influence of different choices of the fragment $f$ described in Sec.~\ref{sec:stochastic} as well as the extent of randomness.
We treat GroupNet as our baseline approach and utilize 5 binary bases. The results are reported in Table~\ref{tab:exp_stochastic}.
We explore two different structures of $f$, including one convolutional layer and one residual block which corresponds to \emph{layerdrop} and \emph{blockdrop} respectively. We further incorporate the randomness of quantizing weights and activations into $f$ and is denoted by \emph{W/A}. From the results, we can find that all the four cases show improved performance compared with the baseline, which justifies adding randomness is a general way for relaxing the low-precision network training.
By comparing the result of \emph{layerdrop+W/A} with \emph{layerdrop}, we can observe performance drop with the increase of randomness. However, \emph{blockdrop+W/A} performs slightly better than \emph{blockdrop}. This shows that adding excessive stochasticity can make the gradient updating direction deviate while appropriate extent of randomness can relax the non-differentiable problem to facilitate optimization. Moreover, the accuracy of \emph{layerdrop} and \emph{blockdrop} are very close, which shows that the structure of $f$ is not sensitive to the final performance.

\begin{table}[!htb]
	\centering
	\caption{Accuracy (\%) of different stochastic policies on the ImageNet validation set.}
	\scalebox{1.0}
	{
		\begin{tabu}{c | c | c c}
		    model &	method &top-1 acc. &top-5 acc. \\
			\tabucline[1pt]{-}
			\multirow{5}{*}{ResNet-18} & GroupNet (5 bases) &64.8  &85.7  \\
		    & GroupNet + blockdrop &65.6  &86.3 \\
            & GroupNet + layerdrop &65.7  &86.5 \\
            & GroupNet + blockdrop + W/A &\bf{65.9}  &\bf{86.6}  \\
            & GroupNet + layerdrop + W/A &65.0  &86.1  \\
	\end{tabu}}
	\label{tab:exp_stochastic}
\end{table}

\subsection{Effect of quantizing all layers} \label{exp:all_layers}
In this part, we further explore the effect of quantizing the first convolution layer and the last classification layer to the final performance. We report the performance in Tables~\ref{tab:exp_twostep_progressive} \& \ref{tab:whole_ResNet50_imagenet} \& \ref{tab:whole_AlexNet_cifar} \& \ref{tab:whole_AlexNet_imagenet}. With ``2W, 2A'', we can observe that the performance of ResNet-50 beats ResNet-50* by a large margin. This shows that keeping the first and the last layer to high-precision is crucial to preserve the quantized model accuracy.
Moreover, the proposed advanced training approaches improves the baseline significantly. For 2-bit precision, the gap between ``ResNet-50* TS+PP+KD'' and baseline* is 2.3\% while ``ResNet-50 TS+PP+KD'' improves baseline by 1.2\%. It further justifies the claim in Sec.~\ref{exp:joint} and Sec.~\ref{exp:scratch} that the proposed training algorithms can be more effective when the model is more challenging to be optimized.

\subsection{Combining different training strategies}
Finally, we come to our complete approach by combining TS, PP, SP and KD.
We first combine TS, PP with KD and the results are shown in Tables~\ref{tab:whole_ResNet50_imagenet} \& \ref{tab:whole_AlexNet_imagenet}.
Moreover, we also combine the one-stage SP strategy with KD and the full results are reported in Tables~\ref{tab:whole_ResNet} \& \ref{tab:whole_PreResNet} \& \ref{tab:whole_AlexNet_cifar}.

We can observe that the proposed approaches can benefit with each other and further improve the performance on all settings. For instance, with ``2W, 2A" in Table~\ref{tab:whole_ResNet}, we find a 2.4\% relative gap between the baseline on ResNet-50. Even with the basic quantizer in DoReFa-Net, the difference in Top-1 error is only 3\%.
This strongly justifies that the proposed joint knowledge distillation and the stochastic precision are general training approaches for improving low-bit neural networks.

\begin{table*}[!htb]
	\centering
	\caption{Accuracy (\%) of ResNets on the ImageNet validation set using SP and KD.}
	\scalebox{1.0}
	{
	\begin{tabular}{cccccccc}
		\toprule
		Precision & &\tabincell{c}{ResNet-18 \\ Baseline} & \tabincell{c} {ResNet-18 \\ with ResNet-18}  & \tabincell{c} {ResNet-34 \\ Baseline} &
		\tabincell{c} {ResNet-34 \\ with ResNet-34}  &\tabincell{c} {ResNet-50 \\ Baseline} & \tabincell{c} {ResNet-50 \\ with ResNet-50}  \\ \midrule
	    \multirow{2}{*}{32W, 32A} &Top-1\% &69.7  &- &73.2 &- &75.6 &-  \\
	     &Top-5\% &89.0  &- &91.4 &- &92.2 &- \\ \midrule
	    \multirow{2}{*}{2W, 2A} &Top-1\% &64.7 &\bf{65.9}  &68.2  &\bf{70.1} &70.2 &\bf{72.6} \\
	     &Top-5\% &86.0 &\bf{87.0} &88.1 &\bf{89.6} &89.1 &\bf{90.9}  \\
		\bottomrule
	\end{tabular}}
	\label{tab:whole_ResNet}
\end{table*}

\begin{table*}[!htb]
	\centering
	\caption{Accuracy (\%) of PreResNets on the ImageNet validation set with SP and KD.}
	\scalebox{1.0}
	{
	\begin{tabular}{cccccccc}
		\toprule
		Precision & & \tabincell{c}{PreResNet-18 \\ Baseline} & \tabincell{c} {PreResNet-18 \\ with ResNet-18}  & \tabincell{c} {PreResNet-34 \\ Baseline} &
		\tabincell{c} {PreResNet-34 \\ with PreResNet-34}  &\tabincell{c} {PreResNet-50 \\ Baseline} & \tabincell{c} {PreResNet-50 \\ with PreResNet-50}  \\ \midrule
	    \multirow{2}{*}{32W, 32A} &Top-1\% &70.0   &- &73.5  &- &76.1  &-  \\
	     &Top-5\% &89.2  &- &91.3  &- &92.8  &- \\ \midrule
	    \multirow{2}{*}{2W, 2A} &Top-1\% &64.5  &\bf{65.7}  &69.3  &\bf{70.3}  &71.2 &\bf{72.3} \\
	     &Top-5\% &85.9 &\bf{86.6}  &89.0 &\bf{89.5}  &90.1 &\bf{90.7}  \\
		\bottomrule
	\end{tabular}}
	\label{tab:whole_PreResNet}
\end{table*}

\begin{table}[!htb]
	\centering
	\caption{Accuracy (\%) of ResNet-50 and ResNet-50* on the ImageNet with TS, PP and KD.}
	\scalebox{0.8}
	{
	\begin{tabular}{cccccc}
		\toprule
		Precision & & \tabincell{c}{Baseline*} &\tabincell{c} {ResNet-50* \\ TS + PP+ KD} &Baseline & \tabincell{c} {ResNet-50 \\ TS + PP + KD}  \\ \midrule
	    \multirow{2}{*}{32W, 32A} &Top-1\% &75.6  &- &- &- \\
	     &Top-5\% &92.2  &- &- &- \\ \midrule
	     \multirow{2}{*}{4W, 4A} &Top-1\% &75.1 &\bf{75.7} &74.5 &\bf{75.3} \\
	     &Top-5\% &91.9  &\bf{92.0} &91.5 &\bf{91.7} \\ \midrule
	    \multirow{2}{*}{2W, 2A} &Top-1\% &67.7  &\bf{70.0} &70.2  &\bf{71.4} \\
	     &Top-5\% &84.7  &\bf{87.5} &89.1 &\bf{90.0} \\
		\bottomrule
	\end{tabular}}
	\label{tab:whole_ResNet50_imagenet}
\end{table}

\begin{table}[!htb]
	\centering
	\caption{Accuracy (\%) of AlexNet* on the CIFAR-100 with SP and KD.}
	\scalebox{1.0}
	{
	\begin{tabular}{cccc}
		\toprule
		Precision & & \tabincell{c}{AlexNet* \\ Baseline} & \tabincell{c} {AlexNet* \\ SP + KD}  \\ \midrule
	    \multirow{2}{*}{32W, 32A} &Top-1\% &65.4  &-  \\
	     &Top-5\% &88.3  &- \\ \midrule
	    \multirow{2}{*}{2W, 2A} &Top-1\% &63.9  &\bf{64.9}  \\
	     &Top-5\% &87.6 &\bf{88.1}  \\
		\bottomrule
	\end{tabular}}
	\label{tab:whole_AlexNet_cifar}
\end{table}

\begin{table}[!htb]
	\centering
	\caption{Accuracy (\%) of AlexNet* on the ImageNet with TS, PP and KD.}
	\scalebox{1.0}
	{
	\begin{tabular}{ccccc}
		\toprule
		Precision & & \tabincell{c}{AlexNet* \\ Baseline} &\tabincell{c} {AlexNet* \\ TS + KD} & \tabincell{c} {AlexNet* \\ TS + PP + KD}  \\ \midrule
	    \multirow{2}{*}{32W, 32A} &Top-1\% &57.2  &- &-  \\
	     &Top-5\% &80.3  &- &- \\ \midrule
	     \multirow{2}{*}{4W, 4A} &Top-1\% &56.8  &57.8  &\bf{58.0} \\
	     &Top-5\% &80.0  &80.9  &\bf{81.1} \\ \midrule
	    \multirow{2}{*}{2W, 2A} &Top-1\% &48.8  &51.1  &\bf{51.6} \\
	     &Top-5\% &72.2  &75.3  &\bf{76.2} \\
		\bottomrule
	\end{tabular}}
	\label{tab:whole_AlexNet_imagenet}
\end{table}

\section{Introduction}

State-of-the-art deep neural networks ~\cite{krizhevsky2012imagenet, simonyan2014very, he2016deep} usually involve millions of parameters and need billions of FLOPs %
for training and inference.
The significant memory consumption and computational cost can %
make it intractable to deploy models to
 mobile, embedded hardware devices.
 %
 To improve the computational and memory efficiency, various solutions have been proposed, including network pruning \cite{zhuang2018discrimination, he2017channel, li2017pruning}, low rank approximation of weights \cite{kim2015compression, zhang2016accelerating}, training a low-precision network \cite{zhou2017incremental, courbariaux2015binaryconnect, zhu2016trained, zhou2016dorefa} and efficient architecture design~\cite{guo2019single, howard2017mobilenets, zhang2017shufflenet}. In this work, we follow the idea of training a low-precision network and our focus is to improve the training process of such a network.
Thus, our work targets the problem of training network with both extremely low-bit weights and activations.

The solutions proposed in this paper contain three components. They can be applied independently or jointly. The first component is the progressive quantization which consists of two schemes.
The first strategy is to adopt a two-stage training process. At the first stage, only the weights of a network is quantized. After obtaining a sufficiently good solution of the first stage, the activation of the network is further required to be in low-precision and the network
is
trained again. Essentially, this two-step approach first solves a related sub-problem, i.e., training a network with only low-bit weights and the solution of the sub-problem provides a good initial point for training our target problem. Following the similar idea, we propose our second scheme by performing progressive training on the bit-width aspect of the network. Specifically, we incrementally train a serial of networks with the quantization bit-width (precision) gradually decreased from full-precision to the target precision.

However, the above progressive quantization needs several retraining steps which introduces additional training burdens. To solve this problem, we further propose our second component termed
stochastic precision to effectively combine these two strategies into only one training stage. Inspired by dropout strategies~\cite{srivastava2014dropout, huang2016deep}, we randomly select a portion of the model (i.e., layers, blocks) and activations or weights to quantize while keeping other parts full-precision.
This way we effectively improve the gradient flow for training quantized neural networks.

The third component is inspired by the recent progress of  information distillation \cite{romero2014fitnets, hinton2015distilling, parisotto2016actor, zagoruyko2016paying, ba2014deep}. The basic idea of those works is to train a target network alongside another guidance network. For example, the works in \cite{romero2014fitnets, hinton2015distilling, parisotto2016actor, zagoruyko2016paying, ba2014deep} propose to train a small student network to mimic the deeper or wider teacher network. They add an additional regularizer by minimizing the difference between student's and teacher's posterior probabilities~\cite{hinton2015distilling} or intermediate feature representations~\cite{ba2014deep, romero2014fitnets}. It is observed that by using the guidance of the teacher model, better performance can be obtained with the student model than directly training the student model on the target problem.  Motivated by these observations, we propose to train a full-precision network alongside the target low-precision network.
In our work, the student network has the similar topology as that of the teacher network, except that the student network is low-precision while the teacher network keeps full-precision operations. Moreover, in contrast to standard knowledge distillation methods, we allow the teacher network to be jointly optimized with the student network rather than being fixed since we discover that this treatment enables the two networks adjust better to each other. Interestingly, the performance of both the full-precision teacher and the low-precision student can be improved.

The contributions of this paper can be summarized as follows:
\begin{itemize}
    \item  We propose two progressive quantization schemes for tackling the non-differentiability of quantization operations during training. In the first scheme, we propose a two-step training manner, where the weights are first quantized to serve as a good initialization on further quantizing activations. In the second scheme, we progressively reduce the bit-width during training to find better local minima.

    \item To reduce the extra training burden, we introduce structured stochastic training,
    %
    leading to an effective, simplified one-stage training approach.

    \item To  our knowledge, we are the first to propose to improve the low-precision network training using knowledge distillation technique where the full-precision teacher and the quantized student are jointly optimized to adapt to each other. We explore different distilling schemes in Sec.~\ref{exp:experiment} and all produce improved accuracy for the low-precision model.

    \item We conduct extensive experiments under various precision for variants of ResNet architectures on the image classification task.

\end{itemize}

This paper extends the preliminary conference version~\cite{zhuang2018towards} in several aspects. 1) Even though the multi-stage progressive quantization in~\cite{zhuang2018towards} improves the performance clearly,
the multiple re-initialization and fine-tuning steps
make the training complex and introduce computation overhead.
To solve this problem, we propose a one-stage stochastic precision strategy that enjoys the advantage of the multi-stage progressive quantization.
2) We extend the hint-based joint knowledge distillation to posterior-based scheme, which produces a more comprehensive framework.
3) We  now conduct extensive experiments on ImageNet over various architectures to formulate strong and comprehensive baselines for future work. We study several schemes which produce low-precision networks using different distilling strategies and provide interesting analysis.

\section*{Acknowledgments}
\else
  %
  \section*{Acknowledgment}
\fi

L. Liu was in part supported by ARC DECRA Fellowship DE170101259.
M. Tan was partially supported by National Natural Science Foundation of China (NSFC) 61602185, Program for Guangdong Introducing Innovative and Enterpreneurial Teams 2017ZT07X183.

\ifCLASSOPTIONcaptionsoff
  \newpage
\fi

\bibliographystyle{IEEEtran}

\section{Methods}

In this section, we  first describe the progressive quantization schemes in Sec.~\ref{sec:progressive}. Then we %
explain the stochastic precision approach in Sec.~\ref{sec:stochastic}
Then we %
elaborate the joint knowledge distillation in Sec.~\ref{sec:mutual}.

\subsection{Progressive quantization} \label{sec:progressive}

\subsubsection{Two-step optimization}\label{sec:two-stage}
With the straight-through estimator, it is possible to directly optimize the low-precision network. However, the gradient approximation of the quantization function inevitably introduces noisy signal for updating network parameters. Strictly speaking, the approximated gradient may not be the right updating direction. Thus, the training process can be more likely to get trapped at a poor local minimal than training a full precision model. Applying the quantization function to both weights and activations further worsens the situation.

To %
alleviate
this training difficulty, we devise a two-stage optimization procedure as follows.
At the first stage, we only quantize the weights of the network while setting the activations to be full precision. After the converge (or after certain number of iterations) of this model, we further apply the quantization function on the activations as well and retrain the network. Essentially, the first stage of this method is a related sub-problem of the target one. Compared to the target problem, it is easier to optimize since it only introduces quantization function on weights. Thus, we are more likely to arrive at a good solution for this sub-problem. Then, using it to initialize the target problem may help the network avoid poor local minima which are likely to be encountered if we train the network from scratch.

Let $M_{low}^{K}$ be the high-precision model with $K$-bit. We propose to learn a low-precision model $M_{low}^{k}$ in a two-stage manner with $M_{low}^{K}$ serving as the initial point, where $k<K$.
 The detailed algorithm is shown in Algorithm \ref{algo:two-stage}.

\begin{algorithm}[tb]
	\KwIn{Training data $\{ ({{\bf{x}}_i},y_i)\}_{i=1}^N$; A $K$-bit precision model $M_{low}^K$.}
	\KwOut{A low-precision deep model $M^k_{low}$ with weights ${{\bf{W}}_{low}}$ and activations being quantized into $k$-bit.}

	\textbf{Stage 1}: Quantize ${{\bf{W}}_{low}}$:\\
	\For{ $\mathrm{epoch} = 1,...,L$}
	{
		\For{ $i = 1,...N$}
		{
			Randomly sample a mini-batch data;\\
			Quantize the weights ${{\bf{W}}_{low}}$ into $k$-bit by calling some quantization methods with $K$-bit activations\;
		}
	}
	\textbf{Stage 2}: Quantize activations:\\
	Initialize ${{\bf{W}}_{low}}$ using the converged $k$-bit weights from \textbf{Stage 1} as the starting point; \\
	\For{ $\mathrm{epoch} = 1,...,L$}
	{
		\For{ $i = 1,...N$}
		{
			Randomly sample a mini-batch data;\\
			Quantize the activations into $k$-bit  by calling some quantization methods while keeping the weights to $k$-bit;
		}
	}
	\caption{Two-stage optimization for $k$-bit quantization}
	\label{algo:two-stage}
\end{algorithm}

\subsubsection{Progressive precision} \label{sec:precision}

The aforementioned two-stage optimization approach suggests the benefits of using a relatively easy-to-optimize problem to find a good initialization. However, separating the quantization of weights and activations is not the only solution to implement the above idea. In this paper, we also propose a second  scheme which progressively lowers  the bitwidth of the quantization during the course of network training.
Specifically, we progressively conduct the quantization from higher precisions to lower precision (e.g., 32-bit $\to$ 16-bit $\to$ 4-bit $\to$ 2-bit). The model of higher precision will be used as the starting point of the relatively lower precision,  analogously  to annealing.

Let $\{{b_1},...,{b_n}\}$ be a  sequence precision, where  $b_n<b_{n-1}, ..., b_2<{b_1}$, $b_n$ is the target precision and $b_1$ is set to 32 by default. The whole progressive optimization procedure is summarized in Algorithm~\ref{algo:progressive optimization}.

 Let $M_{low}^{k}$ be the low-precision model with $k$-bit and $M_{full}$ be the full precision model. In each step, we propose to learn $M_{low}^{k}$, with the solution in the $(i-1)$-th step, denoted by $M_{low}^{K}$, serving as the initial point, where $k<K$.

 \begin{algorithm}[bt]
	\KwIn{Training data $\{ ({{\bf{x}}_j},y_j)\}_{j=1}^N$; A pre-trained 32-bit full-precision  model ${M_{full}}$ as baseline; the precision sequence $\{{b_1},...,{b_n}\}$ where $b_n<b_{n-1}, ..., b_2<{b_1} = 32$.}
	\KwOut{A low-precision deep model $M_{low}^{b_n}$.}
	Let $M_{low}^{b_1}=M_{full}$, where $b_1 = 32$\;
	\For{ $i = 2,...N$}
	{
		Let $k = b_i$ and $K=b_{i-1}$\;
		Obtain $M_{low}^{k}$ by calling some quantization methods with $M_{low}^{K}$  being the input\;
		%
		%
		%
	}
	\caption{Progressive precision for accurate CNNs with low-precision weights and activations}
	\label{algo:progressive optimization}
\end{algorithm}

\subsection{Stochastic precision} \label{sec:stochastic}

There is still a problem %
in the proposed two-stage optimization and the progressive precision schemes, which gradually quantizes the network to low-precision in multi-round training stages. However, the efficiency of training is
sacrificed due to the sequentially re-initialization and fine-tuning. To solve this problem, we further propose to improve the progressive quantization strategy into a single stage. Inspired by the recent studies that incrementally or stochastically quantize a certain part of the network, we propose to incorporate the stochasticity into the progressive training by employing a stochastic precision (SP) strategy.

The term ``stochastic structure'' means that we %
randomly choose a network structural component, namely, layers, blocks, activations or weights to quantize and keep the rest to be full precision. The specific scheme is elaborated as follows.

Suppose that we decompose the low-precision network $M_{low}$ into $Z$ fragments $M_{low}=\{{m_1},...,{m_Z}\}$, where ${m_i}$ can be any structure such as a convolutional layer or a residual block.
For each iteration, we intend to partition the fragments into two sets, a low-precision set ${G_q} = \{ {m_{q1}},...,{m_{q{N_q}}}\}$ and a full-precision set ${G_r} = \{ {m_{r1}},...,{m_{r{N_r}}}\}$, which satisfies the condition:
\begin{equation}
   {G_q} \cup {G_r} = M_{low},\ \mathrm{and}\ {G_q} \cap {G_r} = \emptyset.
\end{equation}
where $N_q$ and $N_r$ are the number of elements in two sets respectively.

In our method, we randomly partition $M_{low}$ into $G_q$ and $G_r$. This is implemented by introducing a binary indicator ${\bf{b}} \in {\mathbb{R}^N}$. We randomly set ${\bf{b}}(i) = 1$ with probability $(1-\delta)$, and if ${\bf{b}}(i) = 1$ the $i$-th fragment
is
quantized and otherwise %
are
kept to be in full precision. We linearly decrease $\delta$ to 0 to ensure the whole network being quantized in the end. Note that this procedure implicitly achieves the effect of~\cite{zhou2017incremental} but without the need of multi-round training.

To further increase the randomness in quantizing $m$, we can stochastically choose whether to quantize weights or activations or both of them. This can be implemented by randomly sample a binary indicator matrix ${\bf{B}} \in {\mathbb{R}^{N \times 2}}$, where its first column is used to decide whether to quantize the weights in the corresponding fragment and the second column is used to decide whether to quantize activations respectively.

As a result, $G_q$ can be further partitioned into three subsets $\{{G_{qwa}},{G_{qw}},{G_{qa}}\}$, which represents quantizing both weights and activations, only quantizing weights and only quantizing activations, respectively.
Thus,
SP can share the advantage of the progressive training in Sec.~\ref{sec:two-stage} and Sec.~\ref{sec:precision}.

Moreover, in Sec.~\ref{exp:different_SP}, we will explore the effect of different structure choices of $m$ as well as the extent of randomness to the final performance.

\begin{algorithm}[tb]
		\KwIn{Training data $\{ {{\bf{x}}^t},{{\bf{y}}^t}\}$; low-precision network $M_{low}$ with parameters ${\mathbf W}_{low}$; stochastic ratio $\delta^t$ and decay rate $ \mu$.
		}
		\KwOut{Updated parameters ${\mathbf W}_{low}$; stochastic ratio $\delta ^ {t+1}$.}
		Partition $M_{low}$ into $N$ fragments $\{{f_1},...,{f_N}\}$; \\
		\If{ $\delta^t > 0$}
		{ Obtain the binary indicator matrix ${\bf{B}}^t$ via uniform sampling with probability $\delta^t$; \\
		Partition the network $M_{low}$ into quantized set $\{ {G_{qwa}},{G_{qw}},{G_{qa}}\}$ and full-precision set $G_r$ according to ${\bf{B}}^t$; \\
		Obtain the mixed-precision parameter set $\widetilde Q^t = \{ q({{\mathbf W}_{qwa}}),q({{\mathbf W}_{qw}}),{{\mathbf W}_{qa}},{{\mathbf W}_r}\}$ accordingly;
	    }

	    \Else{$\widetilde Q^t = q({\mathbf W}_{low}) $;}

		${{\bf{\widetilde{y}}}^t} = \mathrm{Forward}({{\bf{x}}^t},\widetilde Q^t, {\bf{B}}^t)$; \\
		Compute the loss ${\cal L}({{\bf{y}}^t},\widetilde {{\bf{y}}}^t) $; \\
		$\frac{{\partial {\cal L}}}{{\partial {\widetilde Q^t}}} = \mathrm{Backward}(\frac{{\partial {\cal L}}}{{\partial {{\bf{\widetilde{y}}}^t}}},{\widetilde Q^t}, {\bf{B}}^t)$; \\
	    Update parameters ${\mathbf W}_{low}$ using Adam; \\
	    $\delta^{t+1} = \delta^t  - \mu$;
		\caption{Stochastic precision training algorithm.}
		\label{algo:stochastic}
	\end{algorithm}

\subsection{Joint knowledge distillation on quantization}\label{sec:mutual}

\begin{figure*}[tb]
	\centering
	\resizebox{0.9\linewidth}{!}
	{
		\begin{tabular}{c}
			\includegraphics{pdf/1.pdf}
		\end{tabular}
	}
	\caption{Demonstration of the guided training strategy. We use the residual network structure for illustration.}
	\label{fig:knowledge_transfer}
\end{figure*}

The third approach proposed in this
work here
is inspired by the success of using information distillation ~\cite{romero2014fitnets, hinton2015distilling, parisotto2016actor, zagoruyko2016paying, ba2014deep} to train a relatively shallow network. Specifically, these methods usually use a teacher model (usually a pretrained deeper network) to provide guided signal for the shallower network. Following this spirit, we propose to train the low-precision network alongside another guidance network. Unlike the work in \cite{romero2014fitnets, hinton2015distilling, parisotto2016actor, zagoruyko2016paying, ba2014deep}, the guidance network shares the similar architecture as the target network but is pretrained with full-precision weights and activations.

However, a pre-trained model may not be necessarily optimal or may not be suitable for quantization. As a result, directly using a fixed pretrained model to guide the target network may not produce the best guidance signals. To mitigate this problem, we do not fix the parameters of a pretrained full precision network as in the previous work \cite{zhang2017deep}.

By using the guidance training strategy, we assume that there exists some full-precision models with good generalization performance, and an accurate low-precision model can be obtained by directly performing the quantization on those full-precision models. In this sense, the feature maps of the learned low-precision model should be close to that obtained by directly performing quantization on the full-precision model. To achieve this, essentially, in our learning scheme, we can jointly train the full-precision and low-precision models. This allows these two models adapt to each other. We even find by doing so the performance of the full-precision model can be slightly improved in some cases.

Formally, let ${{\bf{W}}_{full}}$ and ${{\bf{W}}_{low}}$ be the weights of the full-precision model and low-precision model, respectively. Then we employ two distillation strategies: hint-based training and posterior-based training.

\textbf{{Hint-based training}}   \hspace{1em}
Let $\mu ({\bf{x}};{{\bf{W}}_{{full}}})$ and $\nu ({\bf{x}};{{\bf{W}}_{{low}}})$ be the nested feature maps (e.g., activations) of the full-precision model and low-precision model, respectively. To create the guidance signal, we may require that the nested feature maps from the two models should be similar. However,  $\mu ({\bf{x}};{{\bf{W}}_{{full}}})$ and $\nu ({\bf{x}};{{\bf{W}}_{{low}}})$  is usually not directly comparable since one is full precision and the other is low-precision.
To link these two models,  we can directly quantize the weights and activations of the full-precision model. For simplicity, we denote the quantized feature maps by  $Q(\mu ({\bf{x}};{{\bf{W}}_{{full}}}))$. Thus, $Q(\mu ({\bf{x}};{{\bf{W}}_{{full}}}))$ and  $\nu ({\bf{x}};{{\bf{W}}_{{low}}})$ will become comparable. Then we can define the guidance loss as:
\begin{equation}
	R({{\bf{W}}_{full}},{{\bf{W}}_{low}}) = \frac{1}{2}\parallel Q(\mu ({\bf{x}};{{\bf{W}}_{{full}}})) - \nu ({\bf{x}};{{\bf{W}}_{{low}}}){\parallel^2},
\end{equation}
where $\parallel\cdot\parallel$ denotes some proper norms.

Let ${L_{{\theta _1}}}$ and ${L_{{\theta _2}}}$ be the cross-entropy classification losses for the full-precision and low-precision model, respectively. The guidance loss will be added to ${L_{{\theta _1}}}$ and ${L_{{\theta _2}}}$, respectively, resulting in two new objectives for the two networks, namely
\begin{equation} \label{eq:objective1}
	L_1({{\bf{W}}_{full}})  = {L_{{\theta _1}}} + \lambda R({{\bf{W}}_{full}},{{\bf{W}}_{low}}),
\end{equation}
and
\begin{equation} \label{eq:objective2}
	L_2({{\bf{W}}_{low}})  = {L_{{\theta _2}}} +  \lambda R({{\bf{W}}_{full}},{{\bf{W}}_{low}}),
\end{equation}
where $\lambda$ is balancing hyper-parameter. Here, the guidance loss $R$ can be considered as some regularization on ${L_{{\theta _1}}}$ and ${L_{{\theta _2}}}$.

\textbf{{Posterior-based training}} \hspace{1em}
Similar to \cite{hinton2015distilling}, we can also employ the posterior probability as the guidance signal. Let ${{\bf{p}}_{full}}$ and ${{\bf{p}}_{low}}$ be the full-precision teacher network and low-precision student network predictions, respectively. To measure the correlation between
the two distributions, we employ the Kullback–Leibler (KL) divergence:
\begin{equation}
{D_{KL}}({{\bf{p}}_{full}}|{{\bf{p}}_{low}}) = \sum\limits_{i = 1}^N {{{\bf{p}}_{full}}({{\bf{x}}_i})} \log \frac{{{{\bf{p}}_{full}}({{\bf{x}}_i})}}{{{{\bf{p}}_{low}}({{\bf{x}}_i})}}
\end{equation}
Then the final objectives for the two networks respectively become:
\begin{equation}
{L_1}({{\bf{W}}_{full}}) = {L_{{\theta _1}}} + \beta  {D_{KL}}({{\bf{p}}_{full}}|{{\bf{p}}_{low}}),
\end{equation}
and
\begin{equation}
{L_2}({{\bf{W}}_{low}}) = {L_{{\theta _2}}} + \beta  {D_{KL}}({{\bf{p}}_{low}}|{{\bf{p}}_{full}}),
\end{equation}
where $\beta$ is the balancing hyper-parameter.

Compared with the hint-based training strategy, the KL divergence based regularizer does not need to select the positions to add the signals, which may be sensitive to the final performance.
We empirically compare these two guidance strategies in Sec.~\ref{exp:hintvsposterior}.

In the learning procedure, both ${{\bf{W}}_{full}}$ and ${{\bf{W}}_{low}}$ will be updated by minimizing $L_1({{\bf{W}}_{full}})$ and $L_2({{\bf{W}}_{low}})$ separately, using a mini-batch stochastic gradient descent method. The detailed algorithm is shown in Algorithm \ref{algo:one-mutual learning}. A high-bit precision model $M_{low}^K$ is used as an initialization of $M_{low}^k$, where $K>k$. Specifically, for the full-precision model, we have $K=32$. Relying on $M_{full}$, the weights and activations of $M_{low}^k$ can be initialized respectively.

Note that the training process of the two networks are different.
When updating ${{\bf{W}}_{low}}$ by minimizing $L_2({{\bf{W}}_{low}})$, we use full-precision model as the initialization and apply STE to fine-tune the model. When updating ${{\bf{W}}_{full}}$ by minimizing $L_1({{\bf{W}}_{full}})$, we use conventional forward-backward propagation to fine-tune the model.

\begin{algorithm}[b]
	\KwIn{Training data $\{ ({{\bf{x}}_i},y_i)\}_{i=1}^N$; A pre-trained 32-bit full-precision model $M_{full}$; A $k$-bit precision model $M_{low}^k$.}
	\KwOut{A low-precision deep model $M^k_{low}$ with weights and activations being quantized into $k$ bits.}
	Initialize $M_{low}^k$ based on $M_{full}$;\\
	\For{ $\mathrm{epoch} = 1,...,L$}
	{
		\For{ $i = 1,...N$}
		{
			Randomly sample a mini-batch data;\\
			Quantize the weights ${{\bf{W}}_{low}}$  and activations into $k$-bit by minimizing $L_2({{\bf{W}}_{low}})$\;
			Update $M_{full}$ by minimizing $L_1({{\bf{W}}_{full}})$\;
		}

	}
	\caption{Guided training with a full-precision network for $k$-bit quantization}
	\label{algo:one-mutual learning}
\end{algorithm}

\subsection{Remarks on the proposed methods}
The proposed three approaches tackle the difficulty in training a low-precision model with different strategies. They can be applied independently. However, it is also possible to combine them together. For example, we can apply the progressive precision to any step in the two-stage approach; we can also apply the joint knowledge distillation to any step in the progressive quantization; we can combine stochastic precision with the joint knowledge distillation approach.
Detailed analysis on possible combinations will be empirically evaluated in the experiment section.

\section{Related work}
\label{sec:related_work}

A few
methods have been proposed to compress deep models and accelerate inference during testing. We can roughly summarize them into four main categories: limited numerical precision, low-rank approximation, efficient architecture design and network pruning. %

\textbf{Limited numerical precision.}
When deploying DNNs to hardware chips like FPGA, network quantization may be one of the  a must process for efficient computing and storage. Fixed-point quantization can be divided into uniform and non-uniform strategies.
Uniform approaches~\cite{zhou2016dorefa, zhuang2018towards} design quantizers with a constant quantization step. To reduce the quantization error, the non-uniform strategy~\cite{Cai_2017_CVPR} is a popular choice. Cai et al.~\cite{Cai_2017_CVPR} propose to explore the statistics of network activations and batch normalization to select the quantization levels. To make quantization more precise, authors of \cite{zhang2018lq, jung2018joint} propose to jointly learn the quantizer and model parameters for better accuracy.
Binary neural networks (BNNs)~\cite{rastegari2016xnor, hubara2016binarized} constrain both weights and activations into binary values (i.e., $+1$ or $-1$), where the multiply-accumulations can be replaced by the bitwise $\rm xnor(\cdot)$ and $\rm popcount(\cdot)$ operations, which can bring
significant computation
benefits to specialized hardware platforms. The development of binary neural networks may be summarized into two categories.  On one hand, some works focus on improving the training of BNNs. In~\cite{rastegari2016xnor}, scales are introduced to weights and activations in order  to improve the accuracy. Moreover, In \cite{hou2017loss} authors propose to directly minimize the loss w.r.t.\
the binarized weights. In~\cite{Liu_2018_ECCV}, a shortcut is added for each binary convolution layer to improve the gradient backpropagation. On the other hand, some works have proposed multiple binarizations to approximate full-precision tensors~\cite{lin2017towards, li2017performance, guo2017network, tang2017train} or structures~\cite{zhuang2019structured}.
A common fundamental problem of fixed-point quantization and BNNs is to approximate gradients of the non-differentiable quantizer (e.g., $\rm sign(\cdot)$, $\rm round(\cdot)$). Most works in  literature simply employ ``pseudo-gradients'' according to the straight-through estimator (i.e., STE)~\cite{bengio2013estimating}. Recently, Christos~{\em et al.\/}\, ~\cite{louizos2019relaxed} propose to smooth the discrete quantizer by transforming continuous distributions into categorical distributions which is then relaxed using gumbel-softmax for efficient gradient-based optimization. In contrast, we propose three relaxed training algorithms which can be built upon general quantization approaches, including the quantizer smoothing methods.

\textbf{Low-rank approximation.}
Among existing works, some methods attempt to approximate low-rank filters in pre-trained networks~\cite{kim2015compression, zhang2016accelerating}. In~\cite{zhang2016accelerating}, reconstruction error of the nonlinear responses are minimized layer-wisely, with subject to the low-rank constraint to reduce the computational cost.
Other seminal works attempt to restrict filters with low-rank constraints during training phrase~\cite{novikov2015tensorizing, tai2015convolutional}. To better exploit the structure in kernels,
low-rank tensor decomposition approaches are introduced to remove the redundancy in convolutional kernels in pretrained networks
\cite{denton2014exploiting, novikov2015tensorizing}.

\textbf{Efficient architecture design.}
The increasing demand for %
highly energy efficient neural networks that are deployable to embeded  hardware devices has motivated the network architecture design. GoogLeNet~\cite{szegedy2015going} and SqueezeNet~\cite{iandola2016squeezenet} propose to replace $3\times3$ convolutional filters with $1\times1$ size, which tremendously increase the depth of the network while decreasing the complexity. %
ResNet~\cite{he2016deep} and its variants~\cite{zagoruyko2016wide, he2016identity} utilize residual connections to relieve the gradient vanishing problem when training very deep networks. Recently, depthwise separable convolution employed in Xception~\cite{chollet2017xception} and  MobileNet~\cite{howard2017mobilenets} have been proved to be efficient
and effective.
Based on it, ShuffleNet~\cite{zhang2017shufflenet} generalizes the group convolution and the depthwise separable convolution to %
achieve
state-of-the-art results. Since it is infeasible to manually explore the optimal architecture from the %
enormous
design space, neural architecture search (NAS) has been attracting more and more attention to automatically search for  efficient structures. A main category of methods~\cite{zoph2016neural, pham2018efficient, zoph2017learning, liu2017progressive} are based on the reinforcement learning framework while %
others are
built on  approximated differentiable gradient-based search %
\cite{liu2019darts, cai2019proxylessnas, guo2019single}.

\textbf{Pruning.}
Network pruning can be categorized into fine-grained pruning~\cite{han2015learning}, channel pruning~\cite{luo2017thinet, yu2018nisp, he2017channel, liu2017learning, zhuang2018discrimination} and block pruning~\cite{huang2018data, wu2018blockdrop}.
Han~{\em et al.\/}\,  \cite{han2015learning, han2015deep} popularize pruning by introducing ``deep compression'' with  a three-stage pipeline: pruning, trained quantization and Huffman coding to effectively reduce the memory requirement of CNNs with almost no loss of accuracy.
Channel pruning removes both channels and the related filters from the network, which is the most prominent pruning method and it can be well supported
by existing deep learning libraries with little additional effort. The key issue of channel pruning is to evaluate the importance of channels. Li~{\em et al.\/}\, ~\cite{li2017pruning} propose to measure the importance of channels by calculating the sum of absolute values of weights. Gao~{\em et al.\/}\, ~\cite{gao2019dynamic} learn soft attention on channels for removing redundant ones. Liu~{\em et al.\/}\, ~\cite{liu2017learning} leverage the scaling layers in batch normalization to effectively identify and prune unimportant channels in the network.
Authors of \cite{zhuang2018discrimination, yu2018nisp} %
use
the gradient magnitude as the channel importance metric. Furthermore, recent works~\cite{ashok2018n2n, he2018amc} employs AutoML which leverages reinforcement learning to automatically search the pruning ratio for each layer.

\textbf{Knowledge distillation.}
Knowledge distillation is initially proposed for model compression, where a powerful wide/deep teacher distills knowledge to a narrow/shallow student to improve its performance~\cite{romero2014fitnets, hinton2015distilling}. In terms of representation of the knowledge to be distilled from the teacher, existing models typically use
teacher's class probabilities~\cite{hinton2015distilling} and/or feature representation~\cite{romero2014fitnets, zagoruyko2016paying}.  Knowledge distillation has been widely used in many computer vision tasks. Zhang~{\em et al.\/}\, ~\cite{zhang2016real} propose to transfer the knowledge learned with optical flow CNN to improve the action recognition performance. Moreover, several works propose to learn efficient object detection~\cite{chen2017learning, wei2018quantization} and semantic segmentation~\cite{he2019knowledge} with distillation. In contrast to previous approaches, we concentrate on improving the performance of the quantized neural network. By adapting the teacher and student altogether, we can steadily improve the performance of the quantized student network and even the full-precision teacher network.

\textbf{Dropout.}
Dropout~\cite{srivastava2014dropout}, Maxout~\cite{goodfellow2013maxout}, DropConnect~\cite{wan2013regularization} and DropIn~\cite{smith2016gradual} are a category of approaches that propose to stochastically drop intermediate nodes or connections during training to prevent the network from overfitting. %
They perform different types of regularization.
Huang~{\em et al.\/}\, ~\cite{huang2016deep} further propose stochastic depth regularization via randomly dropping a subset of layers during training.
Dong~{\em et al.\/}\, ~\cite{dong2017learning} propose to randomly quantize a portion of weights to low-precision in the incremental training framework~\cite{zhou2017incremental}. The method in~\cite{dong2017learning} is developed for only quantizing weights of a network. In our method, we develop extension of it by further randomly quantizing a portion of the network, i.e.,\textit{ layers or blocks as well as activations and weights}. Moreover, we \cite{zhuang2018towards} propose two progressive training strategies: quantizing weights and activations in a two-stage manner; progressively decreasing the bit-width from high-precision to low-precision during the course of training. However, the multi-stage strategy sacrifices the training efficiency. In contrast, we improve the progressive quantization into only a single stage.
Our study shows that this extended scheme is complementary to the proposed joint knowledge distillation approach.

redpajama_set_name: RedPajamaArXiv


Laminate Wood Flooring Ideas Lowe's DIY friendly wood look laminate flooring looks great all around the house, and it won't bust ... Laminate wood look floors resist abuse from a little boy's toy truck.
Laminate Flooring | eBay Find great deals on eBay for Laminate Flooring in Tile and Flooring. ... Plank laminate flooring gives you the ultimate feeling of hardwood without the cost and ... Swiss Krono Pro Line Coffee House Walnut PL2173 Laminate Flooring SAMPLE.
Lowe's Style Selections Laminate Flooring Review The Spruce Apr 4, 2017 ... Cracking open and installing boxes of Swiftlock laminate flooring, and then reviewing it. ... As a house brand, prices are kept consistently low.
Laminate Flooring Cost & Installation Pricing | 2018 Cost Guide Oct 26, 2017 ... If it's time to replace your old floors and laminate flooring is on your radar, .... Some of the more popular ones include, Lowe's, ... She loves to write about things that will transform your house into your dream home.
Laminate and Hardwood Flooring, Official PERGO Site | PERGO ... PERGO floors are easy to install, simple to maintain and have the industry's best ... Get maximum savings on Pergo Max Hardwood from Lowe's. ... LAMINATE.
Lowe's vs Prices| Lumber, Carpet, Kitchen Cabinets ... Apr 16, 2018 ... We've compared prices for lumber, carpet, kitchen cabinets, flooring, ... Large retailers like and Lowe's have their own house brands and ... or appliances or laminate flooring, and anticipating additional needs.
Laminate and Laminate Wood Flooring Lowe's Shop laminate flooring and laminate floor accessories at Lowe's. ... grains and texturized styles offer the look and feel of real hardwoods at an affordable price.
swiftlock laminate flooring at lowes the lettered cottage bungalow ... swiftlock laminate flooring at lowes the lettered cottage bungalow barn I really like ... design ideas and inspiration for the transitional home : Rossmoor house finished! ..... vends extravagant vinyl plank flooring from prime brands at marked down rates. .... Shaw Castel Hickory Hickory Solid Hardwood Flooring ft) at Lowe's.


text:
Brenda Lyn Wordell Gill
Little Compton Historical Society > Historical Resources > Little Compton Women's History Project > Brenda Lyn Wordell Gill
Brenda Lyn Wordell Gill. Courtesy Braden Lasso.
"Though she be but little, she is fierce."
This is a quote she wears on her bracelet. Brenda is not very tall but those who know her would describe her as someone that is persistent and strong. She is a go-getter. She is my mom.
Brenda Lyn Wordell was born in May of 1973, at St. Anne's Hospital in Fall River, Massachusetts. Her parents are Roger and Noreen Wordell. She has one older brother, Bradford Wordell, making Brenda the youngest child. She was a normal baby with parents that loved her a lot. She lived in Little Compton all of her childhood. She even went to Wilbur & McMahon school as a student.
Brenda had an awesome childhood. Her mom was lucky enough to stay home while Brenda and Brad were kids. When she did go to work, she worked in the school where Brenda and her brother went. Brenda's cousins lived close by, so she was never without a playmate. They enjoyed going to the beach together, and they still do to this day! They are so close, that Brenda has become an honorary "sister" to her cousins. Every Friday in the summertime is considered "Sisters Day" where they all meet at Briggs Beach and spend the day together. As a child, Brenda enjoyed the outdoors and loved ballet and gymnastics.
When she was about ten, she had a ballet teacher who brought her to Boston to see if she would be interested in attending the Boston Conservatory. They took a bus from Fall River with her mom. It was a long ride. In the end, she ended up not going because it was a big decision and it would be a big lifestyle change. She would have to be homeschooled and take a bus to Boston every day. She did not feel ready for that.
Brenda thought that she wanted to be a professional dancer. She dreamed of being on Broadway! She had always loved being on stage, performing. She practiced whenever she got the chance. This led her to be a great dancer. Her favorite role was being Dorothy, from the Wiz. She loved the sound of the audience's applause for a good performance. She was so into dance that she taught dance classes in Tiverton at the Creative Dance Studio at the age of twelve. These were the best years of her life. She made friends that she still has today. She loved teaching, and from then on, she knew she was going to be a teacher.
After she graduated from Middletown High School, she went to Plymouth State College (now Plymouth University). She majored in education and psychology. She loved living in New Hampshire and was happy to have the opportunity to leave Little Compton for a few years. She eventually came back, and got her first teaching job in Fall River, at Atlantis Charter School teaching second grade. She also went back to school for her master's degree at Lesley University. She taught there for nine years and then went to a new job in Somerset, Massachusetts. She still works today teaching third grade.
A few things Brenda is proud of are her two children, Braden and Sofie. One thing she always says is that she is raising two "good humans". Another thing she's proud of is building her own house by herself. She was given the land from her grandmother, Edith, on her 13th birthday. When she was thirteen, she didn't really care about having her own land. When she was an adult, she realized how lucky she really was! She built the house before she had a husband and kids and was involved in all aspects of it. She did all the painting, banged a lot of nails, learned to use power tools, and made every decision. It was A LOT of work. She spent all her life savings on building the house. She said she "felt empowered as a woman to say, 'I did this!'." The last thing she is proud of is going to graduate school.
Some of Brenda's influences include her parents. Her dad, Roger, influenced her with his knowledge and understanding of business and money and general knowledge of the world. Her mom, Noreen, influenced her sense of nurturing and caring and the importance of family. Also, some of her teacher friends inspire her professionally, her cousins who are like sisters to her and influence the importance of family, and her close friends inspire her to be a better person every day. She says her husband brings her balance in life because he helps her with most everything! Finally, she is influenced by her kids who make her see everything in a different light and appreciate the "small stuff". She said she always knew she meant to be a mom.
All in all, my mother, Brenda, is a woman who is an example of "Though she be but little, she is fierce". She has accomplished so much in her life and never let anything get in her way.
Based on an interview with Brenda (Wordell) Gill, conducted on March 20, 2020.
Written by: Braden Lasso, Grade 5 – Wilbur & McMahon School

redpajama_set_name: RedPajamaCommonCrawl


When Employee Benefits last conducted research into voluntary benefits in 2005, employers' main reason for offering such schemes was to provide staff with access to as wide a range of benefits choices as possible. However, while voluntary benefits provide a cost-effective way of offering employees extra perks and help enhance an organisation's package with additional options, employers now seem to be more concerned with ensuring that any scheme adds value for both the organisation and staff.
Attitudes around the cost of offering a voluntary benefits scheme have also changed. Back in 2005, 54% of respondents said it was important that the voluntary benefits they offered didn't cost them anything. This year, the figure has dropped to 39%.
Furthermore, employers are now finding that they are having to think about how their discounts can compete with the prices offered by online retailers. One way is by offering a wider choice of providers for each product.
Three-quarters of respondents provide voluntary benefits for employees, which is a drop from the 90% who did so in 2005 when Employee Benefits last conducted specific research in to this area.
This may be because voluntary benefits providers are now more inclined to charge employers for schemes or respondents do not classify some perks as voluntary benefits. Of those that provide voluntary benefits, 76% (286 respondents) are responsible for their implementation.
With increasing levels of sickness absence and the debate over who should be responsible for the nation's health rarely out of the headlines, it is perhaps unsurprising that the past two years have seen a rise in the number of employers that offer private medical insurance (PMI) as a voluntary benefit. Just under three-quarters (72%) of employers now offer PMI, up from 58% in 2005, making it the most commonly-provided voluntary option.
Snapping on its heels is life assurance. The percentage of employers offering this benefit has also risen significantly, from 49% in 2005 to 69%.
With the cost of premiums still reported to be rising in many instances, and an ever-increasing focus on gaining a return on investment on benefits spend, it seems that many employers are using their voluntary arrangements to pass at least some of the cost on to employees.
Pensions also remain high on the priority list for many employers. Approximately two-thirds of respondents now offer the means to make additional voluntary contributions to a defined benefit or money purchase pension scheme (68%) or access to a group personal pension plan (GPP) (66%), rendering both options more popular than two years ago. Stakeholder pensions, meanwhile, are not too far behind, being offered by 48%.
Providing staff with the means to contribute to or top up their pension fund can help to create a favourable impression of an organisation, often at very little cost. Setting up pension contributions through a salary sacrifice arrangement, meanwhile, can even save both employers and employees money.
At the other end of the scale, the popularity of employee share schemes offered as voluntary benefits has fallen considerably over the past two years.
Just 16% of employers now offer a HM Revenue and Customs-approved sharesave scheme, compared with 35% two years ago, while the number of respondents who provide staff with a HM Revenue and Customs-approved share incentive plan (Sip) has fallen from 21% in 2005 to 14% this year.
Concern for the environment appears to be taking hold as 4% of respondents are now offering carbon credits or offsetting as an option.
Historically, employers have often looked to voluntary benefits schemes to offer staff as wide a choice of benefits options as possible. When it comes to the number of options offered by this year's respondents, however, employers seem to prefer to provide quality over quantity. More than half (53%) of respondents say they offer less than 10 options through a voluntary benefits scheme for staff, while just under a quarter (23%) offer between 11-25 perks. Just 4% of respondents say they offer more than 1,000 benefits through their scheme.
Offering a smaller number of benefits that are tailored to suit an organisation's workforce demographic can often be much more effective than taking a much wider 'hit and hope' approach by providing as many voluntary options as possible. Providing a large choice of perks may initially appear overwhelming for staff, particularly if they are unable to find what they are looking for quickly. If employers do offer a select number of options, however, this may prove demotivating if they are not tailored to suit their employees' interests and life stages.
Placing details of perks on an organisation's intranet or internet is the most the favoured form of communication, most likely because it enables employers to create a reference source that can be easily updated. This usually supports other, often more traditional and direct forms of communication such as booklets or brochures (63%) and induction meetings (61%), which remain popular. These are both good ways of reaching staff in industries where employees do not have access to a computer during the working day.
A handful of employers, however, have begun to supplement their messaging with new forms of IT, such as videos (used by 3%), webinars (2%) and text messages (2%).
Although increasing numbers of employers are focused on ensuring that employees value the benefits on offer, only a relatively small percentage (15%) use total reward statements to communicate the savings that staff make through taking up voluntary perks.
Many employees, therefore, may not realise the true value of what they receive from their employer.
Encouragingly, just under half (49%) of respondents say that they are considering using total reward statements to communicate the value of the savings made by staff through voluntary benefits. If they were to do so then they may find that the move helps to enhance employee engagement, loyalty and performance.
The government's removal of the tax break around its home computing scheme last year dealt a severe blow to the benefits community leaving many employers in shock and wondering whether they should risk introducing other perks that can be payable through salary sacrifice arrangements.
Yet, employers appear to have overcome their concerns and the popularity of offering perks on a voluntary basis through salary sacrifice has continued to increase. More than three-quarters (77%) of respondents now offer voluntary benefits on this basis, up from 68% in 2005.
Employers are also looking to make the most of tax-efficient legislation around other perks by offering salary sacrifice on food in on-site restaurants and bus travel, which are options now provided by 5% and 4% of respondents respectively.
The popularity of voluntary benefits offered through a salary sacrifice arrangement is also borne out by the growth in the use of individual perks offered in this way. Just a handful of respondents do not offer childcare vouchers through salary sacrifice. For example, 92% of respondents have now opted to offer the perk in this manner, compared to 71% in 2005.
Bicycle loans have also seen substantial growth over the past two years. In 2005, just 8% of employers offered a bikes-for-work scheme through salary sacrifice to staff, compared with 46% of respondents this year. Mobile phone loans have also increased in popularity as an option.
Such growth may, in part, be due to employers' increased understanding of salary sacrifice and how such perks operate. Fears around the complexity of setting up and administering salary sacrifice schemes have fallen considerably. Just 6% of respondents now believe that benefits offered in this way are too complicated to set up, and administer, compared with 24% and 23% respectively two years ago.
The two main reasons for offering benefits through salary sacrifice, however, have not changed. Respondents still believe that is a good way to save employees money and that it forms an important aspect of employers' HR strategies. The savings employers make also prove useful in funding other perks, according to 30% of respondents.
The last few years have seen a substantial rise in the number of employers that pay a fee for their voluntary benefits package, not least because the number of providers that offer their products for free has dwindled. Half of employers now pay a fee for their voluntary benefits scheme, compared with less than a third (32%) two years ago.
Most specialist third-party administrators now charge a fee for at least some of their services, even if they previously offered some of these free of charge.
Negotiating deals for staff directly with local businesses or some product providers, however, typically doesn't come at a cost, so it may be that the 42% of respondents who do not pay a fee for their voluntary benefits package, and would not be prepared to start doing so, have negotiated perks in this way.
In fact, dealing with providers directly is employers' favoured method of sourcing the voluntary benefits that they offer – 40% currently do so.
However, the percentage of respondents which use a specialist voluntary benefits company to either set up and negotiate deals on their organisation's behalf or to provide an off-the-shelf package, has risen over the past two years.
More than a quarter (27%) of respondents now source their scheme in this way, compared with 18% in 2005.
This method still appears to be the domain of larger organisations, however, possibly due to the costs involved.
Of those employers that use a specialist voluntary benefits provider to source their scheme, just over a third (34%) of respondents employ a workforce of 1,000 staff or less.
More than a quarter (28%) of respondents, meanwhile, use a combination of these two approaches when sourcing voluntary benefits, compared with 35% two years ago.
Voluntary benefits schemes are often used as a stepping stone towards offering a flexible benefits plan. The very nature of such schemes means they are a good way for employers to introduce the concept of choice around benefits for staff, before moving to a full flex roll out. This also enables employers to test the water for flex by offering a range of benefits with the minimum of effort. But despite this, just 13% of respondents plan to move to flex in the next 12 months.
However, as we have seen time and time again, many employers that plan to introduce flexible benefits never do so for a variety of reasons, such as cost constraints or due to a finding of unsuitability following a feasibility study.
Environmental initiatives and corporate social responsibility have risen rapidly up many organisations' agendas over the past few years in an attempt to project a positive business image. As a consequence, employers have also begun to examine their operations internally not only in relation to working conditions but also in respect of reward. This makes sound business sense as staff have begun to attach greater importance to working for a responsible employer.
The benefits market has responded accordingly, with a number of providers launching, or adapting, products with an environmental ethos. It is perhaps little surprise, therefore, that carbon credits or offsetting schemes rank as the top benefit that employers plan to add to their voluntary benefits scheme over the coming year.
Dental insurance, meanwhile, follows closely behind, as many employers are planning to offer staff who are struggling to obtain treatment on the National Health Service an alternative.
Health screening and financial advice or education tie for this year's third place. Employees' physical and financial wellbeing are both issues which have received much media attention over the past two years and, as a consequence, employers' interest in these perks has increased.
Staff who feel that they are supported in these areas may often be more likely to be more productive than those who have concerns about their finances or health.
Very few employers plan to remove any options from their voluntary benefits scheme. This ties in with what we have seen in practice, as many employers use voluntary schemes to offer as wide a choice of perks as possible. It will be interesting to see if this changes at all over the next couple of years as employers continue to look to tailor schemes to better suit their workforce demographics.
The tax efficiencies attached to offering certain benefits through a salary sacrifice arrangement makes such perks a popular choice for employers. Childcare vouchers, for example, are offered by the majority of our respondents (91%), which could account for the fact that just 2% say that they plan to introduce these on a voluntary basis.
With an increasing focus on both employee wellbeing and environmental issues, it is perhaps not surprising that tax-efficient bicycle loans are the most popular benefit that respondents plan to introduce on a voluntary basis. Just under a quarter (24%) say they intend to do so.
Employers are also showing increasing interest in offering bus travel to staff paid for via salary sacrifice. This also supports employers' environmental strategies, which may be behind the fact that 8%of respondents intend to introduce it for staff.
This perk has yet to become widespread and a number of employers may be waiting to see howitworks in other organisations before going ahead with a tax-efficient scheme themselves.
Salary sacrifice arrangements on food in onsite restaurants, which 9% of respondents plan to introduce, is also emerging as a key salary sacrifice perk.
However, this benefit may have a limited appeal as it relies on employers having the necessary on-site restaurant facilities and technology in place before they can offer it to staff.
This article is brought to you by our sponsor AIG Benefits.
The voluntary and salary exchange benefits marketplace, like many others, is experiencing rapid change. New methods of accessing and distributing voluntary benefits are becoming more widely used by employees, particularly online routes; while salary exchange benefits, such as green and health-related products, are increasingly in demand by both employers and employees.
The future of any market lies in providers becoming more reactive to the needs of their customers. This is particularly true for the employee benefits market, where it is no longer acceptable for providers to develop offerings that do little to meet the needs of organisations and their increasingly diverse and regularly-changing workforce. We need to find real ways of genuinely extending the buying power of employees every month; not just when they take a holiday and park their car at the local airport at discounted rates.
Communication is a key element of the voluntary benefits market and the methods used to reach employees need to adapt and evolve as new media becomes available. An employee's level of engagement with specific benefits products, which is a basic reason for establishing a benefits package, may be directly affected by the method of communication that is used to explain or deliver it.
For example, products that are inherently more complicated, such as personal accident or cancer-associated policies, are often better understood by individuals if explained at a face-to-face meeting. Despite the advances of online technology in explaining and delivering voluntary benefits, the personal touch is still welcomed by many existing and potential customers where the product is not commoditised and where they want the opportunity to discuss it and decide whether they feel it is appropriate for them.
As benefits providers, we all need to listen closely to the people that we serve and work hard to anticipate their needs. The growing take-up of green benefits is an example of how, as an industry, we have successfully managed to do this.
Green schemes, including recycling, green energy and healthy eating, are quickly being adopted by businesses that see the benefit of offering their people options that genuinely make an impact on their personal lives. Also, given their growing status on the corporate agenda, green benefits have the potential to contribute to an organisation's environmental performance and play a major part in developing green policies and strategies throughout the business.
The future, it seems, is positive for the voluntary and salary exchange benefits markets as long as providers continue to observe basic business rules. The challenge for all those in the industry is to continue to innovate and develop new products that meet the needs and requirements of an ever-changing and dynamic workforce, as well as using the most appropriate communication channels to deliver them. By doing so, we will keep the market fresh and ensure that individual employees and their employers get the best products for even the most competitive budgets.
The views and opinions in this article are those of our sponsor, AIG Benefits, and do not necessarily reflect those of www.employeebenefits.co.uk.


The supermodel shared a glimpse of her outfit for this year's celebration on Instagram. "First fitting for Halloween," she wrote "What am I going to be this year ? Can you guess !?!?!?!?!?!"
It might not look like much yet, but it's sure to be a major hit!
Last year Heidi dressed as Jessica Rabbit, and rocked a LOT of prosthetic's to pull it off.
What's cooler than a freaking huge butterfly?!
A really old lady, that's what's cooler! Heidi slayed with this grandma vibe.
This was potentially one of the most OTT looks Heidi ever rocked, and it was phenomenal! Bow down to Cleopatra y'all!
This bloody amazing look wowed fans.
Robo-Klum was a major hit... and freakishly tall!
This vampire-burlesque mix was scarily odd, but we loved it all the same!
Don't know what the hell she was, but it doesn't mean we didn't love it. This golden masterpiece was absolutely jaw-dropping.
This has to be one of Heidi's crowning achievements, she looks EXACTLY like the famous Miss Betty Boop.


Why is so much wonderful cotton shirting described as a man's fabric???
I couldn't pass on Mood Fabrics' beautiful pink/blue Italian cotton shirting even if the description read – This is a light weight 100% cotton shirting with a woven pattern. Smooth, soft, sleek, ideal for men's dress shirts.
I knew this soft textured fabric with its baby colored stripes would be adaptable to a feminine style. I was also itching for an embroidery project and created an embroidered border for the top. Having sewn McCall's 7543 twice before, once in linen and once in a cotton poplin, I thought a lightweight shirting would endure the embroidery process and gently drape from the gathers.
For the embroidery, I selected a single motif on the Pfaff Creative Icon which I repeated in groups of three.
…then I placed the pattern piece on the fabric and cut the bodices.
After 'finishing' the top, I wanted a little more pop and got the scissors back out.
Finding the versatility in a fabric is always rewarding for me. Many thanks to Mood Fabrics for providing me with the opportunity for discovery -Be sure to check out Mood's shirtings for your next sewing adventure!
Now….. boy or girl ??