text:
\section{Introduction}

For last many decades, ZnO has remained a material of
great research interest both in fundamental and applied aspects. Due
to high excitonic binding energy ($\thicksim$60 meV) and wide
bandgap of 3.37 eV, ZnO has potential applications in optoelectronics
devices like light emitting diodes (LED), photodetectors and laser
diodes\cite{Chu2008_ZnO_laser_diode_app, Choi2010_ZnO_LED_app, Chu2011_ZnO_lasing}. In particular, this compound has been a subject of
active investigation to exploit its potential as a transparent conducting
oxide which can serve as an alternate to indium tin oxide (ITO)\cite{Minami_2000_TCO}. The conductivity in n-type doped ZnO can be enhanced
as high as $\thicksim$7000 S/cm through substitution of group
III and IV elements at the Zn site\cite{Bhosle2006_MIT_Ga_conc, Kuznetsov2015_SiZnO_conductivity, Das2014_SiZnO_MIT}. The large conductivity in this system is primarily attributed to the delocalized p-obitals carrying
the donor electrons\cite{Sans2009_doped_ZnO_BS}. Among these n-type dopants, Ga is
the most suitable candidate to enhance the conductivity of ZnO since
both Ga and Zn have almost similar ionic radii which helps in increasing
the solubility of a dopant in the host material.

Recently, Look \textit{et al}. has achieved a minimum resistivity of $1.96\times10^{-4}$
$\Omega$-cm with 3\% Ga doping in ZnO thin films grown by pulsed
laser deposition (PLD) technique\cite{Look2011_GaZnO_res_mobility}. Earlier, Bhosle
\textit{et al}. had reported that 5\% Ga doping yields minimum resistivity
in ZnO thin films\citep{Bhosle2006_MIT_Ga_conc}. It is observed that resistivity first
decreases as doping concentration increases and after reaching a minimum
value, it again increases since dopant itself starts acting as a scattering
center\cite{Bhosle2006_MIT_Ga_conc}. Therefore, ZnO can exhibit metal insulator transition
(MIT) by tuning the factors such as doping concentration. In fact,
MIT in Ga doped ZnO as a function of doping concentration has been reported by several other groups \cite{Bhosle2006_MIT_Ga_conc, Snure2007_MIT_Ga_conc_Mott_limit}. With sufficiently high doping
concentration, the dopant electrons make ZnO degenerate as these electrons form a dispersive band. The MIT can
also be achieved in degenerately doped ZnO by reducing the thickness. It has been experimentally observed that the conducting
Si doped ZnO becomes highly resistive below a critical thickness of
15 nm\cite{Das2014_SiZnO_MIT}. However, so far the thickness dependent MIT investigation is
limited to Si doping only. Therefore, a detailed mechanism that causes
the MIT has not been completely understood yet. In this work, we examine the possibility of MIT in Ga:ZnO, which shows maximum conductivity among all n-type dopants, as a function of film thickness.

To gain an in-depth understanding behind the thickness driven MIT,
experimentally it is imperative to study the low temperature electrical
properties of degenerately Ga:ZnO films as a function of thickness.
Temperature dependent resistivity measurement undoubtedly reveals
the metallic or insulating nature. For metallic sample temperature
coefficient of resistance (TCR) is positive whereas for insulator
TCR becomes negative. In addition, the occupancy and k-dispersion
of the bands are good measures to examine the metallicity and thereby,
the MIT. In this context, solutions to the quantum mechanical many
body problems through \textit{ab initio} calculations based on density-functional
theory (DFT) are desirable.

In this paper, we present the structural, electrical and magnetotransport
properties of Ga:ZnO thin films of varying thickness, in the range
of 6 \textendash{} 51 nm, grown on c-sapphire substrate. From the
temperature dependent resistivity measurement, we observe MIT in Ga:ZnO films as a function of film thickness. For 6 nm thick
film, the resistivity is found to be close to 0.05 $\Omega$-cm which
lies in the insulating regime and conduction is governed by the variable range hopping (VRH) mechanism. For the film with thickness 20 nm and above,
the resistivity is of the order of $10^{-3}$ $\Omega$-cm. These films exhibit 2D weak localization phenomena at low temperature regime. Temperature dependent resistivity and magnetoresistance data confirm the presence of weak localization in films with thickness 20 nm and more while in 6 nm thick film exhibits strong localization behavior. Our density functional calculations suggest that, in thicker films, lattice distortion due to Ga dopant is weak and donor p-electron partially occupies a dispersive band in k\textsubscript{x}-k\textsubscript{y} plane which is the source of metallic type of conduction. However, in 6 nm film, surface reconstruction and the dopant induce sufficient lattice disorders which in turn create localized states in the vicinity of Fermi level and hence, the film exhibits insulating behavior.
\section{Experimental details}

1\% Ga doped ZnO (Ga:ZnO) thin films were deposited on (001) sapphire substrate
(5 mm $\times$ 5 mm) by PLD technique.
A KrF laser of wavelength 248 nm of fluence 2 J/cm\textsuperscript{2}
and 2 Hz pulse frequency was used for the deposition. The substrate
to target distance was 4.5 cm. During the deposition, O\textsubscript{2}
partial pressure inside the chamber was maintained at $5\times10^{-3}$
mbar and the substrate temperature was 500 \textsuperscript{o}C for
all the samples. The number of laser pulses were varied to grow thin
films of different thicknesses. A 4-circle high resolution XRD (HRXRD)
(Rigaku Smartlab, Japan) was employed to investigate the structural
quality and also to determine the thickness of the samples from X-ray
reflectivity (XRR). These experiments were performed using a Ge (220)
monochromator and the Cu K\textsubscript{$\alpha$} source. Temperature
dependent electrical and magnetotransport measurements were carried out using
a physical property measurement system (PPMS, Quantum Design) in the
temperature range of 4 \textendash{} 300 K.

\section{Experimental results}

\subsection{X-ray reflectivity}

To precisely determine the film thickness, we have carried out X-ray reflectivity (XRR) on Ga:ZnO films. Figure \ref{fig:XRR} displays XRR profile of Ga:ZnO thin films
of different thickness grown on c-sapphire substrate. XRR data was
fitted using Globalfit software (Rigaku) to estimate the thickness
and roughness of the films. It can be observed from the Fig. \ref{fig:XRR} that the fitted data are in excellent agreement with the experiment.
The thickness values as obtained from the fit are indicated in the
figure and have been used to analyze the electrical transport data.
The root mean square (rms) roughness of the surface of each film lies
below 0.7 nm implying an extremely smooth nature of Ga:ZnO films.

\begin{figure}
\includegraphics[width=8.5cm]{fig_1.png}

\caption{X-ray reflectivity profiles of Ga:ZnO thin films grown on sapphire
using PLD. Inset shows XRR profile of 6 nm film in larger x-axis range. The clear oscillations indicate that the film surface and
the interface are very smooth. The rms roughness of the samples is
below 0.7 nm. Red lines are fitting of experimental data using globalfit
software. Each profile is multiplied by a factor of 10 for easy visualization.\label{fig:XRR}}
\end{figure}

\subsection{X-ray diffraction}

It is essential to check the crystallinity and the epitaxial nature
of Ga:ZnO films particularly in low thickness. Figure \ref{fig:XRD}(a) depicts the
$2\theta/\omega$ scan of the films with different thickness grown
by PLD on c-sapphire substrate. Only (002) and (004) peaks of ZnO
were observed along with the substrate (006) peak.
This confirms that films are highly textured along the c-axis. The absence
of additional peak confirms that there are no Ga\textsubscript{2}O\textsubscript{3}
or other phases present in the samples. As film thickness decreases, full width at half maxima (FWHM) of (002) peak increases and the intensity of the peak decreases monotonically. The FWHM of 51 and 6 nm film is 0.56\textsuperscript{o} and 1.6\textsuperscript{o} respectively\cite{Dong2007GaZnO_XRD}. The broadening of (002) peak in the thinnest film can be attributed to the lattice disorder and strain present in the film. Moreover, to confirm the epitaxial nature of the
films, pole figure measurements were carried out on all the films. Pole figure measurement was performed on {[}10-11{]} set of planes of ZnO
and Fig. \ref{fig:XRD}(b) displays the 3D plot of pole figure of 20 nm thick sample. Six equally spaced peaks were obtained
which confirm the six fold symmetry of ZnO\cite{Opel2014_ZnO_Epitaxy, Snure2007_MIT_Ga_conc_Mott_limit}. In our films, we do not observe any 30\textsuperscript{o} in-plane domain rotation of ZnO on c-sapphire substrate\cite{Opel2014_ZnO_Epitaxy}.

\begin{figure}
\includegraphics[width=8.5cm]{fig_2.png}
\caption{XRD patterns of Ga doped ZnO thin films grown on saphhire substrates
by PLD. The sharp peak at 41.7\textsuperscript{o} corresponds to
Sapphire (006). (b) 3D pole figure of Ga:ZnO thin film of thickness
20 nm. Peaks corresponding to ZnO and sapphire are marked with red and yellow arrow. It can be seen that substrate and film peaks are 30\textsuperscript{o}
apart.\label{fig:XRD}}
\end{figure}

\subsection{Resistivity}

To investigate the thickness driven metal-insulator transition (MIT)
in Ga:ZnO films, temperature (T) dependent sheet resistance measurements
on all the samples were carried out using standard 4-probe method. Irrespective of T, the sheet
resistance ($R$) increases as film thickness is reduced.
The values of $R$ for 51, 33 and 20 nm thick films are 29 , 51 and 140 $\Omega$ respectively. With further decrease in film thickness, there
is a sharp increase in $R$ as shown in Fig. \ref{fig:sheet resistance}. At 300 K, $R$ of 6 nm film is found to be 19.7 k$\Omega$ . Similar behavior is observed in Si doped ZnO thin films by Das et al\cite{Das2014_SiZnO_MIT}. Sheet resistance (R), resistivity ($\rho$) and carrier concentration ($n$) of Ga:ZnO films measured at 300 K are listed in the Table \ref{tab:electrical paramaters}.
\begin{table*}
\caption{The list of sheet resistance ($R_{300}$, measured at 300 K), resistivity ($\rho_{300}$),  $\rho_{4}/\rho_{300}$ and electron concentration ($n$) values of Ga:ZnO thin films of different thickness.
\label{tab:electrical paramaters}}
\begin{tabular}{c@{\hskip 1cm}c@{\hskip 1cm}c@{\hskip 1cm}c@{\hskip 1cm}c}
\hline
\hline
Film thickness (nm) & $R_{300}$($\Omega$) &  $\rho_{300}$($\Omega$-cm)  & $\rho_{4}/\rho_{300}$  & $n$(cm\textsuperscript{-3}) \tabularnewline
\hline
51 & 29 & $6.6\times10^{-4}$ & 0.92 & $1.5\times10^{21}$ \tabularnewline
33 & 51 & $7.7\times10^{-4}$ & 0.97 & $1.3\times10^{21}$ \tabularnewline
20 & 140 & $1.3\times10^{-3}$ & 1.2 & $8.6\times10^{20}$ \tabularnewline
6 & 19667 & 0.05 & 188 & $7.1\times10^{20}$ \tabularnewline
\hline
\hline
\end{tabular}
\end{table*}
The carrier concentration is found to be $1.3\times10^{21}$ and $7.1\times10^{20}$ cm\textsuperscript{-3} for 51 and 6 nm films respectively. Figure \ref{fig:sheet resistance} depicts the behavior
of sheet resistance ($R$) as a function of temperature for all Ga:ZnO
films of different thickness. We find that there is a weak dependence
of $R$ on temperature for thicker films (20 nm and above). However, in the case of 6 nm thick film,
there is a sharp increase in $R$ below 50 K which is the characteristic feature of an insulator.
\begin{figure}
\includegraphics[width=8.5cm]{fig_3.png}
\caption{Temperature dependent sheet resistance of Ga doped ZnO thin films
of different thickness. Note that the y- axis is in
logarithimic scale. As thickness decreases, the sheet resistance increases
monotonically. The film of thickness 6 nm shows an insulating behavior
whereas others exhibit metallic type of conduction.\label{fig:sheet resistance}}
\end{figure}

For further analysis of electrical transport properties, we have plotted resistivity
($\rho$) as a function of temperature as shown in Fig. \ref{fig:resistivity}. At
300 K, $\rho$ of the 51 nm thick film is found to be 0.66 m$\Omega$-cm which is very close to the earlier reports\cite{Bhosle2006_MIT_Ga_conc}. For 33 and 20 nm films, $\rho$ at 300 K is 0.77 and 1.3 m$\Omega$-cm respectively. Figures \ref{fig:resistivity}(a) and (b) display the plot of $\rho$ as a function of temperature for 33 and 20 nm films. As temperature
is lowered down from 300 K, $\rho$ decreases and reaches to a minimum
value. On further reduction in temperature below the transition temperature (T\textsubscript{MIT}), $\rho$ increases. The resistivity minimum
is generally observed in degenerate semiconductors\cite{Das2014_SiZnO_MIT}. Both the samples show positive temperature coefficient of resistivity (TCR) at high temperature
regime and below the T\textsubscript{MIT}, TCR becomes negative. Transition temperature (T\textsubscript{MIT}) is observed
at 125 K and 190 K for the films with thickness 33 and 20 nm respectively.
It should be noted that T\textsubscript{MIT} increases as the film thickness is reduced.
The negative TCR below the T\textsubscript{MIT} can be explained with the
frame of weak localization (WL) phenomena. In the case of weak
localization, due to quantum interference, the electronic wave functions
diffuse and thereby enhance the backscattering which in turn increases
the resistivity.

The conductivity in the weak localization regime
for a 2D system is given as \cite{Lee1985_RMP_disorder, Scherwitzl2011_LNO_MIT_2D_WL},
\begin{equation}
\sigma=\sigma_{0}+\frac{pe^{2}}{\pi h}ln\left(T/T_{0}^{'}\right)
\end{equation}

where, $\sigma_{0}$ is Drude conductance and $p$ is a parameter that depends on the nature of inelastic dephasing
mechanism. The value of $p$ is one for electron-electron collision
and 3 for electron-phonon collision\cite{Scherwitzl2011_LNO_MIT_2D_WL}. The constant $T_{0}^{'}$
is related to the mean free path for elastic collision. Insets of Fig. \ref{fig:resistivity} (a)
and (b) displays the $\sigma$ \textit{vs.} $ln(T)$ plot of the 33 nm
and 20 nm films at low temperature regime ($<$T\textsubscript{MIT}) respectively.
\begin{figure}
\includegraphics[width=7cm]{fig_4.png}
\caption{Temperature dependent resistivity plot of (a) 33, (b) 20 and (c) 6 nm thick Ga:ZnO films. Insets of (a) and (b) represent $\sigma$ \textit{vs.} $ln(T)$ plot of 33 and 20 nm thick films respectively. Inset of (c) shows the linear behavior
of $ln\left(\sigma\right)$ \textit{vs.} $1/T^{1/3}$ at low temperature regime below 50 K which indicates the existence of 2D Mott\textquoteright{}s VRH mechanism. Red lines are linear fitting to the experimental data.\label{fig:resistivity}}
\end{figure}
The linear behavior of $\sigma$ with $ln(T)$ can be observed
from these insets. The slopes are found to be $3.8\times10^{-4}$
and $5.2\times10^{-4}$ for the 33 nm and 20 nm samples respectively. Therefore, we infer that the transport follows the 2D weak localization behavior
in these samples. The resistivity value and the positive TCR suggests that the films with thickness
20 nm and above exhibits metallic type conduction. Moreover, the resistivity
at 5 K and 300 K are very close in these samples which also indicate
a metallic type behavior\cite{Mukherjee2017_IGZO}.

The temperature dependent resistivity plot of 6 nm thick film is reverse
to the trend shown by thicker samples.
From Fig. \ref{fig:resistivity}(c) we observe negative temperature coefficient of resistivity
(TCR) throughout the temperature range which conclusively predicts
the insulating behavior. The slope at low temperature regime ($<50$
K) is higher than the high temperature regime as can be realized from
Fig. \ref{fig:resistivity}(c). The two different slopes can be attributed to the two
different conduction mechanisms present in the two different temperature
scales. Generally at high temperature regime, the transport is mainly
governed by thermally activated band conduction. For thermally activated
band conduction, the conductivity is given by $\sigma\left(T\right)=\sigma'exp\left(-E_{a}/k_{B}T\right)$
where, $\sigma'$ is the pre-exponent factor, $E_{a}$ is the activation
energy. The Arrhenius fitting of the conductivity data yields the activation
energy as 27 meV. Resistivity sharply increases at the low temperature
regime below 50 K. At low temperature, the thermal energy is insufficient
to activate the carriers. Therefore, the conduction process mainly takes place
through the variable range hopping mechanism. In this process electrons
can hop from one site to other sites that are close to the Fermi level
irrespective of their spatial separation. The relation between electrical
conductivity and temperature for VRH mechanism in 2D system is given
by\cite{Stepina2009_2D_VRH},
\begin{equation}
\sigma\left(T\right)=\sigma'_{0}exp\left(-\frac{T_{0}}{T}\right)^{1/3}
\end{equation}

where, $\sigma'_{0}$ is a pre-exponent constant and $T_{0}$ is the
characteristic temperature. Inset of Fig. \ref{fig:resistivity}(c) displays the variation
of $ln\left(\sigma\right)$ \textit{vs.} $1/T^{1/3}$ of 6 nm thick sample
in the temperature range of 50 to 10 K. The linear fitting (red solid
line in Fig. \ref{fig:resistivity}(c)) of the plot indicates that the VRH mechanism is
dominant in the conduction below 50 K.
\subsection{Magnetoresistance}
\begin{figure}
\begin{centering}
\includegraphics[width=8cm]{fig_5.png}
\par\end{centering}

\caption{Magnetoresistance of (a) 33, (b) 20 and (c) 6 nm thick Ga:ZnO
film. Solid red lines in (c)
are fit of the experimental data according to the Eq. (\ref{eq:MR_Khosla-ve}). (d) Fitting parameter b vs. 1/T plot. The parameter b scales inversely with temperature as expected from Eq. (\ref{eq:b-parameter}). (e) MR\% as a function of film thickness at 10 K. As thickness decreases, MR amplitude increases in all temperatures.
Films with thickness 20 nm and above (metallic) show a small MR \%
compared to the 6 nm (insulating) film.  (f)The
first derivative of MR\% with respect to the applied field and (g) normalized MR of 33, 20 and 6 nm thick films. For metallic samples (20
nm and above), amplitude of first derivative is more at low field
and decreases as field is increased. This indicates that at low field
in these samples, the variation of MR\% is higher compared to the
high field regime. However, for 6 nm thick sample, amplitude of first
derivative increases with field and after reaching a maximum value
it decreases again.(h)Magnetoconductance behavior of 20 nm thick film which agrees well with 2D weak localization model. Red line is the fit according to the Eq. (\ref{eq:2D-WL})\label{fig:MR}}
\end{figure}
The resistivity data discussed in the last section not only infers metal insulator transition (MIT), but also it provides indication of strength of electron localization. The resistivity data of 6 nm film fits well with the proposed VRH model suggesting strong localization (SL). At the same time, the small upturn at low temperature regime for thicker (20 nm and above) films hints toward weak localization (WL) which agrees well with 2D WL model. In order to substantiate the MIT and to verify the nature of localization inferred from the resistivity data, we have carried out magnetoresistance measurement and the results are displayed in Fig. \ref{fig:MR}. As expected, from Fig. \ref{fig:MR} (a-c), we find that irrespective of the film thickness, the MR became more negative with decrease in temperature. The negative MR has been observed on Ga doped ZnO thin films by Reuss \textit{et al}\cite{Reuss2005_Gadoped_MR}. While the MR vs. H is qualitatively similar, there is a large quantitative difference between metallic and insulating samples. As film thickness decreases, magnitude of MR increases which can be realized from Fig. \ref{fig:MR}(e). At 10 K with 5 T field, the maximum MR observed in 20 and 33 nm thick samples is 1.3 and 2.4 \% respectively. Generally, in WL regime, a small magnitude ($\thicksim$1-2 \%) of MR is observed due to the suppression of WL in the presence of magnetic field\cite{Andrearczyk2006_NgMR_WL, Mukherjee2017_IGZO}.

To understand the variation of MR on applied magnetic field, we have plotted (d(MR)/dH) and normalized MR in Figs. \ref{fig:MR}(f) and (g) respectively. From the normalized plot, it can be observed that for the metallic samples, the magnitude of MR increases sharply at low field regime ($<0.5$ T). Therefore, we observe a large value of d(MR)/dH at small H (see Fig. \ref{fig:MR}(f)). On further increase in field strength, d(MR)/dH approaches to saturation indicating smaller and constant change in MR with field. This is because a small magnetic field can suppress the weak localization effect\cite{Mukherjee2017_IGZO}. Further increasing the field strength yields a lower change in MR. Therefore, the change in MR with respect to the field is more prominent at low magnetic field compared to the higher field.
To fit the experimental data, we have applied 2D WL model for 20 nm and above thick films. We have calculated the magnetoconductance (MC) for the metallic
samples defined as $MC=\sigma\left(H\right)-\sigma\left(H=0\right)$.
In these samples, temperature dependent resistivity measurement confirmed
the presence of weak localization at low temperature regime. In 2D
system, MC due to weak localization effect is given by \cite{Lee1985_RMP_disorder, Scherwitzl2011_LNO_MIT_2D_WL},
\begin{equation}
MC=\frac{e^{2}}{\pi h}\left[\varphi\left(\frac{1}{2}+\frac{1}{x}\right)+ln\left(x\right)\right]\label{eq:2D-WL}
\end{equation}
where, $e$ is electronic charge, $h$ is the Planck's constant, $\varphi$
is Digamma function. The term $x$ is defined as $x=l_{in}^{2}\frac{4eH}{h}$
where, $l_{in}$ is the inelastic scattering length. The magnetoconductance
data of Ga:ZnO films with thickness 20 nm and above are fitted to
the Eq. (\ref{eq:2D-WL}). Figure \ref{fig:MR}(h) shows magnetoconductance behavior of 20 nm thick film. It can be observed that the experimental data is well agreed with the 2D WL model. The inelastic scattering length was found to be 65 nm at 10 K and 15 nm at 100 K in this film.

Unlike to the metallic samples, the MR of 6 nm thick sample is different in two aspects. First the magnitude is much larger than that of metallic samples and the dependence of MR on H is also different. A maximum value of 12.7 \% MR is observed in 6 nm thick sample which shows strong localization behavior in the resistivity measurement. This magnitude of MR agrees with the MR observed in other doped ZnO films exhibiting strong insulating behavior\cite{Agrawal2014_Mgdoped_MR, Mukherjee2017_SbZnO}. For insulating sample d(MR)/dH vs. H curve exhibits a minimum at a magnetic field of 0.5 T at 10 K. Similar behavior is observed in La\textsubscript{0.7}Ca\textsubscript{0.3}MnO\textsubscript{3} system in the insulating regime\cite{Pekala2008_LCMO}. Below this minimum ($<0.5$ T), magnitude of d(MR)/dH increases with field. After reaching the maximum, the magnitude of d(MR)/dH decreases which indicates a slower rate of MR with field. In strong localization limit negative MR is explained by the scattering center. Upon application of a magnetic field, scattering centers align among themselves. As, a high field is required to align the scattering centers, d(MR)/dH first increases with field and after reaching the maximum value it starts decreasing. It has been observed by many\cite{Mukherjee2017_SbZnO} that in the strong localization regime, negative MR follows the semi-emperical Khosla-Fischer model\cite{Khosla1970_MR}. We have analyzed the MR data of 6 nm thick film with a semi-empirical model proposed by Khosla and Fischer. According to this model, negative MR is given by\cite{Khosla1970_MR},
\begin{equation}
MR=\frac{\triangle\rho}{\rho}=-a^{2}ln\left(1+b^{2}B^{2}\right)\label{eq:MR_Khosla-ve}
\end{equation}

The parameters a and b are defined as

\begin{equation}
a^{2}=A_{1}JN\left(E_{F}\right)\left[S\left(S+1\right)+\left\langle M^{2}\right\rangle \right]
\end{equation}

and

\begin{equation}
b^{2}=\left[1+4s^{2}\pi^{2}\left(\frac{2JN\left(E_{F}\right)}{g}\right)^{4}\right]\left(\frac{g\mu_{B}}{\alpha k_{B}T}\right)^{2}\label{eq:b-parameter}
\end{equation}

where constant $A_{1}$ is a measure of spin based scattering, $J$
is the exchange integral, $N(E_{F})$ is the density of states at
the Fermi energy, $S$ is the spin of the localized moment, $M$ is
the average magnetization, $g$ is the g-factor and $\alpha$ is a
numerical constant. The MR data of 6 nm thick Ga:ZnO film was fitted
using the Eq. (\ref{eq:MR_Khosla-ve}). Figure \ref{fig:MR}(c) suggests that experimental data are in good agreement with the Khosla
and Fischer model. The fitting parameter $b$ is plotted in Fig. \ref{fig:MR}(d) and it varies inversely with temperature as expected from Eq. (\ref{eq:b-parameter})\cite{Gacic2007_twobandmodel}. The other parameter $a$ slightly decreases with temperature\cite{Mukherjee2017_SbZnO, May2004_khosla_b, Peters2010_khosla_b}.

\section{Lattice disorder and electronic structure}

To examine the electronic contribution to the transport properties
of Ga:ZnO film, in this section we analyzed the band structure  of this doped system obtained from the DFT calculations.
While experimentally synthesized samples have 1$\%$ Ga doping, due to computational limitations, theoretical studies are performed with 3$\%$ Ga doping. The doping is realized by replacing one Zn atom by Ga in a $3\times3\times2$ supercell as shown in Fig. \ref{fig:Structural-disorder}(a).  Also due to computational limitations, calculations are performed only on bulk Ga:ZnO and two unit cell thick (1 nm) film along (001) (see Fig. \ref{fig:Structural-disorder}(e)). While the former can represent the experimentally synthesized thicker films, the latter closely stands for the  6 nm thin film. The film was constructed by introducing 15 \AA vacuum on the surface of 2 unit cell thick supercell as shown in the Fig. \ref{fig:Structural-disorder}(e). The ground state structure is realized through complete relaxation of the experimental structure of bulk ZnO\cite{Sans2009_doped_ZnO_BS}.

The calculations are carried out using the Vanderbilt
ultra-soft pseudopotential\cite{Vanderbilt1990_Pseudopotential} and plane wave basis sets as implemented in Quantum Espresso\cite{Giannozzi2009_QE} and within the frame work of PBE-GGA exchange correlation approximation\cite{Perdew1996_PBE}. The kinetic energy cutoff to fix the number of plane waves is taken
as 30 Ry. The Brillouin zone integration is performed using the Gaussian method on a $16\times16\times16$ k-grid for bulk and $16\times16\times1$ k-grid for film.
\begin{figure}
\begin{center}
\includegraphics[width=8cm]{fig_6.png}
\end{center}
\caption{Strength of crystallinity in Ga doped ZnO in bulk (a, b, c and d)
and in two unit cell thick (1nm) film (e, f, g and h). While (a) and
(e) show the fully relaxed structure, (b) and (f) indicate the distortion
occurring at the immediate neighborhoods of the dopant. Figs. (c)
and (g) plot the deviation of all nearest neighbor (NN) bond lengths
from that of the undoped bulk compound ($l-l_{0}$) . More
the deviation, both in count and magnitude, stronger the disorder.
Similarly, Figs. (d) and (h) plot the deviation of second neighbor
(SN) bond lengths. Total number of NN and SN bonds for the considered
$3\times3\times2$ supercell are 90.\label{fig:Structural-disorder}}
\end{figure}

With introduction of impurities (vacancy, dopant etc.), crystalline
disorder occurs and the extent of disorderness primarily depends on
the charge state and ionic radii of the impurities. Subsequently,
the disorder introduces new localized states and thereby alters the
transport properties of the compound.
To explain this aspect, we have measured the crystalline disorder
of Ga:ZnO both in bulk and thin film. The results are shown in Fig. \ref{fig:Structural-disorder}. Figures \ref{fig:Structural-disorder}(a) and (e) respectively show the fully relaxed
structure of bulk and film and it is visible that in film disorder is more than that in the bulk. Moreover, from Fig. \ref{fig:Structural-disorder}(b) we find that for bulk Ga:ZnO, the impurity Ga-O bond in the nearest neigborhood is isotropic with length 1.88 \AA. However, in the case of film (see Fig. 6(f)), the Ga neighborhood is highly anisotropic with Ga-O bond length varies from 1.45 to 1.92 \AA.

 To provide a quantitative measure of disorderness in the doped
system, we estimated the deviation of nearest neighbor (NN) and second
neighbor (SN) bond length from that of the undoped bulk ZnO. The results
are presented in Figs. \ref{fig:Structural-disorder}(c-d) and \ref{fig:Structural-disorder}(g-h). We find that out of 64 possible
NN bonds, a maximum of 7 bonds are deviated by 0.1 \AA from the
original value in the case of bulk compound. However, in the case
of film nearly 56 bonds are deviated significantly and the deviation
lies in the range of -0.5 to 0.2 \AA. The same can be realized from
the SN bond lengths even though the number of deviations is comparatively
less. Therefore, collectively we find that the extent of disorder
in film is significant and likely to impact the transport properties which can be understood from the band structure and density
of states.

ZnO is a known semiconductor with the Fermi level lies between the occupied O-p dominated valence bands and Zn-4s dominated dispersive conduction band.  Figure \ref{fig:Band-structure-DOS} (a) shows the band structure of pure ZnO plotted using the Brillouin zone of a 3$\times$3$\times$2 supercell. The lower conduction band is highly dispersive and with a large band width of 7 eV. Due to band folding this band consists of several subbands which are degenerate along A-L and also at H.  With Ga doping, both the orbital and crystal symmetry break along with a minor crystal disorder which is discussed in Fig. \ref{fig:Structural-disorder}. Such disorder and symmetry breaking lead to the lifting of the subband degeneracy (See the band dispersion in the range 1-3 eV for pure and -0.5 to 1 eV for doped sytem with respect to E$_F$) which can be seen from Fig. \ref{fig:Band-structure-DOS}(b). Now the Ga-p dominated lower subband, which is parabolic with width 2 eV and separated from the upper subbands of the conduction spectrum, is partially occupied to make the system a n-type degenerate semiconductor. The parabolic behavior of the impurity band is in good agreement with our experimental study where
we have shown that the thickest film (51 nm) exhibits positive temperature coefficient of resistivity like a metal. The density of states in Fig \ref{fig:Band-structure-DOS}(b) reflects the same where the tail
of the conduction band is smooth and well below E$_F$.

\begin{figure}
\includegraphics[width=7cm]{fig_7.png}\caption{ Band structure and DOS of (a) bulk ZnO, (b) bulk Ga:ZnO and (c) 2 unitcell thick Ga:ZnO film. The band structure for the bulk compounds are plotted using the Brillouin zone of a 3$\times$3$\times$2 supercell.For the film, the BZ is that of a 3$\times$3$\times$(2+vacuum) supercell (see Fig. \ref{fig:Structural-disorder}(e)).
As expected the bulk doped band structure represents  n-type degenerate semiconductor with a dispersive parabolic band crossing the Fermi level (E$_F$). In the case of film, the strong disorder introduces highly localized states, as indicated, to decrease the conductivity. The blue curve in the bottom is a guide to eye to distinguish between the delocalized and localized DOS of Ga:ZnO in bulk and film respectively. The high symmetry points defining the Brillouin zone of the bulk are A (0,0,0.47), L (0, 0.5, 0.47),$\Gamma$ (0, 0, 0), H (0.33, 0.33, 0.47),  and M (0, 0.5, 0). For 2 unit cell thin film, the considered high symmetry points are A (0,0,0.5), M (0, 0.5, 0), $\Gamma$ (0,0,0), and K (0.33, 0.33, 0.47).\label{fig:Band-structure-DOS}}
\end{figure}
As discussed earlier, the crystalline disorder in film is very profound in the film. As a consquence, the bulk covalent band gap disappears. Instead several localized state appears below and above E$_F$ as can be seen from the bands and DOS of Fig \ref{fig:Band-structure-DOS}(c).
 Since, one electron is donated by Ga, it is natural to see a partially occupied
band. Howevere, unlike to the bulk case, such a band is  of narrow band width due to localization effect.
From Fig. \ref{fig:Band-structure-DOS}(c) we observe that the
band width of this partially occupied band is approximately 0.2 eV which is
almost one order less than that of the doped bulk system signifying the strength
of the crystal disorder.

In earlier studies on hydrogenated amorphous Si\cite{Mott1980_gap_states}\cite{Okamoto1977_gap_states}, such
localized states in the vicinity of the E\textsubscript{F} have been
found. These states are  reported to be responsible for the insulating behavior, mediated through the VRH mechanism, of these compounds. The localized states of Ga:ZnO film
here resembles to that of amorphous Si and favor the insulating behavior. The experimentally observed insulating behavior of 6 nm film resonates with the lattice disorder led insulating phenomena of 2 unit cell thick film. We may note that Fig. \ref{fig:XRD} indicates a broad peak in the XRD pattern of the 6 nm film as opposed to the narrow peaks shown by the 20 nm and above thicker films.

\section{Summary and conclusions}

To sumarize, structural, electrical and magnetotransport properties
of PLD grown Ga doped ZnO thin films were extensively studied and
analyzed. Metal-insulator transition was observed as a function of
the film thickness. While the sample with thickness of 20 nm and above exhibits
metallic type of conduction, the film of 6 nm thick is
found to be insulating. Temperature dependent resistivity and magnetoresistance measurements further confirm the presence of weak localizatin in 20 nm and above thicker samples. However, strong localization is observed in 6 nm thick film as resistivity data follow Mott-VRH model. Moreover, the magnetoresistance data is well matched with Khosla-Fischer model to further substantiate the strong localization behavior in 6 nm film. Our structural relaxation calculations, carried out using density functional theory, suggests that the dopant as well surface reconstruction introduce larger disorder in thinner samples. As a consequence, the system has disorder induced localized states in the vicinity of Fermi level. The donor electrons occupy some of these localized states to weaken the mobility and exhibit insulating behavior.
\begin{acknowledgments}
Authors thank Department of Science and Technology (DST),
New Delhi, that facilitated the establishment of \textquotedblleft{}Nano
Functional Materials Technology Centre\textquotedblright{} (Grant:
SRNM/NAT/02-2005). J. M. thanks Mr. Ajit Jana for his help in DFT calculations. Institute HPCE facility was used for the computations. J. M. wants to thank UGC for SRF grant.\end{acknowledgments}

redpajama_set_name: RedPajamaArXiv


text:
\section{Introduction}
	The success of deep convolutional neural networks (CNNs) has been well demonstrated in several real-world applications, \emph{e.g. }, image classification~\cite{krizhevsky2012imagenet,oguntola2018slimnets}, object detection~\cite{ren2015faster}, semantic segmentation~\cite{noh2015learning}, and low-level computer vision~\cite{tai2017image}. Massive parameters and huge computational complexity are usually required for achieving the desired high performance, which limits the application of these models to portable devices such as mobile phones and smart cameras.

	In order to reduce the computational costs of deep neural networks, a number of works have been proposed to compress and accelerate the original cumbersome model into a portable one. For example, filter pruning methods~\cite{luo2017thinet,zhuang2018discrimination} aim to sort the filters based on their importance scores in which unimportant filters will be treated as redundancy and removed to derive lightweight architectures. Knowledge distillation methods~\cite{hinton2015distilling,park2019relational} are explored to transfer the knowledge inside a teacher model to a student model. Tensor decomposition method~\cite{rabanser2017introduction} decomposes the original large weight tensor into several small ones to simplify the overall inference process. Model quantization methods~\cite{gupta2015deep,hubara2016binarized} quantize the widely used 32-bit floating point weights and activations into low-bit ones to save the memory consumption and computational cost.

	Among the aforementioned algorithms, binary neural network (BNN) is an extreme case of model quantization method in which the weights and activations in the original neural network will be quantized to 1-bit (-1 and +1) values. Obviously, the memory usage for weights and activations in BNNs are 32$\times$ lower than that in the full-precision network with the same model architecture. BNN was firstly proposed by Hubara \emph{et.al.}~\cite{hubara2016binarized} by utilizing straight-through estimator (STE)~\cite{bengio2013estimating} for handling gradient problem of sign function. Then, XNOR-Net~\cite{rastegari2016xnor} introduced a channel-wise scaling factor for weights to reduce the quantization error. Dorefa-Net~\cite{zhou2016dorefa} used a single scaling factor for all the channels and achieved competitive performance. In addition, there are a number of methods for optimizing the original structure to improve the performance of learned BNN. ABCNet~\cite{lin2017towards} used multiple parallel binary convolutional layers. Bi-Real Net~\cite{liu2018bi} inserted skip-connections to enhance the feature representation ability of deep BNNs.

	\begin{figure*}[t]
		\centering
		\includegraphics[width=1\columnwidth]{fig/functions.png}%
		\caption{From top to bottom: original functions in spatial domain, corresponding functions in frequency domain and the difference between the current function and sign function in frequency domain. From left to right: sign function, combination of sine functions, tanh function in~\cite{gong2019differentiable} and SignSwish function in~\cite{darabi2018bnn} (short as SS).}
		\label{fig:1}
	\end{figure*}

	The optimization difficulty caused by the sign function in BNNs is a very important issue. In practice, the original full-precision weights and activations will be quantized into binary ones with sign function during the feed-forward procedure, but the gradient of sign function cannot be directly used for back-propagation. Therefore, most of the approaches mentioned above adopt the straight-through estimator (STE)~\cite{bengio2013estimating} for gradient back-propagation to train BNNs. The inconsistency between the forward and backward pass caused by the STE method prevents BNN from achieving better performance. Additionally, several attempts were proposed to estimate the gradient of sign function for replacing STE. For example, DSQ~\cite{gong2019differentiable} introduced a tanh-alike differentiable asymptotic function to estimate the forward and backward procedures of the conventional sign function. BNN+~\cite{darabi2018bnn} used a SignSwish activation function to modify the back-propagation of the original sign function and further introduced a regularization that encourages the weights around binary values. RBNN~\cite{lin2020rotated} proposed a training-aware approximation function to replace the sign function when computing the gradient. Although these methods have made tremendous effort for refining the optimization of binary weights and activations, the approximation function they used may corrupt the \textbf{main direction} of de facto gradient for optimizing BNNs.

	To better maintain the main direction of de facto gradient in BNNs, we propose a frequency domain approximation approach by utilizing the Fourier Series (FS) to estimate the original sign function in the frequency domain. The FS estimation is a lossless representation of the sign function when using infinite terms. In practice, the high-frequency coefficients with relatively lower energy will be ignored to avoid the huge computational overhead, and the sign function will be represented as the combination of a fixed number of sine functions with different periods. Compared with the existing approximation methods, the proposed frequency domain approximation approach does not affect the low-frequency domain information of the original sign function, \emph{i.e. }, the part that occupies most energy of sign function as shown in Fig.~\ref{fig:1}. Thus, the main direction of the corresponding gradient \emph{w.r.t. } the original sign function can be more accurately maintained. In addition, to further compensate the subtle approximation error, we explore a noise adaptation module in the training phase to refine the gradient. To verify the effectiveness of the proposed method, we conduct extensive experiments on several benchmark datasets and neural architectures. The results show that our method can surpass other state-of-the-art methods for training binary neural networks with higher performance.

	\section{Approach}
	In this section, we first provide the preliminaries of BNNs and STE briefly. Then, we propose to estimate sign function with the combination of sine functions and analyze the inaccurate estimation problem. Finally, we introduce a noise adaptation module to solve the problem and derive our FDA-BNN framework.

	\subsection{Preliminaries}
	The conventional BNNs quantize the weight matrix $W\in\mathbb R^{c_i\times c_o\times k\times k}$ and activation matrix $A\in\mathbb R^{b\times c_i\times h\times w}$ to 1-bit, and obtain the binary matrices $W_b\in\mathbb R^{c_i\times c_o\times k\times k}$ and $A_b\in\mathbb R^{b\times c_i\times h\times w}$ for the intermediate convolution layers using sign function as follows:
	\begin{equation}
	w_b = \left\{
	\begin{aligned}
	1 &, & w_{f}>0,\\
	-1 &, & w_{f}\leq 0,
	\end{aligned}
	\right.
	~~a_b = \left\{
	\begin{aligned}
	1 &, & a_{f}>0,\\
	-1 &, & a_{f}\leq 0,
	\end{aligned}
	\right.
	\end{equation}
	where $w_{f}$ and $a_{f}$ are the elements in the original 32-bit floating point weight matrix $W$ and activation matrix $A$, respectively, and $w_{b}$ and $a_{b}$ are the elements in the binarized weight matrix $W_b$ and activation matrix $A_b$, respectively. Given the 1-bit weights and activations, the original convolution operation can be implemented with bitwise XNOR operation and the bit-count operation~\cite{liu2018bi}. During training, both the full-precision weight matrix $W$ and the 1-bit weight matrix $W_b$ are used, while only $W_b$ is kept for inference.

	Note that it is hard to apply back-propagation scheme on the sign function. The gradient of sign function is an impulse function that is almost zero everywhere and is unable to be utilized in training. Thus, the STE method~\cite{bengio2013estimating} is introduced to compute its gradient as:
	\begin{equation}
	\frac{\partial \mathcal L}{\partial w}=Clip(\frac{\partial \mathcal L}{\partial w_b},-1,1),
	\label{eq:ste}
	\end{equation}
	in which $\mathcal L$ is the corresponding loss function for the current task (\emph{i.e. }, cross-entropy loss for image classification) and
	\begin{equation}
	Clip(x,-1,1) = \left\{
	\begin{aligned}
	-1 &, ~~\text{if } x<-1,\\
	x &, ~~\text{if } -1 \leq x<1.\\
	1 &, ~~\text{otherwise}.
	\end{aligned}
	\right.
	\end{equation}

	Since STE is an inaccurate approximation of the gradient of sign function, many works have focused on replacing STE during back-propagation. BNN+~\cite{darabi2018bnn} introduced a SignSwish function, while DSQ~\cite{gong2019differentiable} proposed a tanh-alike function and minimized the gap between the sign function with a learnable parameter. Such methods focus on approximating sign function in spatial domain, and corrupt the main direction of the gradient. Thus, we propose a new way to estimate sign function to maintain the information in frequency domain.

	\subsection{Decomposing Sign with Fourier Series}
	In the following sections, we use $f(\cdot)$ and $f'(\cdot)$ to denote an original function and its corresponding gradient function.

	It has been proved in the area of signal processing that any periodical signal with period $T$ can be decomposed into the combination of Fourier Series (FS):
	\begin{equation}
	f(t)=\frac{a_0}{2}+\sum_{i=1}^{\infty}[a_i\cos(i\omega t)+b_i\sin(i\omega t)],
	\end{equation}
	in which $\omega=2\pi/T$ is the radian frequency, $a_0/2$ is the direct component and $\{b_i\}_{i=1}^\infty$ ($\{a_i\}_{i=1}^\infty$) are the coefficients of the sine (cosine) components. Specifically, when the periodical signal is the square wave, we have
	\begin{equation}
	a_i=0 \text{ for all } i; ~~
	b_i = \left\{
	\begin{aligned}
	\frac{4}{i\pi} &, & \text{if $i$ is odd},\\
	0 &, & \text{otherwise}.
	\end{aligned}
	\right.
	\end{equation}
	and derive the FS for the square wave $s(t)$:
	\begin{equation}\label{eq:wave}
	s(t) = \frac{4}{\pi} \sum_{i=0}^\infty \frac{\sin\left((2i+1)\omega t\right) }{2i+1}.
	\end{equation}
	Note that when restricting the signal into a single period, the sign function is exactly the same as the square wave:
	\begin{equation}
	sign(t)=s(t),~~|t|< T.
	\end{equation}
	Thus, the sign function can also be decomposed into the combination of sine functions, and its derivative is computed as:
	\begin{equation}
	s'(t) = \frac{4\omega}{\pi} \sum_{i=0}^\infty \cos\left((2i+1)\omega t\right) .
	\label{eq:dsin}
	\end{equation}
	In this way, we propose to replace the derivative used in STE (Eq.~\ref{eq:ste}) with Eq.~\ref{eq:dsin} during back-propagation for better approximating the sign function.

	\subsection{Inaccurate Estimation Problem in FDA-BNN}
	The FS decomposition is a lossless representation of the sign function when using infinite terms by transforming the signal from spatial domain into frequency domain, as shown in Theorem~\ref{theory1}. First, we can rewrite Eq.~\ref{eq:wave} as
	\begin{equation}
	\hat s_n(t) = \frac{4}{\pi} \sum_{i=0}^n \frac{\sin\left((2i+1)\omega t\right) }{2i+1},
	\label{eq:snt}
	\end{equation}
	where $n$ is the number of FS terms. The corresponding derivative is
	\begin{equation}
	\hat s_n'(t) = \frac{4\omega}{\pi} \sum_{i=0}^n \cos\left((2i+1)\omega t\right).
	\label{eq:dsnt}
	\end{equation}
	In the following, we show that as $n$ increases, the mean squared error between the estimator $\hat s_n(t)$ and the ground-truth $s(t)$ decreases, and converges to $0$ when $n\rightarrow \infty$.
	\begin{myTheo}\label{theory1}
		(Mean Square Convergence~\cite{rust2013convergence}) Given $s(\cdot)$ as the square wave function which is integrable in a period, and $\hat s_n(\cdot)$ is the $n^{th}$ partial sum of the Fourier Series of $s(\cdot)$, then $\hat s_n(\cdot)$ converges to $s(\cdot)$ in the mean square sense, \emph{i.e. },
		\begin{equation}
		\frac{1}{T}\int_T ||s(t)-\hat s_n(t)||_2^2 dt\rightarrow 0 ~~\text{  as  } ~~n\rightarrow\infty.
		\end{equation}
	\end{myTheo}

	The above theorem shows that the estimator $\hat s_n(t)$ converges to the ground-truth $s(t)$ when using an infinite number of FS terms, and an error
	\begin{equation}
	r(t)=s(t)-\hat s_n(t)
	\label{eq:et}
	\end{equation}
	occurs in reality when $n$ is finite. The error comes from the infinite high-order FS terms and has little energy. In practice, we use a fixed number of FS terms for two reasons: 1) We can avoid the huge computational overhead by ignoring the high-frequency coefficients. 2) We can still accurately maintain the main direction of the corresponding gradient \emph{w.r.t. } the original sign function. Compared to other estimation methods that directly estimate sign function in the spatial domain, our method benefits to the approximation process since the proposed approximation approach does not influence the low-frequency domain information of the original sign function, which occupies most of the energy.

	Note that when doing back-propagation in BNN, the gradient of sign function $s'(t)$ is not the optimal choice and is never used as the optimal gradient. However, the optimal gradient $s'^*(t)$ which will reduce the loss function to the global optimal should not be far away from $s'(t)$. Thus, we can treat $s'(t)$ as contaminating with noise, \emph{i.e. },
	\begin{equation}
	s'(t)=s'^{*}(t) + n'(t),
	\label{eq:nt}
	\end{equation}
	in which $n'(t)$ is the gradient of unknown noise function $n(t)$. Combining Eq.~\ref{eq:et} and Eq.~\ref{eq:nt}, we have:
	\begin{equation}
	s'^*(t) = \hat s'_n(t)+\hat e'(t),
	\end{equation}
	in which $\hat e'(t)=r'(t)-n'(t)$. Thus, the optimal gradient of sign function is divided into two parts. The first part is the gradients of finite number of low-order FS terms which is the main part of the estimation. The second part includes the error from infinite high-order FS terms and noise function, which is inexpressible since $r'(t)$ goes to infinity at $t=0$ and $n'(t)$ is unknown.

	\begin{figure}[t]
		\centering
		\includegraphics[width=0.9\columnwidth]{fig/noise_layer2.png}%

		\caption{\textbf{(a)} The forward pass of the combination of sign function and noise adaptation module. \textbf{(b)} Corresponding backward pass of (a). \textbf{(c)} The forward pass of the combination of the sine module and the noise adaptation module. \textbf{(d)} Corresponding backward pass of (c). The actual forward and backward pass are \textbf{bolded} in the figure.}
		\label{fig:2}

	\end{figure}

	\subsection{Solving the Inaccurate Estimation Problem}
	In the previous works, there are several ways to deal with noise. The first is label correction which correct the wrong labels into the right one~\cite{lee2018cleannet}. The second is to change the training framework to derive a robust network for noisy labels~\cite{han2018co,han2018masking,yu2019does,goldberger2016training}. The third is to improve the training loss function to a noise robust loss~\cite{zhang2018generalized}.

	Different from them, we propose to use a learnable neural network to solve the problem. Specifically, given the input vector ${\bf t}\in\mathbb R^{1\times d}$, the process of sign function can be replaced by the combination of a sine module $\hat s_n(\cdot)$ and a noise adaptation module $\hat e(\cdot)$ which estimates the error $r(\cdot)-n(\cdot)$ using two fully-connected layers with ReLU activation function and a shortcut:
	\begin{equation}
	\hat e({\bf t}) = \sigma({\bf t}W_1)W_2+\eta({\bf t}),
	\label{eq:e}
	\end{equation}
	in which $\sigma(t)=\max(t,0)$, $W_1\in\mathbb R^{d\times\frac{d}{k}}$ and $W_2\in\mathbb R^{\frac{d}{k}\times d}$ are the parameters of the two FC layers in which $k$ is used to reduce the number of parameters and is set to 64 in the following experiments. $\eta(\cdot)$ is the shortcut connection. During the experiment we use different form of $\eta(\cdot)$, details can be found in Sec.3.2.

	\begin{algorithm}[t]
		\setstretch{1.}
		\caption{Feed-Forward and Back-Propagation Process of a convolutional layer in FDA-BNN.}
		\label{alg:alg1}
		\begin{algorithmic}[1]

			\Require A convolutional layer with weights $W$ and input features (activations) $A$, the noise adaptation module with parameters $W_{a1}$ ($W_{w1}$) and $W_{a2}$ ($W_{w2}$) for activations (weights), the loss function $\ell_{ce}$, the learning rate $\gamma$, and training iterations $T$.
			\For{$\tau=1$ to $T$}
			\State \textbf{Feed-Forward:}
			\State Quantize the weights and activations: $W_b=$ Sign($W$), $A_b=$ Sign($A$);
			\State Apply the convolutional operation on the binarized weights and activations: $W_b \otimes A_b$;

			\State \textbf{Back-Propagation:}
			\State Compute the gradient of $W_b$ and $A_b$ using standard back-propagation, and get $\frac{\partial\ell_{ce}}{\partial {W_b}}$ and $\frac{\partial\ell_{ce}}{\partial {A_b}}$;
			\State Compute the gradient of $A$, and derive $\frac{\partial\ell_{ce}}{\partial {A}}$ using Eq.~\ref{eq:dx};
			\State Compute the gradients on parameters of noise adaptation module for activations: $\frac{\partial\ell_{ce}}{\partial {W_{a1}}}$ and $\frac{\partial\ell_{ce}}{\partial {W_{a2}}}$ using Eq.~\ref{eq:dw1} and Eq.~\ref{eq:dw2}.
			\State Compute the gradients of $W$ and derive $\frac{\partial\ell_{ce}}{\partial {W}}$, and also $\frac{\partial\ell_{ce}}{\partial {W_{w1}}}$ and $\frac{\partial\ell_{ce}}{\partial {W_{w2}}}$, with the same manner in step 7$\sim$8.
			\State \textbf{Parameter Update:}
			\State $W\leftarrow \textbf{UpdateParameters}(W,\frac{\partial\ell_{ce}}{\partial W},\gamma)$;
			\State $W_{a1}\leftarrow \textbf{UpdateParameters}(W_{a1}, \frac{\partial\ell_{ce}}{\partial W_{a1}}, \gamma)$
			\State $W_{a2}\leftarrow \textbf{UpdateParameters}(W_{a2}, \frac{\partial\ell_{ce}}{\partial W_{a2}}, \gamma)$;
			\State $W_{w1}\leftarrow \textbf{UpdateParameters}(W_{w1}, \frac{\partial\ell_{ce}}{\partial W_{w1}},\gamma)$
			\State $W_{w2}\leftarrow \textbf{UpdateParameters}(W_{w2}, \frac{\partial\ell_{ce}}{\partial W_{w2}},\gamma)$.
			\EndFor
		\end{algorithmic}
	\end{algorithm}

	Note that there are two benefits from adding the shortcut connection. The first is that it helps the flow of gradient and alleviates the gradient vanishing problem~\cite{liu2019towards,philipp2018gradients,huang2018learning}. The second is that given $\eta(t)=a\cdot t$ as an example, we have $\eta'(t)=a$ which equals to adding a bias to the gradient. This benefit the training process since given limited training resources and time, only finite number of FS terms $n$ can be used to compute $\hat s_n(t)$ in Eq.~\ref{eq:snt}, and raise the problem that the tail of the estimated gradient $\hat s_n'(t)$ oscillates around $0$ at a very high frequency.
	This phenomenon does harm to the optimization process, since a very little disturbance on the input will results in two different gradients with opposite direction, and adding a bias term on the gradient helps to alleviate this problem.

	Based on the analysis above, given Eq.~\ref{eq:snt} and Eq.~\ref{eq:e}, we are able to approximate sign function with the combination of sine module and noise adaptation module:
	\begin{equation}
	{\bf z} = \hat s_n({\bf t}) + \alpha\hat e(\bf t),
	\label{eq:y}
	\end{equation}
	in which $\alpha$ is the hyper-parameter to control the influence of noise adaptation module.

	In order to keep the weights and activations as 1-bit in inference, the sine module is used only in the backward pass and replaced as sign function in the forward pass, as shown in Fig.~\ref{fig:2}. Besides, $\alpha$ is gradually decreased during training and reaches 0 at the end of training to avoid the inconsistency between training phase and inference phase. The weights of both BNN and noise adaptation module are updated in an end-to-end manner by minimizing the cross-entropy loss $\ell_{ce}$.

	Specifically, given Eq.~\ref{eq:y} and the loss function $\ell_{ce}$, the gradients in Fig.~\ref{fig:2} (d) can be computed as:
	\begin{align}
	\frac{\partial \ell_{ce}}{\partial {\bf t}}

	&=\frac{\partial \ell_{ce}}{\partial {\bf z}} W_2^\top\odot\left(({\bf t}W_1)\geq 0 \right)W_1^\top+ \frac{\partial \ell_{ce}}{\partial {\bf z}}\eta'({\bf t}) \notag \\
	&+ \frac{\partial \ell_{ce}}{\partial {\bf z}} \odot \frac{4\omega}{\pi} \sum_{i=0}^n \cos\left((2i+1)\omega {\bf t}\right),
	\label{eq:dx}
	\end{align}
	in which $\frac{\partial \ell_{ce}}{\partial {\bf z}}$ is the gradient back-propagated from the upper layers, $\odot$ represents element-wise multiplication, and $\frac{\partial \ell_{ce}}{\partial {\bf t}}$ is the partial gradient on ${\bf t}$ that back-propagate to the former layer.

	\begin{align}
	\label{eq:dw1}
	\frac{\partial \ell_{ce}}{\partial W_1}

	&={\bf t}^\top\frac{\partial \ell_{ce}}{\partial {\bf z}} W_2^\top\odot\left(({\bf t}W_1)\geq 0 \right);\\
	\notag\\
	\frac{\partial \ell_{ce}}{\partial W_2}

	&=\sigma({\bf t}W_1)^\top\frac{\partial \ell_{ce}}{\partial {\bf z}} ,
	\label{eq:dw2}
	\end{align}
	in which $\frac{\partial \ell_{ce}}{\partial W_1}$ and $\frac{\partial \ell_{ce}}{\partial W_2}$ are gradients used to update the parameters in the noise adaptation module.

	The formulations from Eq.~\ref{eq:snt} to Eq.~\ref{eq:dw2} are applied for both weights and activations for replacing the back-propagation of sign function. The feed-forward and back-propagation process of a convolutional layer in FDA-BNN is summarized in Alg.~\ref{alg:alg1}.

	\section{Experiments}
	In this section, we evaluate the proposed method on several benchmark datasets such as CIFAR-10~\cite{krizhevsky2009learning} and ImageNet~\cite{deng2009imagenet} to show the superiority of FDA-BNN. All of the proposed FDA-BNN models follow the rule in previous methods that all the convolutional layers except the first and last layers are binarized~\cite{zhou2016dorefa,bulat2019xnor,darabi2018bnn}.

	\begin{table*}[t]
		\centering
		\renewcommand{\arraystretch}{1.15}
		\caption{Experimental results on CIFAR-10 using different 1-bit quantized models. Bit-width (W/A) denotes the bit length of weights and activations, and BackProp denotes the way of computing gradients and updating parameters.} \label{tab:c10}
		\vspace{0.em}
		\setlength{\tabcolsep}{6pt}{
			\begin{tabular}{c||c|c|c|c}
				\hline
				Network & Method & Bit-width (W/A) & BackProp & Acc (\%) \\
				\hline\hline
				\multirow{8}{*}{ResNet-20} &FP32 &32/32 &- & 92.10\\
				\cline{2-5}
				&Dorefa-Net~\cite{zhou2016dorefa}  &1/1 & STE & 84.44\\
				&XNOR-Net~\cite{rastegari2016xnor} &1/1 &STE & 85.23\\
				&LNS~\cite{han2020training} & 1/1 &STE & 85.78\\
				&TBN~\cite{wan2018tbn}&1/2&STE & 84.34\\
				\cline{2-5}
				&DSQ~\cite{gong2019differentiable} &1/1 & Tanh-alike &84.11 \\

				&IR-Net~\cite{qin2020forward} & 1/1 & Tanh-alike &85.40\\
				& FDA-BNN & 1/1 &Fourier Series & \textbf{86.20}\\
				\hline\hline
				\multirow{8}{*}{VGG-small} &FP32 &32/32 &- & 94.10\\
				\cline{2-5}
				& Dorefa-Net~\cite{zhou2016dorefa}  & 1/1 & STE & 90.20\\
				& BinaryNet~\cite{hubara2016binarized} &1/1 & STE & 89.90\\
				& XNOR-Net~\cite{rastegari2016xnor} & 1/1 & STE & 89.80\\
				\cline{2-5}
				& BNN+~\cite{darabi2018bnn} & 1/1 & SignSwish & 91.31\\
				& DSQ~\cite{gong2019differentiable} & 1/1 & Tanh-alike & 91.72\\
				& IR-Net~\cite{qin2020forward} & 1/1 & Tanh-alike & 90.40\\

				& FDA-BNN & 1/1 &Fourier Series & \textbf{92.54}\\
				\hline
			\end{tabular}
		}
	\end{table*}

	\subsection{Experiments on CIFAR-10}
	The CIFAR-10~\cite{krizhevsky2009learning} dataset contains 50,000 training images and 10,000 test images from 10 different categories. Each image is of size $32\times32$ with RGB color channels. Data augmentation methods such as random crop and random flip are used during training. During the experiments, we train the models for 400 epochs with a batch size of 128 and set the initial learning rate as 0.1. The SGD optimizer is used with momentum set of 0.9 and weight decay of 1e-4.

	The widely-used ResNet-20 and VGG-Small architectures in BNN literature~\cite{liu2018bi,gong2019differentiable} are used to demonstrate the effectiveness of the proposed method. Dorefa-Net~\cite{zhou2016dorefa} is used as our baseline quantization method, and the sine module and noise adaptation module are used during training. Other competitors include: 1) methods using STE for back-propagation, \emph{e.g. }, BNN~\cite{hubara2016binarized}, Dorefa-Net~\cite{zhou2016dorefa}, XNOR-Net~\cite{bulat2019xnor}, TBN~\cite{wan2018tbn}; 2) methods using approximated gradient rather than STE during back-propagation, \emph{e.g. }, DSQ~\cite{gong2019differentiable}, BNN+~\cite{darabi2018bnn}, IR-Net~\cite{qin2020forward}; 3) fine-tuning methods, \emph{e.g. }, LNS~\cite{han2020training}. The experimental results are shown in Tab.~\ref{tab:c10}.

	\begin{table}[t]
		\vspace{-1em}
		\centering
		\renewcommand{\arraystretch}{1.}
		\caption{Ablation study on sine module and noise adaptation module.} \label{tab:ab1}

		\setlength{\tabcolsep}{6pt}{
			\begin{tabular}{c|c|c}
				\hline
				Sine Module& Noise Adaptation Module & Acc (\%)\\
				\hline\hline
				$\times$ & $\times$ & 84.44\\
				\checkmark &$\times$ & 85.83\\

				\checkmark& \checkmark& \textbf{86.20}\\
				\hline
			\end{tabular}
		}
	\end{table}

	The results show that the proposed FDA-BNN method outperforms all the competitors and achieves state-of-the-art accuracy. For ResNet-20 architecture, the proposed FDA-BNN achieves an accuracy of 86.20\% which improves the baseline model by 1.76\%, and is 0.42\% higher than the previous SOTA method LNS~\cite{han2020training} which is a fine-tuning based method and costs 120 more epochs than our method. When applying the proposed method to VGG-small model, the proposed method achieves an accuracy of 92.54\% which is also the highest and improves the baseline model by 2.34\%. FDA-BNN method outperforms other gradient estimation methods which approximate sign function in spatial domain such as DSQ~\cite{gong2019differentiable}, BNN+~\cite{darabi2018bnn} and IR-Net~\cite{qin2020forward}.

	\subsection{Ablation Study}
	In this section, we conduct several ablation studies to further verify the effectiveness of each component in the proposed method, the impact of shortcut in noise adaptation module and the effect of hyper-parameters.

	Firstly, we conduct experiments to understand the effect of sine module and noise adaptation module. The models are verified on CIFAR-10 using ResNet-20 architecture. As shown in Tab.~\ref{tab:ab1}, the first line represent the standard Dorefa-Net baseline. We can see that only use sine module can benefit the training process and improve the accuracy from 84.44\% to 85.83\%.
	When combining the sine module and noise adaptation module together, we reach the best performance, \emph{i.e. }, 86.20\% accuracy, which shows the priority of using both modules in FDA-BNN method during training.

	In order to further verify the usefulness of the noise adaptation module, we plug the module into other gradient approximation methods such as DSQ~\cite{gong2019differentiable} and BNN+~\cite{darabi2018bnn}. The experiments are conducted on CIFAR-10 dataset with ResNet-20 model, and the results in Tab.~\ref{tab:ab3} show that fitting the error between the estimated gradient and optimal gradient benefits to different gradient approximation methods.

	\begin{table}[htb]
		\centering
		\renewcommand{\arraystretch}{1.1}
		\caption{Plugging noise adaptation module into different methods.} \label{tab:ab3}
		\vspace{0.em}
		\setlength{\tabcolsep}{6pt}{
			\begin{tabular}{c|c|c}
				\hline
				& w/o noise module& w/ noise module\\
				\hline\hline
				DSQ & 84.11 & \textbf{84.46}\\

				BNN+ &84.59 &\textbf{84.87} \\
				FDA-BNN & 85.83 & \textbf{86.20}\\
				\hline
			\end{tabular}
		}
		\vspace{-0.em}
	\end{table}

	Then we evaluate the influence of using different $\eta(\cdot)$ on the branch of shortcut in noise adaptation module. As shown in Tab.~\ref{tab:ab2}, using shortcut results in a better performance than not using it, and the shortcut function $\eta(x)=a\sin(x)$ performs the best during the experiment. Note that this is consistent to the analysis in Sec.2.3. Given $\eta(x)=a\sin(x)$, the gradient flow can still be strengthened. Besides, the gradient $\eta'(x)=a\cos(x)$ equals to adding a positive bias in the given range, and thus can still alleviate the problem that the tail of the estimated gradient $\hat s_n'(x)$ oscillates around $0$ with a very high frequency. $a$ is set to 0.1 during the experiment.
	\begin{table}[htb]

		\centering
		\renewcommand{\arraystretch}{1.}
		\caption{Ablation study on $\eta(\cdot)$.} \label{tab:ab2}
		\setlength{\tabcolsep}{8pt}{
			\begin{tabular}{c|c}
				\hline
				Functions of shortcut branch & Acc (\%)\\
				\hline\hline
				$\eta(x)=$0 (w/o shortcut) & 85.27\\
				$\eta(x)=$$ax$ & 85.93\\
				$\eta(x)=$$a\sin(x)$ & \textbf{86.20}\\
				\hline
			\end{tabular}
		}
	\end{table}

	Using different number of FS terms during training will influence the final performance of BNN. Intuitively, a larger $n$ will reduce the difference between $\hat s_n(\cdot)$ and $sign(\cdot)$, but at the same time lead to more time on training since a combination of $n$ gradients will be computed (Eq.~\ref{eq:dsnt}). We show the training time spent and the final accuracy achieved by using different $n$ in Fig.~\ref{fig:acc} (a). We can see that as the number of FS terms $n$ increases, the training time continues to increase while the accuracy saturates at about $n=10$, which means a moderate $n$ is enough for training an accurate BNN.

	\begin{figure*}[t]
		\centering
		\begin{tabular}{cc}
			\includegraphics[width=0.45\linewidth]{fig/acc.png} &
			\includegraphics[width=0.43\linewidth]{fig/acc2.png}\\
			(a)  & (b) \\
		\end{tabular}
		\caption{(a) Training time spent and the final accuracy achieved when using different number of FS terms. The blue line denotes the accuracy and the red bar denotes the training time. (b) Accuracies under different settings of hyper-parameter $n$ during training.}
		\label{fig:acc}
	\end{figure*}

	As shown in Fig.~\ref{fig:acc} (a), a small number of FS terms do harm to the performance of BNN while a large $n$ leads to a waste of resources and time. Hence, we explore three different settings of using hyper-parameter $n$ during training.
	\begin{itemize}
		\item Setting 1: the number of FS terms $n$ is fixed to a pre-defined number $n_p$ during training.
		\item Setting 2: the number of FS terms $n$ is gradually increased from 1 to a pre-defined number $n_p$.
		\item Setting 3: the number of FS terms $n$ is gradually increased from 10 (best result in setting 1) to a pre-defined number $n_p$.
	\end{itemize}

	Results of different settings are shown in Fig.~\ref{fig:acc} (b). Setting 1 performs well at a wider range than setting 2 and they reach a comparable performance when $n_p$ is large enough. Setting 3 performs the best, since it start from a moderate $n$ and avoid the large gap between $\hat s_n(\cdot)$ and $sign(\cdot)$ at the beginning of training, and is able to approach the sign function as the training proceed. Thus, in the following experiments we start $n$ from a moderate number $n_s$ (\emph{e.g. }, $n_s=10$) and gradually increase it to $n_p=2n_s$.

	\begin{table*}[t]
		\centering
		\renewcommand{\arraystretch}{1.15}
		\caption{Experimental results on ImageNet using different 1-bit quantization methods. `W/A' denotes the bit-width of weights and activations, and BackProp denotes the way of computing gradients and updating parameters.} \label{tab:imgnet}
		\setlength{\tabcolsep}{6pt}{
			\begin{tabular}{c||c|c|c|c|c}
				\hline
				Network & Method & W/A  & BackProp & Top-1 (\%) & Top-5 (\%) \\
				\hline\hline
				\multirow{19}{*}{ResNet-18} &FP32 &32/32 &- & 69.6 & 89.2\\
				\cline{2-6}
				&TBN~\cite{wan2018tbn}&1/2&STE & 55.6 & 79.0\\
				&Dorefa-Net~\cite{zhou2016dorefa} &1/1 & STE & 52.5 & 67.7\\
				&Binary-Net~\cite{hubara2016binarized} & 1/1 & STE & 42.2 & 67.1\\
				&XNOR-Net~\cite{rastegari2016xnor} &1/1 &STE & 51.2 & 73.2\\
				& ABCNet~\cite{lin2017towards}& 1/1 & STE & 42.7& 67.6\\
				&Bireal-Net~\cite{liu2018bi} & 1/1 & STE & 56.4 & 79.5\\
				&Bireal-Net~\cite{liu2018bi}+PReLU  & 1/1 & STE & 59.0 & 81.3\\
				&BOP~\cite{helwegen2019latent} & 1/1 & STE & 54.2 & 77.2\\
				&LNS~\cite{han2020training} & 1/1 &STE & 59.4 & 81.7\\
				\cline{2-6}
				& HWGQ~\cite{cai2017deep} & 1/2 & Piece-wise function & 59.6& 82.2\\
				&PCNN ($J=1$)~\cite{gu2019projection} & 1/1 & DBPP & 57.3 & 80.0\\
				&Quantization Networks~\cite{yang2019quantization} & 1/1 & Sigmoid & 53.6 & 75.3\\
				&ResNetE~\cite{bethge2019back} & 1/1 & ApproxSign & 58.1 & 80.6\\
				&BONN~\cite{gu2019bayesian} & 1/1 & Bayesian learning & 59.3 & 81.6\\

				&IR-Net~\cite{qin2020forward} & 1/1 & Tanh-alike &58.1 & 80.0\\
				&RBCN~\cite{liu2021rectified} & 1/1 & Project-based & 59.5 & 81.6\\
				& FDA-BNN & 1/1 & Fourier Series &\textbf{60.2} &\textbf{82.3}\\
				\hline\hline
				\multirow{6}{*}{AlexNet} &FP32 &32/32 &- & 56.6 & 80.2\\
				\cline{2-6}
				& Dorefa-Net~\cite{zhou2016dorefa}  & 1/1 & STE & 43.6 & -\\
				& BinaryNet~\cite{hubara2016binarized} &1/1 & STE & 41.2 & 65.6\\
				& XNOR-Net~\cite{rastegari2016xnor} & 1/1 & STE & 44.2 & 69.2\\
				& LNS~\cite{han2020training} & 1/1 & STE & 44.4 & -\\

				& FDA-BNN & 1/1 & Fourier Series &\textbf{46.2}& \textbf{69.7}\\
				\hline
			\end{tabular}
		}
	\end{table*}

	\subsection{Experiments on ImageNet}
	In this section, we conduct experiments on the large-scale ImageNet ILSVRC 2012~\cite{deng2009imagenet} dataset. ImageNet dataset contains over 1.2M training images with 224$\times$224 resolutions from 1,000 different categories, and 50k validation images. The commonly used data augmentation method is applied during training. The training images are resized, random cropped, random flipped and normalized before training, while the test images are resized, center cropped and normalized.

	In the following, we conduct experiments on widely-used ResNet-18 and AlexNet. We use the Adam optimizer with momentum of 0.9 and set the weight decay to 0. The learning rate starts from 1e-3. The experimental results are shown in Tab.~\ref{tab:imgnet}. Following the acknowledged setting~\cite{hubara2016binarized,zhou2016dorefa}, the first and last layers of the network are not binarized for classification, and all methods shown in the table except for ABCNet~\cite{lin2017towards} and Binary-Net~\cite{huang2018learning} do not quantize the down-sample layers in the ResNet shortcut branch.

	We can see that on ResNet-18, the proposed FDA-BNN achieves a top-1 accuracy of 60.2\% and top-5 accuracy of 82.3\%, which improves the baseline method (Bireal-Net + PReLU activation function) by 1.2\% and 1.0\%, and surpass all other competitors including TBN~\cite{wan2018tbn} and HWGQ~\cite{cai2017deep} which use 2-bit activations for quantization. For AlexNet, we use the quantization method in Dorefa-Net~\cite{zhou2016dorefa} as the baseline method, and achieve a top-1 accuracy of 46.2\% and top-5 accuracy of 69.7\%, which also outperforms other state-of-the-art methods such as LNS~\cite{han2020training} by 1.8\% on top-1 accuracy, which shows the superiority of the proposed FDA-BNN method.

	\section{Conclusion}
	In this paper, we propose a new frequency domain algorithm (\emph{i.e. }, FDA-BNN) to approximate the gradient of the original sign function in the Fourier frequency domain using the combination of a  series of sine functions, which does not affect the low-frequency part of the original sign function that occupies most of the energy. Besides, since only finite FS terms are used in reality, a noise adaptation module is applied to fit the error from the infinite high-order FS terms and the noise occurred between the gradient of sign function and the optimal gradient.  In order to gradually approach the sign function, we increase the number of FS terms $n$ during the training. The experimental results on CIFAR-10 and ImageNet datasets using various network architectures demonstrate the effectiveness of the proposed method. Binary networks trained using our method  achieve the state-of-the-art performance.

	{

		\small

		\bibliographystyle{plain}