\section{Introduction}
Molecular motors are enzymatic protein molecules that use chemical energy released from the hydrolysis of an adenosine triphosphate (ATP) molecule to drive cellular transport along cytoskeletal filaments~\cite{Howard:2001}.
Among the wide variety of molecular motors, linear ATP-driven motors include myosins, kinesins, and dyneins that are responsible for various cellular functions, including intracellular vesicle transport~\cite{Howard:2001, Schliwa:2003n}.
Kinesins and dyneins move along a microtubule, whereas myosins move on an actin filament.
The kinetic and mechanistic processes involved in the movement of the motors are inherently stochastic, which arises due to various interactions among the motors themselves as well as environmental factors such as track length, amount of cellular cargo acting as load, available ATP concentrations, etc.~\cite{Bustamante2001}.
The stochasticity present at the cellular level gives rise to experimental complexity at various levels.
Consequently, incorporating stochastic features into theoretical modeling of molecular motor transport becomes essential to gain adequate statistical understanding of the ensuing fluctuations and their effects.
Specifically, modeling such biological transport phenomena with the help of driven diffusive systems has been fruitful in capturing many essential properties of real systems~\cite{Chou2011, Chowdhury2005, Andreas:2008, Helbing:2001}.
Various versions of the (totally) asymmetric exclusion process or (T)ASEP have been developed involving random walkers with hard-core interactions to mimic the driven non-equilibrium dynamics of kinesin and myosin~\cite{Chou2011}.
In contrast, theoretical studies of dynein motors remain mostly unexplored to date, whence this constitutes the present work's central theme.

The most exciting feature that differentiates dynein motors from kinesin or myosin is their ability to vary their step size from 8\,nm to 32\,nm~\cite{Mallik2004, Bhabha:2016}.
In contrast with the kinesin and myosin families of motor proteins, dyneins are structurally much more complicated.
Detailed experimental studies providing a dynamical understanding of dynein motion are still lacking.
However, some recent experiments have been able to capture the dynamics of dynein motors over a microtubule:
Interestingly, kinesins and dyneins show small but discernible sidewise fluctuations on the microtubule tracks consisting of multiple protofilaments~\cite{Reck2006, Toba2006, Ferro2019, Brunnbauer2012, Bormuth2012}.
Thus, modeling of kinesin dynamics with the ability to switch lanes has been considered~\cite{Wilke2018}, whereas the emergent behavior due to incorporating multi-lane dynamics of dynein motors along with variable steps sizes is as yet unknown.
Most of the theoretical modeling of dynein motors has been restricted to the analysis of single-molecule dynamics~\cite{singh2005monte,rai2013molecular}, or of multi-particle dynamics on a single protofilament track~\cite{Kunwar:2006, Riya:2021}.

In the present work, we investigate the combined effects of a dynein's internal state-dependent hopping as well as hindrance caused by the presence of other motors in its path, accounting for the possibility of bypassing traffic via separate lanes.
We explore the emergent features of the collective dynein dynamics by modeling the motor proteins as hard-core particles in a quasi-two-dimensional topology.
This work thus considerably expands on our earlier studied model~\cite{Riya:2021} of dynein motors on a one-dimensional lattice endowed with varied stepping behavior, to now explcitly include off-axis hops.
We quantify the effects of the off-axis movement by exploring the phase diagram as function of the influx ($\alpha$) and outflux ($\beta$) rates of the molecular motors into and out of the lattice that describes the microtubule lanes.
In the presence of low- and intermediate-load environments where a motor is more likely to take longer jumps (in experiment corresponding to steps of 32\,nm), we observe a first-order transition separating the high- and low-density phases for all values of the flux imbalance parameter $\alpha-\beta$.
This discontinuous phase transition can be readily attributed to the two-dimensional geometry.
Environmental factors such as ATP availability and the crowding environment add stochasticity to the dynein stepping behavior, and its effect on dynein dynamics has been characterized through dwell-time distributions in recent experiments \cite{Reck2006, Toba2006, Ferro2019}.
Our simulation results further confirm these experimental observations for the dynein motor dwell-time distribution, which takes an exponential functional form in less-crowded situations, whereas it turns into a double exponential under overcrowded conditions.

\begin{figure}[t]
 \begin{center}
 \includegraphics[width=\columnwidth]{Figure/Schematic1.pdf}

\end{center}
\caption{A Schematic representation of the dynamics of dynein motors over a tube. We have an open longitudinal boundary through which a motor particle enters from the left edge with the rate $\alpha$, and exits on the right with the rate $\beta$.
	Periodic boundary conditions are employed along the transverse direction.
	Each particle comprises four internal sections representing the primary (red circle) and secondary (green circle) ATP binding sites.
	A motor particle may move in any of the three forward and transverse directions, provided the primary ATP binding site is filled.
	The probability of hopping in the up or down direction is the same, $p_y$, whereas the likelihood of forward motion is set to $(1-2p_y)$.}
\label{Schematics}
\end{figure}

\section{Model description and dynamics}
\label{Model}
Our model is defined on a cylindrical lattice with open boundaries at the longitudinal ends.
We refer to the cylinder is quasi-two-dimensional, as we keep the longitudinal length $L_x$ of the lattice much larger than its transverse length $L_y$, $L_y/L_x \ll 1$.
The quasi-two-dimensional cylinder is structurally similar to the open tube-like configuration of a microtubule, where each row or lane mimics a protofilament of the microtubule.
Each lattice site on the cylinder can be occupied by at most one indistinguishable hardcore particle.
A motor particle can only enter from the first lattice site of any lane with a constant rate $\alpha$, and exit from the last longitudinal site of any lane with the rate $\beta$.
In the bulk, a particle may move in one of the three following directions: forward along the length of the lattice, upward or downward in the transverse directions.
Guided by recent experimental results that demonstrate the transverse off-axis motion of the dynein motors to be significantly slower than their forward movement~\cite{Reck2006, Toba2006, Ferro2019}, we choose the probability of moving in the transverse direction $p_y$ to be significantly smaller than the chance of forward movement $p_x$: $p_x = 1 - 2 p_y = 0.95$, $p_y = 0.025$.
Further, depending on the load (vacuoles or ATPs) attached to the motor particles, they can advance between one and up to four lattice sites along any of the three directions, provided that the destination sites are empty.

We are interested in studying the collective kinetics of dynein motors, well-known for their dynamical jump length, which ranges from 8 to 32\,nm depending on various factors such as the available ATP concentration, amount of load, etc.~\cite{Bhabha:2016, Mallik2004}.
We ignore the structural complexity of the dynein molecule and consider the dimeric dynein motor as a hardcore particle with four ATP binding sites.
One primary ATP binding subsite is responsible for hydrolysis, while the remaining three secondary binding subsites carry the load.
The following three distinct internal processes determine the probability of performing a number $n_s$ of steps:
\begin{itemize}
  \item[(i)]\textbf{Attachment:} One unit of ATP attaches to the empty primary binding site with a fixed probability $P_{att}$, or to any of the available secondary sites, with an also constant probability, $S_{att}$.
  \item[(ii)]\textbf{Detachment:} One unit of ATP detaches either from the primary site with the fixed rate $P_{det}$, or from one of the three secondary sites with the constant probability $S_{det}$.
  \item[(iii)]\textbf{ATP hydrolysis:} An occupied primary site hydrolyses it ATP to ADP to generate mechanical energy that enables it to perform movement.
  Thus, if the primary binding site is occupied, the motor attempts to hop $(4-s)$ steps, where $s$ is the number of secondary sites holding ATP (thus, $s$ can be either $1$, $2$, or $3$).
\end{itemize}
To incorporate excluded-volume interactions, we consider the sliding behavior of a motor particle to an empty site which prohibits the particles from overtaking in the same lane.
However, they can bypass other particles present in different lanes.
A schematic of the model is illustrated in Fig.~\ref{Schematics}.
The rate of hopping different step sizes depends on the attachment and detachment probabilities.
Specifically, the rate $R_i$ to jump $i$ steps (given there are no obstructions due to the presence of other particles) is given by
\begin{align}
 R_4 &=P_{att}(1-P_{det})\sum_i(S_{att}S_{det})^i(1-S_{att})^{3-i}, \\
 R_3 &=P_{at}(1-P_{det})S_{att}(1-S_{det})[(S_{att} S_{det})^2 + (1 - S_{att})^2 + (1 - S_{att}) S_{att} S_{det})], \\
 R_2 &=P_{att}(1-P_{det}/4)S_{att}^2(1-S_{det})^2[(S_{att} S_{det})+ (1 - S_{att})], \\
 R_1 &=P_{att} (1 - P_{det}) S_{att}^3 (1 - S_{det})^3.
 \label{rates}
 \end{align}
Further, due to excluded-volume constraints, the rates become modified by the presence of other particles on the path.
Within a standard mean-field approximation, one obtains
\begin{equation}
 v_4 = R_4, \hspace{0.5cm} v_3 = R_4 \rho + R_3, \hspace{0.5cm} v_2 = (R_4 + R_3) \rho + R_2, \hspace{0.5cm} v_1 = (R_4 + R_3 + R_2) \rho + R_1 .
 \label{vrates}
\end{equation}
Here, $\rho$ denotes the mean particle density in the bulk.
Before starting the discussion of the results for the dynamics of this dynein transport model, we provide a brief review of the well-known stationary-state results of the one-dimensional TASEP model with open boundaries~\cite{Derrida1992} in the following subsection.
\subsection{Phase diagram of the one-dimensional TASEP with hop rate r}
In the case of a one-dimensional TASEP, where a hardcore particle can move to its nearest neighboring site with a constant hop rate $r$, the steady-state particle current for periodic boundary conditions is given as $J = r \rho (1-\rho)$ at all lattice sites.
However, for open boundary conditions, depending on the influx ($\alpha$) and outflux ($\beta$) rates from the first and last sites of the lattice, the TASEP stationary states can be classified into three different phases: low-density (LD), high-density (HD), and maximal-current (MC).
From the asymptotic analysis performed in earlier studies~(please see, e.g., Ref.~\cite{Derrida1992} for details), the characteristic features of the three phases are as follows~\cite{schutz2000exactly, Derrida1992}:
\begin{itemize}
  \item [(a)] \textbf{Low-density phase (LD)}: In this phase, the bulk stationary density saturates to a constant value $\rho_{LD}=\alpha/r$, which yields the stationary current $J_{LD} = \alpha(1-\rho_{LD}) = \alpha (1-\alpha/r)$.
  Near the exit boundary, the density corresponds to $\rho_{L_x} = J_{LD}/\beta$.
  \item[(b)]\textbf{High-density phase (HD)}: Here, the bulk stationary density is $\rho_{HD} = 1-\beta$, thus the steady-state particle current is given by $J_{HD} = r\beta(1-\beta)$.
  \item[(c)]\textbf{Maximal-current phase (MC)}: The maximal particle current is obtained when the density in the bulk is $1/2$ and hence $J = J_{max} = r/4$ for $\alpha > r/2$ and $\beta > 1/2$.
  Near the boundaries, the density deviation from its bulk value shows power law decay, which indicates the presence of long-range correlations in the system.
\end{itemize}
The transition across the LD and HD phases is first-order (discontinuous); hence a coexistence phase is observed along the phase transition line.
Equating the LD and HD particle fluxes, one finds the coexistence line located at $\beta = \alpha/r$, as is confirmed by numerical simulations.
Furthermore, these results hold even for the two-dimensional TASEP with periodic boundary conditions in the transverse direction.

\section{Results}
Compared to the one-dimensional TASEP, the dynamics of the dynein particles in both the one-dimensional and quasi-two-dimensional settings are considerably more complicated, hence it is non-trivial to extract any precise analytical understanding.
Consequently, we employ extensive numerical simulations to explore the dynamics under various contrived environmental conditions.
The results shown in the following sections pertain to lattices with dimensions $L_x = 1000$ in the one-dimensional case, and $L_x = 1000$, $L_y = 4$ (i.e., four adjacent microtubule lanes) for the quasi-two-dimensional topology.
The ATP attachment and detachment rates for the primary site are chosen as $P_{att} = 0.8$ and $P_{det} = 0.2$, respectively.
We worked with three different load conditions for each motor as determined by the load attachment rate $S_{att}$ and detachment rate $S_{det} = 1-S_{att}$ at the secondary sites.
We selected the following set of binding rates for each secondary site:
\[
  S_{att}=
\begin{cases}
  0.8, & \text{high-load conditions}, \\
  0.5, & \text{intermediate-load conditions}, \\
  0.2, & \text{low-load conditions}.
\end{cases}
\]

\begin{figure}[ht]
\begin{subfigure}{0.5\columnwidth}
  \begin{center}
 \includegraphics[width=\columnwidth]{Figure/HL_PC_Lane.pdf}
\end{center}
  \caption{High-load condition ($P_{att}=0.8,~S_{att}=0.8$).}
  \label{HLL1}
\end{subfigure}
\begin{subfigure}{.5\columnwidth}
  \begin{center}
  \includegraphics[width=\columnwidth]{Figure/LL_PC_Lane1.pdf}
\end{center}
\caption{Low-load condition ($P_{att}=0.8,~S_{att}=0.2$).}
  \label{LLL1}
\end{subfigure}
\caption{Phase diagram in the influx-outflux rate space for the dynamics of dynein particles in one dimension.
The phase diagram under high-load conditions closely approximates the standard TASEP phase diagram.
However, the maximal current region is reduced to a small portion of the phase space under the low-load condition.
The insets display the density profile and jump length statistics in the respective phases.
In the MC phase, a reduction in the stationary density is observed upon increasing the probability of long jumps; thus, $\rho_{MC}\approx 0.45$ for high-load and $\rho_{MC}\approx 0.34$ for low-load conditions as indicated by the solid green lines in the insets of Figs.~\ref{HLL1} and \ref{LLL1}, respectively.}
\label{PC_lane1}
\end{figure}
\subsection[]{Phase diagram for dynein particles in one dimension}
We aim to evaluate the stationary state phase diagram for the dynein particle exclusion process in one dimension.
To that end, we first write down the mean-field bulk current that has been formulated in our previous work, with periodic boundary conditions~\cite{Riya:2021}:
\begin{equation}
  J = \sum_{i=1}^{4} i ~v_i~ \rho(1-\rho)^i,
\label{current}
\end{equation}
where $v_i$ denotes the probability of taking $i$ steps, and explicitly depends on the attachment and detachment of ATPs to their primary and secondary binding sites, as shown in eqs.~(\ref{vrates}).
Then the bulk density $\rho_{max}$ in the MC phase is obtained from the maximum of the mean-field current given by eq.~(\ref{current}), for different load conditions.
Straightforward algebraic calculations further determine the exact values (within this mean-field approximatioN) for the rates $\alpha$ and $\beta$ beyond which the stationary-state dynamics resides in the maximal-current (MC) phase.
The resulting transition point for the MC phase is
\begin{center}
 $\alpha > \sum_{i=1}^{4} i ~v_i~(1-\rho_{max})^i,~~~~ \beta > 1-\rho_{max}$.
\end{center}

This analysis of the MC transition point is in excellent agreement with our simulation results, as shown in Fig.~\ref{PC_lane1}, both for high- and low-load conditions.
Our findings from the Monte Carlo simulations and data analysis demonstrate that for the one-dimensional dynein particle dynamics, one observes a nonlinear first-order transition line (coexistence line) between the LD and HD phases, in contrast with the linear coexistence line for the one-dimensional simple TASEP.
The bulk density in the LD phase is proportional to the injection rate $\alpha$, and shows intriguing periodic oscillations at the boundaries~\cite{Riya:2021}.
In the HD phase, the bulk density is precisely $1-\beta$, as in the one-dimensional TASEP.
The density profiles in all three distinct nonequilibrium phases are shown in the insets of Fig.~\ref{PC_lane1}.
These insets also display the jump length statistics in the different phases for high- and low-load conditions.
It is apparent in Fig.~\ref{HLL1} that the statistics of taking single steps under high-load conditions is significantly high, so that the results for the stationary-state bulk density and current become similar to the TASEP.
It is important to note that the increased probability for long jumps reduces the phase space area of the maximal-current phase.
Long jumps break particle-hole symmetry and promote the motor particles' nonuniform distribution over the lattice.

\subsection[]{Phase diagram for dynein particles in two dimensions}
\begin{figure}[t]
\begin{subfigure}{.33\columnwidth}
  \begin{center}
 \includegraphics[width=\columnwidth]{Figure/HL_density_step_4.pdf}

\end{center}
  \caption{High load}
  \label{DP_HL}
\end{subfigure}
\begin{subfigure}{.33\columnwidth}
  \begin{center}
 \includegraphics[width=\columnwidth]{Figure/ML_density_step_4.pdf}

\end{center}
 \caption{Intermediate load}
  \label{DP_ML}
\end{subfigure}
\begin{subfigure}{.33\columnwidth}
  \begin{center}
 \includegraphics[width=\columnwidth]{Figure/LL_density_step_4.pdf}

\end{center}
 \caption{Low load}
  \label{DP_LL}
\end{subfigure}
\begin{subfigure}{.33\columnwidth}
  \begin{center}
 \includegraphics[width=\columnwidth]{Figure/HL_Histogram_step_4.pdf}

\end{center}
  \caption{High load}
  \label{step_HL}
\end{subfigure}
\begin{subfigure}{.33\columnwidth}
  \begin{center}
 \includegraphics[width=\columnwidth]{Figure/ML_Histogram_step_4.pdf}

\end{center}
\caption{Intermediate load}
  \label{step_ML}
\end{subfigure}
\begin{subfigure}{.33\columnwidth}
  \begin{center}
 \includegraphics[width=\columnwidth]{Figure/LL_Histogram_step_4.pdf}

\end{center}
\caption{Low load}
  \label{step_LL}
\end{subfigure}
\caption{The upper panels display the density profile in different phases for the three distinct load conditions.
The lower panels provide the statistics of the number of steps taken by the dynein particles.
Under high-load conditions, the count of smaller steps is significantly larger (Fig.~\ref{step_HL}), and the density profile is similar to that of the simple TASEP, as shown in Fig.~\ref{DP_HL}.
For both medium- and low-load conditions, the bulk density in the high-density phase is $(1-\beta)$, whereas it is proportional to the injection rate $\alpha$ in the low-density phase.}
\label{figLa4}
\end{figure}
\begin{figure}[!ht]
\begin{subfigure}{0.5\columnwidth}
  \begin{center}
 \includegraphics[width=\columnwidth]{Figure/PC_Lane_4.pdf}

\end{center}
  \caption{Phase diagram}
  \label{PCL4S4}
\end{subfigure}
\begin{subfigure}{0.5\columnwidth}
  \begin{center}
 \includegraphics[width=\columnwidth]{Figure/Density_currentLane4.pdf}

\end{center}
\caption{Steady-state properties}
  \label{JDL4}
\end{subfigure}
\caption{The graphs depict the steady-state properties obtained from the dynamics of the dynein particles in the quasi-two-dimensional setting.
The main Fig.~\ref{PCL4S4} shows the phase diagram with a first-order transition separating the low- and high-density phases under low- (magenta dot-dashed line) and intermediate-load (green dashed line) conditions.
For comparison, the inset of Fig.~\ref{PCL4S4} displays the phase diagram under high-load condition (dotted black line), where all three phases LD, HD, and MC are present, similar to the simple TASEP (solid gray line).
The main Fig.~\ref{JDL4} plots the current profiles as a function of the influx rate $\alpha$, for three different outflux rates $\beta = 0.2,0.4,0.6$, in the low load conditions and the inset shows the corresponding density profiles.}
\label{SSL4}
\end{figure}
This section discusses our simulation results for the quasi-two-dimensional setting on a rectangular lattice of size $10^3\times 4$ and compares them with the one-dimensional case (where also $L_x=10^3$).
We start with analyzing the jump statistics and the dynein particle current along the longitudinal direction.
The stationary-state current in the transverse direction always vanishes for an infinitely large system owing to the periodic boundary conditions and our choice of equal hopping probabilities for up- and down-moves.

Under high-load conditions, the jump size statistics show that the number of nearest-neighbor hops is more significant than the number of longer-distance jumps, as seen in Fig.~\ref{step_HL}.
One would thus expect the ensuing dynamical steady-state phase diagram to resemble that of the simple TASEP.
Indeed, akin to the TASEP, we observe the presence of all three phases LD, HD, and MC in the $\alpha-\beta$ parameter space as shown in the inset of Fig.~\ref{PCL4S4}, but with a shift in the transition line.
Owing to the small but nonzero number of long jumps performed by the dynein particles, the MC critical point becomes shifted toward a higher value of the influx rate $\alpha$.
In the inset of Fig.~\ref{PCL4S4}, the solid gray line represents the TASEP transition lines, while the broken black lines indicate the phase boundaries obtained for our quasi-two-dimensional dynein model under high loads.
In addition, we recall that the MC phase features long-range correlations.
Hence, the density profile in the MC phase is in fact independent of the rates $\alpha,\beta$ and displays power law behavior near both boundaries with the same standard TASEP decay exponent of 1/2.
Moreover, the density profile in the LD phase is proportional to the rate $\alpha$ and shows damped oscillatory behavior close to the entrance boundary.
These spatial oscillations have a periodicity of four sites, a qualitatively quite similar feature as for the one-dimensional dynein particle model discussed in our previous work~\cite{Riya:2021}.
Further, the bulk density in the HD phase is always approximately near $(1-\beta)$ as demonstrated in Fig.~\ref{DP_HL}.

Interestingly, under both low- and intermediate-load conditions, we only observe two stationary-state phases with a first-order transition line separating them.
The maximal-current phase is absent, as shown in the main Fig.~\ref{PCL4S4}.
Under these conditions, the jump length histogram implies that the probability of taking longer jumps is more significant in the LD phase, whereas shorter jumps dominate in the HD phase.
However, the magnitude of this difference is not as substantial as for high loads, as can be seen in Figs.~\ref{step_ML} and \ref{step_LL}.
The farther jumps and the small side-wise fluctuations in the transverse direction in this two-dimensional setting allow a dynein motor to bypass other particles via different lanes.
Hence the exclusion interactions do not facilitate the system to build long-range particle correlations.
A similar absence of the maximal-current phase has also been observed in a TASEP model with hierarchical long-range connections defined on a network~\cite{Otwinowski:2009}.

The density in the LD phase is again proportional to the influx rate $\alpha$, and the bulk density in the HD phase is approximately close to $(1-\beta)$, as demonstrated in Figs.~\ref{DP_ML} and \ref{DP_LL}.
The results depicted in Fig.~\ref{JDL4} support the discontinuous (first-order) character of the transition between the LD and HD phases for the entire parameter space.
We observe a continuous change in the stationary motor particle current with the influx and outflux rates as evidenced in the main Fig.~\ref{JDL4}, which however causes an abrupt change in the density profile, noticeable in the inset of Fig~\ref{JDL4}).
Before exploring the collective dynamical behavior of the dynein particles in the quasi-two-dimensional lattice geometry, we would like to emphasize that our simulations indicate that an increase in the number of lanes does in fact not lead to any qualitative change in the results.

\begin{figure}[!ht]
\begin{center}
 \includegraphics[width=0.8\columnwidth]{Figure/Trajectories_HL.pdf}

\end{center}
\caption{Single tagged dynein particles' trajectories under high-load conditions.
Particles A, B and C enter at $t=0$, $t\approx L_x$ and $t\approx L_x^2$, respectively.
The influx and outflux rates used here are $\alpha=0.7,~\beta=0.7$, corresponding to the low-density phase.}
\label{SD_HL}
\end{figure}
\subsection{Dynamics of dynein particles in two dimensions}
To understand the change in the dynamics caused by the quasi-two-dimensional geometry, we first explore the single-run statistics under various load conditions in the distinct phases.
Specifically, in Fig.~\ref{SD_HL} we show the dynamics of three different tagged dynein motor particles:
A particle is tagged at different entry times $t_{en}$, which effectively corresponds to different crowding environments until the system reaches its stationary state.
"Particle A" denotes the first particle that enters the lattice.
"Particle B" and "Particle C" enter the system at times proportional to $L_x$ and $L_x^2$.
Examples for single-run trajectories of particles tagged in this manner are displayed in Fig.~\ref{SD_HL}.
Irrespective of $\alpha$ and $\beta$, particle A is more likely to move forward while remaining in the same lane than particles B and C.
It is less probable for particle A to encounter obstacles (other motor particles), and hence sidewise fluctuations are less prominent in its trajectory than for the later-arriving particles.
However, as the overall particle density increases with time, the kinetics becomes increasingly obstructed, and individual particles are forced to switch to other lanes, as is apparent in the more helical structures visible in the trajectories of particles B and C in Fig.~\ref{SD_HL}.
We also observe that the later particles take longer to exit from the last lattice sites due to crowding.
Following these observations, we analyze the trajectories' mean-square displacement (MSD) to better understand the slowing-down of the motor particles transport, and to provide additional information on the changing time scales under different conditions.

We have thus numerically evaluated the mean-square displacement (MSD) of a tagged particle.
The MSD in the transverse direction is always diffusive, owing to the symmetry in the probabilities of up- and downward movement.
However, the dynamics along the longitudinal direction clearly shows modifications in the relevant characteristic time scales.
Hence we present only data for the MSD in the longitudinal (drive) direction.
It is well-known that wide variations of the TASEP exhibit dynamics that follow the scaling exponents of the Burgers/KPZ universality class in one dimension, and correspondingly the MSD grows with time as $t^{2/3}$~\cite{Gupta:2007, Priyanka:2016b}.
In the two-dimensional TASEP, the MSD growth with time becomes linear with logarithmic corrections~\cite{Daquilla:2011}.
For the dynein particles in our model, the simulation data yield that the MSD obeys a power-law growth with logarithmic corrections in the form $t^{\xi}/\ln t$, where $\xi$ takes values in the range between $3/2$ to $1$ depending on the crowding conditions over the lattice.
Irrespective of the load conditions, the dynamics for a single dynein particle is superdiffusive, and the MSD along the drive direction grows alebraically with time with an exponent $3/2$.
However, in a highly crowded environment, the particle kinetics slows down, and $\xi$ approximately takes a value close to unity as shown in Fig.~\ref{MSD_tag}.
The MSD dynamics turns out to be essentially independent of the details of the boundary injection and exit rates.
\begin{figure}[t]
 \begin{subfigure}[t]{.5\columnwidth}
 \begin{center}
 \includegraphics[width=\columnwidth]{Figure/MSD_rp8p8.pdf}

\end{center}
\caption{High load conditions}
\label{MSD_tagHL}
\end{subfigure}
\begin{subfigure}[t]{.5\columnwidth}
 \begin{center}
 \includegraphics[width=\columnwidth]{Figure/MSD_rp8p2.pdf}

\end{center}
\caption{Low load conditions}
\label{MSD_tagLL}
\end{subfigure}
\caption{The time-dependent growth of the mean-square displacement (MSD) for the tagged particles' trajectories at three different entrance times on a finite quasi-two-dimensional lattice.
The different time regimes are also marked; here, $L = 500$, $P_{att}=0.8$, and $S_{att}=0.8$ representing a high-load, and $S_{att}=0.2$ for a low-load condition.
The results were averaged over $10^3$ different realizations.}
\label{MSD_tag}
\end{figure}

The change in the MSD growth power laws is solely due to the difference in the passage times that particles take to move from one site to another in the presence of other particles.
Thus, measuring the particle dwell times adds further information about this mutually constrained particle dynamics.
The dwell-time distribution is in fact an experimentally relevant quantity and has been successfully measured in a number of experiments~\cite{Liao:2007, Reck:2006}.
In our simulations, we define the dwell or waiting time as the temporal duration, in units of Monte Carlo steps (MCS), that a particle waits between two consecutive jumps.
We have numerically obtained the dwell-time statistics under different crowding conditions, as plotted in Fig~\ref{DT_LL}.
For particle A, which invariably encounters a less-crowded environment, the waiting-time distribution for a tagged particle decays exponentially, irrespective of the influx and outflux rates $\alpha$ and $\beta$, as seen in Fig.~\ref{DT_LL}.
But with increasing particle density across the lattice, we observe longer waiting times and slower decay of the associated dwell-time distribution with longer tails of the exponential function.
The dwell-time distribution in the overcrowded environment displays a ``double-exponential'' form with an initial slower decay rate for shorter waiting times, which subsequently crosses over to a faster decay for longer waiting times, as seen in the main Fig.~\ref{DT_LL}, which corresponds to an average particle density $\approx 0.85$.
In addition, we observe intriguing oscillatory features for particle-type B (with $t_{en}\propto L_x$).
These oscillations are prominently visible only if the average density is greater than half-filled, and represent a dynamical phase with many alternating high-density peaks and low-density valleys, akin to coexistence phases.
\begin{figure}[!ht]
\begin{center}
 \includegraphics[width=0.75\columnwidth]{Figure/DT_LL_lane4.pdf}

\end{center}
\caption{Dwell-time distribution for tagged dynein particles at different entry times in the quasi-two-dimensional system.
The main figure corresponds to the HD phase, while the inset displays data for the LD phase.
The waiting times for the low-density phase are always governed by a single exponential distribution with a density-dependent decay rate, as shown in the inset.
In contrast, the main graph displays prominent double-exponential behavior characteristic of an extremely congested system.
The dashed black line shows the best fit to a double exponential, $18(2e^{-0.09x}-e^{-0.32x})$.}
 \label{DT_LL}
\end{figure}

\section{Conclusions}
In this paper, we describe a variation of the asymmetric exclusion process with short- and long-distance jumps to model the dynamics of dynein motors, and explored the ensuing collective dynamics in one dimension as well as in a quasi-two-dimensional topology with open boundary conditions.
Our work highlights the non-trivial effects of geometry in out-of-equilibrium driven diffusive systems that may modify the phase diagram and change the nature of the accompanying phase transitions.
Significant changes observed in the phase diagram can be attributed to the breakdown of particle-hole symmetry for the dynein particle dynamics.
In one dimension, we found a reduced maximal-current (MC) phase region in $\alpha-\beta$ parameters space due to the presence of long jumps.
Using the mean-field expression for the resulting current, we present a precise lower bound for the influx and outflux rates for the maximal-current phase.
We extended our study to the quasi-two-dimensional case with negligible small transverse hop probabilities.
In this setting, the phase diagram shows drastic changes; we observe the disappearance of the maximal-current phase for low- to medium-load conditions caused by the mutual bypassing of the motor particles via different lanes.
We hope these intriguing results will motivate further studies into the nontrivial effects of higher dimensions in variants of the TASEP model and serve as a reference for a wider range of transport processes that involve particles bypassing each other, such as vehicular traffic models and models for certain regulatory genes.

Further, we have also examined the dynamics of the dynein particles under different environmental conditions and measured their MSD and dwell-time distributions.
Due to the two-dimensional topology, a dynein particle shows logarithmic corrections in the superdiffusive time-dependent growth of the MSD.
Our study also demonstrates the slowing-down of the kinetics due to crowded conditions, leading to dynamics closer to diffusive behavior.
Additionally, we observe consistent features in the associated dwell-time distributions.
Dwell-time distributions are directly experimentally measurable and can be easily verified for different control parameters.
We confirmed the effects of crowding and variable-stepping behavior of dyneins on their dwell-time distributions:
We found that under less-crowded conditions, the dwell times are distributed exponentially; whereas in a more crowded environment, the waiting time distribution changes to a double exponential with two characteristic time scales.
Both exponential and double exponential distributions of the dwell times have been observed in real experiments under conditions when the dynein takes longer and shorter jumps, respectively, \cite{Liao:2007,Reck:2006}, c.f. specifically Fig.~5 in Refs.~\cite{Reck:2006}.

\noindent{Authorcontributions:} Priyanka has conceived and designed the problem.
The numerical simulations and analyses were performed equally by Riya Nandi and Priyanka.
All authors have equally contributed to the interpretation of the data and results, and to the preparation of the manuscript.

\noindent{Funding} This research was sponsored by the Army Research Office and was accomplished under Grant No. W911NF17-1-0156.
The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government.
The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
\

\section*{References}

\providecommand{\newblock}{}


\section{Introduction} \label{sec:Intorduction}

Defocus deblurring aims to restore an all-in-focus image from a defocused image, and is highly demanded by daily photographers to remove unwanted blur.
Moreover, restored all-in-focus images can greatly facilitate high-level vision tasks such as semantic segmentation \cite{Noh:2015:ICCV, Wang:2020:DSRSS} and object detection \cite{Dai:2016:RFCN, Cao:2020:HQOD}.
Despite the usefulness, defocus deblurring remains a challenging problem as defocus blur is spatially varying in size, and its shape also varies across the image.

A conventional strategy \cite{Shi:2015:JNB, Andres:2016:RTF, Park:2017:unified, Cho:2017:Convergence, Karaali:2018:DMEAdaptive, Lee:2019:DMENet} is to model defocus blur as a combination of different convolution results obtained by applying predefined kernels to a sharp image.
It estimates per-pixel blur kernels based on the blur model and then performs non-blind deconvolution \cite{Fish:95:BD, Levin:2007:Coded, Krishnan:2008:deconvolution}.
However, this approach often fails due to the restrictive blur model, which disregards the nonlinearity of a real-world blur and constrains defocus blur in specific shapes such as disc \cite{Andres:2016:RTF} or Gaussian kernels \cite{Shi:2015:JNB, Park:2017:unified, Karaali:2018:DMEAdaptive, Lee:2019:DMENet}.

Recently, Abuolaim and Brown \cite{Abuolaim:2020:DPDNet} proposed the first end-to-end learning-based method, DPDNet, which does not rely on a specific blur model, but directly restores a sharp image.
Thanks to the end-to-end learning, DPDNet outperforms previous approaches for real-world defocused images.
They show that dual-pixel data obtainable from some modern cameras can significantly boost the deblurring performance, and present a dual-pixel defocus deblurring (DPDD) dataset.
However, ringing artifacts and remaining blur can often be found in their results, mainly due to its na\"ive UNet \cite{Ronneberger:2015:UNet} architecture, which is not flexible enough to deal with spatially-varying and large blur \cite{Zhou:2019:STFAN}.

In this paper, we propose an end-to-end network embedded with our novel \textit{Iterative Filter Adaptive Network (IFAN)} for single image defocus deblurring.
IFAN is specifically designed for the effective handling of spatially-varying and large defocus blur.
To handle the spatially-varying nature of defocus blur,
IFAN adopts an adaptive filter prediction scheme motivated by recent filter adaptive networks (FANs) \cite{zhang:2018:SVRNN,Zhou:2019:STFAN}. %
Specifically, IFAN does not directly predict pixel values, but generates spatially-adaptive per-pixel deblurring filters, which are then applied to features from an input defocus blurred image to generate deblurred features.

To efficiently handle large defocus blur that requires large receptive fields,
IFAN predicts stacks of small-sized separable filters instead of conventional filters unlike previous FANs.
To apply predicted separable filters to features, we also propose a novel \emph{Iterative Adaptive Convolution (IAC)} layer that iteratively applies separable filters to features.
As a result, IFAN significantly improves the deblurring quality at a low computational cost in the presence of spatially-varying and large defocus blur.

To further improve the single image deblurring quality, we train our network with novel defocus-specific tasks: defocus disparity estimation and reblurring.
The learning of defocus disparity estimation exploits dual-pixel data,
which provides stereo images with a tiny baseline, whose disparities are proportional to defocus blur magnitudes \cite{Garg:2019:DP, punnappurath:2020:modeling, Abuolaim:2020:DPDNet}.
Leveraging dual-pixel stereo images, we train IFAN to predict the disparity map from a single image so that it can also learn to predict blur magnitudes more accurately.

On the other hand, the learning of reblurring task, which is motivated by the reblur-to-deblur scheme in \cite{Chen:2018:R2D}, utilizes deblurring filters predicted by IFAN for reblurring all-in-focus images.
For accurate reblurring, IFAN needs to predict deblurring filters that contain accurate information about the shapes and sizes of defocus blur.
During training, we introduce an additional network that inverts predicted deblurring filters to reblurring filters and reblurs the ground-truth all-in-focus image.
We then train IFAN to minimize the difference between the defocused input image and the corresponding reblurred image.
We experimentally show that both tasks significantly boost the deblurring quality.

To verify the effectiveness of our method on diverse real-world images from different cameras, we extensively evaluate the method on several real-world datasets such as the DPDD dataset \cite{Abuolaim:2020:DPDNet}, Pixel dual-pixel test set \cite{Abuolaim:2020:DPDNet}, and CUHK blur detection dataset \cite{Shi:2014:CUHK}.
In addition, for quantitative evaluation, we present the Real Depth of Field (RealDOF) test set that provides real-world defocused images and their ground-truth all-in-focus images.

To summarize, our contributions include:
\vspace{-5pt}
\begin{itemize}
    \setlength\itemsep{0.01em}
    \item \textit{Iterative Filter Adaptive Network (IFAN)} that effectively handles spatially-varying and large defocus blur,
    \item a novel training scheme that utilizes the learning of defocus disparity estimation and reblurring, and
    \item state-of-the-art performance of defocus deblurring in terms of deblurring accuracy and computational cost. %
\end{itemize}

\section{Related Work}
\paragraph{Defocus deblurring}
Most defocus deblurring approaches,
including classical and recent deep-learning-based ones,
assume a specific blur model and try to estimate a defocus map based on the model.
They estimate the defocus map of an image leveraging various cues, such as
hand-crafted features \cite{Shi:2015:JNB, Krishnan:2008:deconvolution, Cho:2017:Convergence, Karaali:2018:DMEAdaptive}, learned regression features \cite{Andres:2016:RTF}, and deep features \cite{Park:2017:unified,Lee:2019:DMENet}.
They then utilize non-blind deconvolution methods such as \cite{Fish:95:BD, Levin:2007:Coded, Krishnan:2008:deconvolution} to restore a sharp image.
However, their performance is limited due to their restrictive blur models.
Recently, Abuolaim and Brown \cite{Abuolaim:2020:DPDNet} proposed an end-to-end learning-based approach that outperforms previous blur model-based ones.
However, their results often have ringing artifacts and remaining blur as mentioned in \Sec{\ref{sec:Intorduction}}.

\vspace{-13pt}
\paragraph{Filter adaptive networks}
FANs have been proposed to facilitate the spatially-adaptive handling of features in various tasks \cite{Jia:2016:DFN, Niklaus:2017:VFI, Niklaus:2017:VFISPC, Mildenhall:2018:BDKPN, Jo:2018:DUFSR, Wang:2018:RRTSR, Zhou:2019:STFAN, zhang:2018:SVRNN, Su:2019:PAC, Xu:2020:squeezesegv3}.
FANs commonly consist of two components: prediction of spatially-adaptive filters and transformation of features using the predicted filters, where the latter component is called filter adaptive convolution (FAC).
Various FANs have been proposed and applied to different tasks, such as frame interpolation \cite{Jia:2016:DFN, Niklaus:2017:VFI, Niklaus:2017:VFISPC}, denoising \cite{Mildenhall:2018:BDKPN}, super-resolution \cite{Jo:2018:DUFSR, Wang:2018:RRTSR}, semantic segmentation \cite{Su:2019:PAC}, and point cloud segmentation \cite{Xu:2020:squeezesegv3}.
For the deblurring task, Zhang \etal \cite{zhang:2018:SVRNN} proposed pixel-recurrent adaptive convolution for motion deblurring.
However, their method requires massive computational cost as the recurrent neural network must run for each pixel.
Zhou \etal \cite{Zhou:2019:STFAN} proposed a novel filter adaptive convolution layer for frame alignment and video deblurring.
However, handling large motion blur requires predicting large filters, which results in huge computational cost.

\section{Single Image Defocus Deblurring}
In this section, we first introduce the Iterative Adaptive Convolution (IAC) layer, which forms the basis of our IFAN (\Sec{\ref{sec:IAC}}).
Then, we present our deblurring network based on IFAN with detailed explanations on each component (\Sec{\ref{sec:Deblurring_Newtwork}}).
Finally, we explain our training strategy exploiting the disparity estimation and reblurring tasks (\Sec{\ref{sec:training}}).

\subsection{Iterative Adaptive Convolution (IAC)}\label{sec:IAC}

Our IAC is inspired by the filter adaptive convolution (FAC) \cite{Zhou:2019:STFAN} that forms the basis of previous FANs.
For a better understanding of IAC, we first briefly review FAC.

\Fig{\ref{fig:FAC}} shows an overview of a FAC layer.
A FAC layer takes a feature map $e$ and a map of spatially-varying convolution filters $\textbf{F}$ as input.
$e$ and $\textbf{F}$ have the same spatial size $h\!\times\!w$.
The input feature map $e$ has $c$ channels.
At each spatial location in $\textbf{F}$ is a $ck^2$-dim vector representing $c$ convolution filters of size $k\!\times\!k$.
For each spatial location $(x, y)$, the FAC layer generates $c$ convolution filters from $\textbf{F}$ by reshaping the vector at $(x, y)$.
Then, the layer applies the filters to the features in $e$ centered at $(x,y)$ in the channel-wise manner to generate an output feature map $\hat{e} \in \mathbb{R}^{h\times w \times c}$.

For effectively handling spatially-varying and large defocus blur, it is critical to secure large receptive fields.
However, while FAC facilitates spatially-adaptive processing of features, increasing the filter size to cover wider receptive fields results in huge memory consumption and computational cost.
To resolve this limitation, we propose the IAC layer that iteratively applies small-sized separable filters to efficiently enlarge the receptive fields with little computational overhead.

\Fig{\ref{fig:IAC}} shows an overview of our IAC layer.
Similarly to FAC, IAC takes a feature map $e$ and a map of spatially-varying filters $\textbf{F}$ as input, whose spatial sizes are the same.
At each spatial location in $\textbf{F}$ is a $Nc(2k\,\texttt{+}\,1)$-dim vector representing $N$ sets of filters $\{\textbf{F}^1,\textbf{F}^2,\cdots,\textbf{F}^N\}$.
The $n$-th filter set $\textbf{F}^n$ has two 1-dim filters $\textbf{f}_1^{\,n}$ and $\textbf{f}_2^{\,n}$ of the sizes $k\!\times\!1$ and $1\!\times\!k$, respectively, and one bias vector $\textbf{b}^n\!$.
$\ \textbf{f}_1^{\,n}$, $\textbf{f}_2^{\,n}$, and $\textbf{b}^n$ have $c$ channels.
The IAC layer decomposes
the vector
in $\textbf{F}$ at each location into filters and bias vectors, and iteratively applies them to $e$ in the channel-wise manner to produce an output feature map $\hat{e}$.

Let $\hat{e}^n$ denote the $n$-th intermediate feature map after applying the $n$-th separable filters and bias, where $\hat{e}^0\,\texttt{=}\,\,e$ and $\hat{e}^N\texttt{=}\,\,\hat{e}$.
Then, the IAC layer computes $\hat{e}^n$ for $n\in\{1,\cdots,N\}$ as follows:
\begin{equation}
    \hat{e}^n = \textrm{LReLU}(\hat{e}^{(n-1)}\,*\, \textbf{f}_1^{\,n}\,*\,\textbf{f}_2^{\,n} + \textbf{b}^n),
\end{equation}
where $\textrm{LReLU}$ is the leaky rectified linear unit \cite{Maas:2013:Leaky} and $*$ is the channel-wise convolution operator that performs convolutions in a spatially-adaptive manner.

Separable filters in our IAC layer play a key role in resolving the limitation of the FAC layer.
Xu \etal \cite{Xu:2014:Deep} showed that a convolutional network with 1-dim filters can successfully approximate a large inverse filter for the deconvolution task.
Similarly, our IAC layer secures larger receptive fields at much lower memory and computational costs than the FAC layer by utilizing 1-dim filters, instead of 2-dim convolutions.
However, compared to dense 2-dim convolution filters in the FAC layer, our separable filters may not provide enough accuracy for deblurring filters.
We handle this problem by iteratively applying separable filters to fully exploit the nonlinear nature of a deep network.
Our iterative scheme also enables small-sized separable filters to be used for establishing large receptive fields.

\input{data/method/IAC}

\input{data/method/IFAN_camera}
\subsection{Deblurring Network with IFAN}
\label{sec:Deblurring_Newtwork}

\Fig{\ref{fig:network}} shows an overview of our deblurring network based on IFAN.
Our network takes a single defocused image $I_B\!\in\!\mathbb{R}^{H\times W\times3 }$ as input and produces a deblurred output $I_{BS}\!\in\!\mathbb{R}^{H\times W\times 3}$,
where $H$ and $W$ are the height and width of an image, respectively.
The network is built upon a simple encoder-decoder architecture consisting of a feature extractor, reconstructor, and IFAN module in the middle.
The feature extractor extracts defocused features $e_B\!\in\! \mathbb{R}^{h\times w\times c_{e}}$,
where $h~\texttt{=}\,\frac{H}{8}$ and $w~\texttt{=}\,\frac{W}{8}$,
and feeds them to IFAN.
IFAN removes defocus blur in the feature domain by predicting spatially-varying deblurring filters and applying them to $e_B$ using IAC.
The deblurred features $e_{BS}$ from IFAN is then passed to the reconstructor, which restores an all-in-focus image $I_{BS}$.
The detailed architecture of each network component
can be found in our supplementary material.
In the following, we describe IFAN in more detail.

\vspace{-13pt}
\paragraph{Iterative Filter Adaptive Network (IFAN)}\label{ssec:IFAN}
IFAN takes defocused features $e_B$ as input, and produces deblurred features $e_{BS}$.
To produce deblurred features, IFAN takes $I_B$ as additional input, and predicts deblurring filters from $I_B$.
IFAN consists of a filter encoder, disparity map estimator, filter predictor, and IAC layer.
The filter encoder encodes $I_B$ into $e_\textbf{F}\!\in\! \mathbb{R}^{h\times w\times c_{e}}$, which is then passed to the disparity map estimator.
The disparity map estimator is a sub-network specifically designed to exploit dual-pixel data for effectively training IFAN, which will be discussed in \Sec{\ref{sec:training}}.
After the disparity map estimator, the filter predictor predicts a deblurring filter map $\textbf{F}_{deblur} \!\in\! \mathbb{R}^{h\times w\times c_{\textbf{F}_{deblur}}}$, where $c_{\textbf{F}_{deblur}}\texttt{=}\,Nc_{e}(2k\,\texttt{+}\,1)$.
Finally, the IAC layer transforms the input features $e_B$ using the predicted filters $\textbf{F}_{deblur}$ to generate deblurred features $e_{BS}$.

\subsection{Network Training}
\label{sec:training}

We train our network using the DPDD dataset with two defocus-specific tasks: defocus disparity estimation and reblurring.
In this section, we explain the training data, our training strategy using each task, and our final loss function.

\vspace{-13pt}
\paragraph{Dataset}
We use dual-pixel images from the DPDD dataset \cite{Abuolaim:2020:DPDNet} to train our network.
A dual-pixel image provides a pair of stereo images with a tiny baseline, whose disparities are proportional to defocus blur magnitudes.
The DPDD dataset provides 500 dual-pixel images captured with a Canon EOS 5D Mark IV.
Each dual-pixel image is provided in the form of a pair of left and right stereo images $I_B^l$ and $I_B^r$, respectively. %
For each dual-pixel image, the dataset also provides a defocused image $I_B$, which is generated by merging $I_B^l$ and $I_B^r$, and its corresponding ground-truth all-in-focus image $I_S$.
The 500 dual-pixel images are split into training, validation, and testing sets, each of which contains 350, 74, and 76 scenes, respectively.
Refer to \cite{Abuolaim:2020:DPDNet} for more details on the DPDD dataset.
\input{data/method/RBN_camera}
\input{data/experiments/ablation/tbl_ablation_DPD}
\input{data/experiments/ablation/fig_ablation_DPD}

\vspace{-13pt}
\paragraph{Defocus disparity estimation}
As the disparities between dual-pixel stereo images are proportional to blur magnitudes, we can train IFAN to learn more accurate defocus blur information by training IFAN to predict the disparity map.
To this end, during training, we feed a right image $I_B^r$ from a pair of dual-pixel stereo images to our deblurring network.
Then, we train the disparity map estimator to predict a disparity map $d^{r\rightarrow l}\!\in\!\mathbb{R}^{h\times w}$ between the downsampled left and right stereo images.
Specifically, we train the disparity map estimator using a disparity loss defined as:
\begin{equation}
    \mathcal{L}_{disp} = \emph{MSE}(I_{B\downarrow}^{r\rightarrow l}, I_{B\downarrow}^l),
    \label{eq:L_disp}
\end{equation}
where $\emph{MSE}(\cdot)$ is the mean-squared error function.
$I_{B\downarrow}^l$ is a left image downsampled by $\frac{1}{8}$.
$I_{B\downarrow}^{r\rightarrow l}$ is a right image downsampled by $\frac{1}{8}$ and warped by the disparity map $d^{r\rightarrow l}$.
We use the spatial transformer \cite{Jaderberg:2015:STN} for warping.
By minimizing $\mathcal{L}_{disp}$, both filter encoder and disparity map estimator are trained to predict an accurate disparity map and eventually accurate defocus magnitudes.
Note that we utilize dual-pixel images only for training, and our trained network requires only a single defocused image as its input.

\vspace{-13pt}
\paragraph{Reblurring}
We train IFAN also using the reblurring task.
For the learning of reblurring, we introduce an auxiliary reblurring network.
The reblurring network is attached at the end of IFAN and trained to invert deblurring filters $\textbf{F}_{deblur}$ to reblurring filters $\textbf{F}_{reblur}$ (\Fig{\ref{fig:RBN}}).
Then, using $\textbf{F}_{reblur}$, the IAC layer reblurs a downsampled ground-truth image $I_{S\downarrow}\!\in\!\mathbb{R}^{h\times w\times 3}$ to reproduce a downsampled version of the defocused input image.
For training IFAN as well as the reblurring network, we use a reblurring loss defined as:
\begin{equation}
    \mathcal{L}_{reblur}=\emph{MSE}(I_{SB\downarrow}, I_{B\downarrow}),
    \label{eq:L_reblur}
\end{equation}
where $I_{SB\downarrow}$ is a reblurred image obtained from $I_{S\downarrow}$ using $\textbf{F}_{reblur}$, and $I_{B\downarrow}$ is a downsampled input image.
$\mathcal{L}_{reblur}$ induces IFAN to predict $\textbf{F}_{deblur}$ containing valid information about blur shapes and sizes needed for accurate reblurring. Such information eventually improves the performance of deblurring filters used for the final defocus deblurring.
Note that we utilize the reblurring network only for training.

\vspace{-13pt}
\paragraph{Loss functions}
In addition to the disparity and reblurring losses, we use a deblurring loss, which is defined as:
\begin{equation}
\mathcal{L}_{deblur}=\emph{MSE}(I_{BS}, I_{S}).
\label{eq:deblur}
\end{equation}

Our total loss function to train our network is then defined as $\mathcal{L}_{total}\,\texttt{=}\,\mathcal{L}_{deblur}\,\texttt{+}~\mathcal{L}_{disp}\,\texttt{+}~\mathcal{L}_{reblur}$.
Each loss term affects different parts of our network.
While $\mathcal{L}_{deblur}$ trains the feature extractor, IFAN, and reconstructor,
$\mathcal{L}_{disp}$ trains only the filter encoder and disparity map estimator in IFAN.
$\mathcal{L}_{reblur}$ trains both IFAN and reblurring network.
Note that we use dual-pixel stereo images $(I_B^l, I_B^r)$ only for training.
Both $\mathcal{L}_{deblur}$ and $\mathcal{L}_{reblur}$ utilize $I_B$ while $\mathcal{L}_{disp}$ utilizes $(I_B^l, I_B^r)$.
In this way, we can fully utilize the DPDD dataset for training our network.

\section{Experiments}

We implemented our models using PyTorch \cite{paszke2017automatic}.
We use rectified-Adam \cite{Liu:2020:RADAM} with $\beta_1\,\texttt{=}~0.9$, $\beta_2\,\texttt{=}~0.99$ and weight decay rate 0.01 for training our network.
We use gradient norm clipping with the value empirically set to 0.5.
The network is trained for 600k iterations with an initial learning rate of $1.0\!\times\!10^{\texttt{-}4}$, which is step-decayed to half at the 500k-th and 550k-th iterations, which are experimentally chosen.
We set the number of filters $N\,\texttt{=}\,17$ for $\textbf{F}_{deblur}$ and $\textbf{F}_{reblur}$.
For each iteration, we randomly sample a batch of images from the DPDD training set.
We use a batch size of 8, and randomly augment images with Gaussian noise, gray-scale image conversion, and scaling, then crop them to $256\!\times\!256$.

For the evaluation of defocus deblurring performance, we measure the Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM) \cite{wang2004ssim}, Mean Absolute Error (MAE), and Learned Perceptual Image Patch Similarity (LPIPS) \cite{zhang2018perceptual} between deblurred results and their corresponding ground-truth images.
We evaluate our models and previous ones on a PC with an NVIDIA GeForce RTX 3090 GPU.

\subsection{Ablation Study}
To analyze the effect of each component of our model, we conduct an ablation study (\Tbl{\ref{tbl:ablation}} and \Fig{\ref{fig:ablation}}).
All models in the ablation study are trained under the same conditions (\eg, optimizer, batch size, learning rate, random seeds, etc.).
For the validation, we compare a stripped-down baseline model and its five variants.
For the baseline model, we restructure the key modules specifically designed for our network using conventional convolution layers and residual blocks.
Specifically, we change the channel sizes of the last layers of the disparity map estimator and filter predictor blocks to $c_{e}$ so that they predict conventional feature maps.
We also replace the IAC layer with residual blocks that take a concatenation of $e_B$ and $\textbf{F}_{deblur}$ as input, and detach the reblurring network.
The baseline model is trained with neither \Loss{disp} nor \Loss{reblur} but only \Loss{deblur}.
For the other variants, we recover combinations of the restructured components one by one from the baseline model.
For the variants with the disparity map estimator, we train them also with \Loss{disp}.
Similarly, for the variants with the reblurring network, we train them also with \Loss{reblur}.
The detailed architectures of all the models used in the ablation study can be found in the supplementary material.
\input{data/experiments/comparison_DP/fig_comparison_DP_bests}

\vspace{-13pt}
\paragraph{Explicit deblurring filter prediction}
We first verify the effect of the filter prediction scheme implemented using the filter predictor and IAC layer.
\Tbl{\ref{tbl:ablation}} shows that
introducing the filter predictor and IAC layer to the baseline model increases the deblurring performance (the first and third rows in the table),
confirming the advantage of \textit{explicit} pixel-wise filter prediction in flexible handling of spatially-varying and large defocus blur.
In addition,
compared to the gain (PSNR: $0.36\pct$ and LPIPS: $3.21\pct$) obtained when the disparity map estimator is embedded in the baseline model (the second row in the table),
there is more performance gain (PSNR: $0.44\pct$ and LPIPS: $16.31\pct$) when the disparity map estimator is added to a model with the filter predictor and IAC layer (the fourth row in the table).
This observation validates that explicit utilization of deblurring filters has more potential in absorbing extra defocus blur-specific supervision provided by dual-pixel images.
\Fig{\ref{fig:ablation}} shows a qualitative comparison.
As shown in the figure, the filter predictor and IAC layer substantially enhance the deblurring quality ((b) and (c) vs. (d) and (e) in the figure).

\vspace{-13pt}
\paragraph{Disparity map estimation and reblurring}
We analyze the influence of the disparity map estimator and reblurring network.
Specifically, we compare the combinations $(\text{filter predictor}\,\texttt{+}\,\textrm{IAC}\,\texttt{+}\,\textrm{disparity map estimator)}$ and $(\text{filter predictor}\,\texttt{+}\,\textrm{IAC}\,\texttt{+}\,\textrm{reblurring network})$.
\Tbl{\ref{tbl:ablation}} show that the model with the disparity map estimator performs better than the model with the reblurring network in recovering textures (lower LPIPS), as the disparity map estimator helps more accurately estimate per-pixel blur amounts (\Fig{\ref{fig:ablation}}e and f).
On the other hand, the model with the reblurring network better restores overall image contents (higher PSNR), as the reblurring network guides deblurring filters to contain information about blur shapes and blur amounts.
We can also observe that the model with both disparity map estimator and reblurring network achieves the best performance in every measure (\Fig{\ref{fig:ablation}}g).
This shows that the disparity map estimator and reblurring network have synergistic effects, complementing each other. %

\subsection{Comparison with Previous Methods}
In this section, we compare our method with previous defocus map-based approaches as well as recent end-to-end learning-based approaches:
Just Noticeable Blur estimation (JNB) \cite{Shi:2015:JNB},
Edge-Based Defocus Blur estimation (EBDB) \cite{Karaali:2018:DMEAdaptive},
Defocus Map Estimation Network (DMENet) \cite{Lee:2019:DMENet},
DPDNet$_{S}$ and DPDNet$_{D}$ \cite{Abuolaim:2020:DPDNet}.
Among these, JNB, EBDB, and DMENet are defocus map-based approaches that first estimate a defocus map and perform non-blind deconvolution.
DPDNet$_{S}$ and DPDNet$_{D}$ are end-to-end learning-based approaches that directly restore all-in-focus images.
They share the same network architecture, but DPDNet$_{S}$ takes a single defocused image as input while DPDNet$_{D}$ takes a pair of dual-pixel stereo images.

For all the previous methods, we use the code (and model weights for learning-based methods, DMENet and DPDNet) provided by the authors.
For JNB, EBDB, and DMENet, we used the deconvolution method \cite{Krishnan:2008:deconvolution} to obtain all-in-focus images using the estimated defocus maps.
For DPDNet$_{S}$, we retrain the network on 8-bit images with the training code provided by the author, as the authors provide only a model trained with 16-bit images (refer to the supplementary material for the evaluation of our model trained on 16-bit images).
We measure computational costs in terms of the number of network parameters, number of multiply-accumulate operations (MACs) computed on a $1280\!\times\!720$ image, and average computation time computed on test images. %
For JNB and EBDB, which are not learning-based methods, we measure only their computation times.

We compare the methods on the DPDD test set \cite{Abuolaim:2020:DPDNet}.
\Tbl{\ref{tbl:comparison:DP}} shows a quantitative comparison.
The previous defocus map-based methods show poor performance on the real-world blurred images in the DPDD test set, which is even lower than the input defocused images due to their restrictive blur models.
On the other hand, the recent end-to-end approaches, DPDNet$_{D}$ and DPDNet$_{S}$, achieve higher quality compared to the previous methods.
Our model outperforms DPDNet$_{S}$ by a significant gap with a smaller computational cost.
Moreover, although our model uses a single defocused image, it outperforms DPDNet$_{D}$ as well, proving the effectiveness of our approach.

\Fig{\ref{fig:comparison:DP}} shows a qualitative comparison.
Due to their inaccurate defocus maps and restricted blur models, the results of the defocus map-based methods have a large amount of remaining blur (\Fig{\ref{fig:comparison:DP}}b).
DPDNet$_{S}$ and DPDNet$_{D}$ produce better results than the previous ones, however, tend to produce artifacts and remaining blur (\Fig{\ref{fig:comparison:DP}}c and \ref{fig:comparison:DP}d).
On the other hand, our method shows more accurate deblurring results (\Fig{\ref{fig:comparison:DP}}e), even with a single defocused input.
Especially, compared to DPDNet$_{D}$, \Fig{\ref{fig:comparison:DP}} show that our method better handles spatially-varying blur (the first row), large blur (the second row), and image structures as well as textures (the third row).
More qualitative results are in the supplementary material.

\vspace{-10pt}
\subsubsection{Generalization Ability}
\label{ssec:comp:Real}

\vspace{-3pt}
As our method is trained using the DPDD training set, which is captured by a specific camera (Canon EOS 5D Mark IV), one question naturally follows how well the model generalizes to other images from different cameras.
To answer this question, we evaluate the performance of our approach on other test sets.
\input{data/experiments/comparison_Real/tbl_comparison_Real}
\input{data/experiments/comparison_Real/fig_comparison_Real}

\vspace{-13pt}
\paragraph{RealDOF test set}
To quantitatively measure the performance of our method on real-world defocus blur images, we prepare a new dataset named \textit{Real Depth of Field (RealDOF)} test set.
RealDOF consists of 50 scenes. For each scene, the dataset provides a pair of a defocused image and its corresponding all-in-focus image.
To capture image pairs of the same scene with different depth-of-fields, we built a dual-camera system with a beam splitter as described in \cite{rim:2020:ECCV}.
Specifically, our system is equipped with two cameras attached to the vertical rig with a beam splitter.
We used two Sony a7R IV cameras, which do not support dual pixels, with Sony 135mm F1.8 lenses.
The system is also equipped with a multi-camera trigger to synchronize the camera shutters to capture images simultaneously.
The captured images are post-processed for geometric and photometric alignments, similarly to \cite{rim:2020:ECCV}.
More details about the RealDOF test set are in the supplementary material.

\Tbl{\ref{tbl:comparison:Real}} shows a quantitative comparison on the RealDOF test set.
The table shows that our model clearly improves the image quality, showing that the model can generalize well to images from other cameras.
Moreover, our model significantly outperforms the previous state-of-the-art single image deblurring method, DPDNet$_{S}$, by more than 2\,dB in terms of PSNR.
\Fig{\ref{fig:comparison:Real}} qualitatively compares our method and DPDNet$_{S}$.
While the result of DPDNet$_{S}$ contains some amount of remaining blur, ours looks much sharper with no remaining blur.

\vspace{-13pt}
\paragraph{CUHK blur detection dataset}
The CUHK blur detection dataset \cite{Shi:2014:CUHK} provides 704 defocused images collected from the internet without ground-truth all-in-focus images.
\Fig{\ref{fig:comparison:CUHK}} shows a qualitative comparison between DPDNet$_{S}$ and ours on the CUHK dataset.
The result shows that our method removes defocus blur and restores fine details more successfully than DPDNet$_{S}$.
Refer to the supplementary material for more qualitative results.

\input{data/experiments/effect/tbl_effect_stack}
\input{data/experiments/comparison_CUHK/fig_comparison_CUHK_camera}

\vspace{-13pt}
\paragraph{Pixel dual-pixel test set}
The DPDD dataset \cite{Abuolaim:2020:DPDNet} provide an additional test set consisting of dual-pixel defocused images captured by a Google Pixel 4 smartphone camera.
We refer the readers to the supplementary material for the qualitative evaluation on the Pixel dual-pixel test set.

\subsection{Analysis on IFAN}
In this section, we further investigate the effect of different components of IFAN.
We first analyze the effect of the number of separable filters $N$ in $\textbf{F}_{deblur}$.
Then, we evaluate the effect of IAC compared to \Net{FAC} \cite{Zhou:2019:STFAN}.

\vspace{-13pt}
\paragraph{Number of separable filters $N$}
A larger number of separable filters in $\textbf{F}_{deblur}$ leads IFAN to establish larger receptive fields and more accurate deblurring filters.
Consequently,
when $N$ is larger,
we can handle large defocus blur more accurately.
\Tbl{\ref{tbl:effect:stack}} compares the performances with different values of $N$,
where
the deblurring quality increases with $N$.
Based on the result, we choose $N\,\texttt{=}\,17$ for our final model, as the improvement is small for $N\,\texttt{>}\,17$.

\vspace{-13pt}
\paragraph{IAC vs. FAC}
Finally, we analyze the effectiveness of the proposed IAC compared to FAC \cite{Zhou:2019:STFAN}.
To compare IAC and FAC, we replace the IAC layers in both IFAN and reblurring network in our final model with FAC layers.
For the FAC layers, we use $k~\texttt{=}\,11$ to match the computational cost to that of our final model for fairness.
\Tbl{\ref{tbl:effect:IAC}} and \Fig{\ref{fig:effect:IAC}} respectively show quantitative and qualitative comparisons between IAC and FAC.
The comparisons show that IAC outperforms FAC even with fewer parameters and operations, as IAC is better in handling large defocus blur by covering a much larger receptive field ($35\!\times\!35$) on defocused features than the receptive field ($11\!\times\!11$) of FAC.

\input{data/experiments/effect/tbl_effect_SFAC}
\input{data/experiments/effect/fig_effect_SFAC}

\section{Conclusion}
This paper proposed a single image defocus deblurring framework based on our novel Iterative Filter Adaptive Network (IFAN).
IFAN flexibly handles spatially-varying defocus blur by predicting per-pixel separable deblurring filters.
For efficient and effective handling of a large blur, we proposed Iterative Adaptive Convolution (IAC) that iteratively applies separable filters on features.
In addition, IFAN learns to estimate defocus blur from a single image more accurately through the learning of disparity map estimation and reblurring.
In the experiments, we verified the effect of each component in our model, and showed that our method achieves state-of-the-art performance.

The proposed network is still limited in handling significantly large defocus blur (\eg, \Fig{\ref{fig:effect:IAC}}c has remaining blur).
Our network works best with a typical isotropic defocus blur,
but may not properly handle blur with irregular shape (\eg, swirly bokeh) or strong highlight (\ie, glitter bokeh).
Refer to the supplementary material for the failure cases.
We plan to extend our RealDOF test set by adding more image pairs of diverse blur types captured with various cameras and lenses.


\section{Introduction}
Deep learning algorithm has become the state-of-the-art in many computer vision tasks, yet demanding a large amount of labeled data \cite{dl}. So,  semi-supervised learning (SSL) \cite{SSL} has appeared to be an appealing method of utilizing unlabeled data, and one popular class of SSL method is consistency training \cite{consistency}. With this method, model obtains a predicted label from an input data, and uses it as pseudo-label on the input perturbed or model modified. Here, we present a method of consistency training with Grad-CAM loss, which has better generalization on prediction and adjustability for differnet datasets with different properties.

The training scheme with saliency-map consistency constraint has been shown in \cite{ECT}, while it is not fully examined as it uses old explanation method. While \cite{cam_ct} uses developed version of class activation map (CAM), this also follows the same issue of old version. In our method, by using Grad-CAM \cite{gradcam}, we did not only train the model to produce robust results on perturbation, but also control the training procedure by attaching the Grad-CAM module in different layers. Since Grad-CAM module can be attached in any target layers in the architecture unlike other methods, it can be used to guide the model in different scales.

We conducted several experiments on the baseline model ResNet \cite{resnet} with and without Grad-CAM consistency loss, and have shown improvement on image classification task. The ablation study is followed to show Grad-CAM consistency loss without any conventional consistency training is useful. Also, examination on the method by targeting different layers suggests better use-case of our method.

\begin{figure}[!tb]
 	\centering
    	\includegraphics[width=7cm]{./Fig/overview.png}
    	\caption{ An overview of consistency training with Grad-CAM loss. While conventional supervised learning algorithms use label for supervision, consistency training uses pseudo-label predicted from original unlabeled image, and train the model to produce consistent result among the original and augmented image. In our setting, the model outputs consistent Grad-CAM results. }
      	\label{fig:overview}
 \end{figure}

\section{Related Works}

\textbf{Semi-supervised learning} is the main paradigm of unlabeled data exploitation. Many SSL applications are based on 'smoothness assumption' \cite{SSL_smooth}, meaning that similar data would have similar labels. Another example is entropy minimization strategy \cite{SSL_entropy}, leveraging 'low-density assumption' implying that the decision boundary of trained model does not pass high data-density region in the space. Similarly, 'manifold assumption' is also often used for SSL training \cite{SSL_manifold}, so that the the data is assumed to be lied on a low dimensional manifold.

\textbf{Consistency training} is one of the most popular SSL approaches~\cite{MixMatch, meanteacher, VAT}. During the training, the model is trained to produce consistent results on inputs with various perturbations (e.g, crop or jittering). This method is under the assumption of the class probability should be same, so minimizing the error between $ || p(y; x) - p(y; Aug(x)) ||_2 ^2  $ where $ x $ is the input, $ y $ is the output, and $ Aug( \cdot ) $ is a stochastic transformation function with invariant label. There can be variety on using the loss function, as cross-entropy loss is chosen in FixMatch \cite{fixmatch}.

\textbf{Grad-CAM} is a way of visualizing the model decision. With necessity of model explanation, several works have developed method of visualizing saliency-maps via using gradient of model decisions~\cite{saliency, CAM}, and one of the state-of-the-art approaches \cite{ECT} leveraged the idea of explanation as the surrogate of causality by exploiting gradient-based saliency-map explanation consistency training. On top of this feasibility test on using saliency-map consistency training, we here show better application by using more advanced method, for example, Grad-CAM.

\section{Method}
The main idea of our method is using Grad-CAM for consistency training. As the second and fourth columns in \autoref{fig:overview} show, the model outputs different Grad-CAM results from an image with perturbation. Following smoothness assumption, we train the model to output similar results not only on labels but also on Grad-CAM result with specified target layer.

The objective of conventional SSL training can be written as: given model $ f( \cdot ) $ with labeled dataset $ \{X_L, Y_L\} = \{(\mathbf{x}_i, y_i)\}_{i=1}^N $ and unlabeled dataset $ X_U = \{ \mathbf{x}_{j} \}_{j=1}^M $,

\begin{align*}
    & \underset{\theta \in \Theta }
        {\operatorname{min}} \frac{1}{N}  \sum_{i=1}^{N} l( f(\mathbf{x}_i;\theta) ,y_i )
                    + \alpha \frac{1}{U}  \sum_{j=1}^{M} k( \mathbf{x}_j , Aug(\mathbf{x}_j) ) \\
\end{align*} where $ l, k $ denotes empirical supervised loss and consistency loss (e.g., $ L2 $-loss), respectively, and $ Aug(\cdot) $ is a perturbation function (e.g., adversarial augmentation \cite{VAT}).

Here, we adopt consistency training with explanation, and let the objective function $ k( \cdot ) $ be $ \frac{1}{M} \sum_{j=1}^M ( E(\mathbf{x}_j;\theta, L) - E( Aug(\mathbf{x}_j);\theta, L) )^2 $ where $ E (\mathbf{x};\theta, L) $ captures the Grad-CAM result from the input data \(\mathbf{x}\) with model parameter $\theta$ and the target layer $L$.

To be more specific, we add additional label consistency loss, meaning negative of cross entropy between predicted labels by the model from the original data and augmented data \( \sum_{j=1}^M f( \mathbf{x}_j ) log( f( Aug( \mathbf{x}_j ) ) )  \). More detailed algorithm is below.

\begin{algorithm}
\caption{Consistency training with Grad-CAM loss}\label{alg:cap}
\begin{algorithmic}
\State Initialize model function $ f(\cdot) $ with parameter $ \theta $
\State Set training coefficients $\alpha, \beta, \gamma$
\State Set target layer $ L $ for Grad-CAM result
\For {epoch 1, E}
\For {sample 1, N}
\State Sample labeled data $ (\mathbf{x}_l, y_l) $ from $ \{ X_L, Y_L \} $
\State Compute supervised loss $ L_{S} $ from $ f(\mathbf{x}_l) $ and $ y_l $
\EndFor
\For {sample 1, N}          \Comment{same number of samples}
\State Sample unlabeled data $ \mathbf{x}_u $ from iterator of  $ \{ X_U \} $
\State Set augmented data $\mathbf{z}_u = Aug(\mathbf{x}_u) $
\State Compute pseudo-label loss $ L_{P} $ from pseudo-label $ f(\mathbf{x}_u) $ and prediction $ f(\mathbf{z}_u)$
\State Compute Grad-CAM loss $ L_{G} $ from $ E(\mathbf{x}_u;\theta, L) $ and $ E( Aug(\mathbf{z}_u);\theta, L) $
\State Set overall loss $L = \alpha L_{S} + \beta L_{P} + \gamma L_{G} $
\State Update parameter $ \theta  \gets  \theta - \nabla L $
\EndFor
\EndFor
\end{algorithmic}
\end{algorithm}

As the former layers and the latter layers learn features with different scales, i.e., respectively local and global regions, we attempted the model to be trained to output similar output in different scales by targeting different layers of the baseline model.

\begin{figure}[!tb]
 	\centering
    	\includegraphics[width=8cm]{./Fig/training.png}
    	\caption{ How Grad-CAM result changes after training. The first row shows the differences between Grad-CAM result from original image and augmented image. The second and third row show how the results become similar. The last column shows the augmented version of Grad-CAM result from original image, so this was used as the pseudo-label. The difference between the second and the third row is the target layer. Here, the crop and rotation are only used for augmentation. }
      	\label{fig:training}
 \end{figure}

\section{Result}

\subsection{Experiment setting}
Here, we will explain the experiment setting details, especially on the dataset. ResNet50 was used for the baseline model during the experimentation. The model is trained with CIFAR-10 dataset \cite{cifar10} for image classification.

We used 1/5 of the training data for the validation dataset, and divided the training dataset into labeled training dataset and unlabeled training dataset. To test different ratios, the number of labeled training data was fixed to 5000, so each label contains 100 images on average. For the target ratios of unlabeled to labeled, 0.5, 1.0, 2.0 were selected, meaning 2500, 5000, 10000 images, respectively.

For image augmentation, we used crop, rotation, and color-jitters. In terms of fair comparison, we trained the baseline model with randomly augmented labeled data.

\begin{figure}[b]
 	\centering
    	\includegraphics[width=8cm]{./Fig/result.PNG}
    	\caption{ Results from different ratios and target layers with our method of using consistent training with Grad-CAM loss on image classification task. The average results and standard deviations from several tests are stated. Results from ablation study are also shown together for comparison. }
      	\label{fig:result}
 \end{figure}

\subsection{Performance test}

\autoref{fig:training}  shows, the Grad-CAM result from two different images became similar as the model gets trained. Comparing to the baseline model, our method showed improved accuracy rate result on image classification task. \autoref{table:training} states the empirical statistics of Grad-CAM consistency loss with test on validation set.

With more unlabeled data, model showed better improvements. This indicates that the use of more training data results in better generalization.

The target layer stated in the \autoref{table:training} indicates the last layer of the block (following the implementation of pytorch library \cite{pytorch_resnet}). The model showed best performance with the target layer 3, and showed a trend of more improvements as the target goes to the backside. We argue that this result is not arbitrary, but has strong correlation with the label distribution of the dataset.

\begin{table}[t]
\begin{center}
\caption{Performance test on consistency training with Grad-CAM}
\begin{tabular}{|c||c|c|c|c|c|}
\hline
\multirow{2}{*}{\textbf{Ratio}}
                         & \multicolumn{5}{c|}{\textbf{Accuracy [\%]}} \\ \cline{2-6}
                         & \textbf{Layer1} & \textbf{Layer2}  & \textbf{Layer3} & \textbf{Layer4} & \textbf{Baseline} \\
\hline
\multirow{2}{1em} {0.5}  & 88.87           & 89.46            & 89.89           & 89.63           &                   \\
                         & $\pm$ 0.88      & $\pm$ 1.58       & $\pm$ 0.41      & $\pm$ 0.15      &                   \\
\cline{1-5}
\multirow{2}{1em} {1.0}  & 89.33           & 89.59            & 90.41           & 90.10           &  89.61            \\
                         & $\pm$ 1.47      & $\pm$ 1.24       & $\pm$ 0.12      & $\pm$ 0.03      &  $\pm$ 0.84       \\
\cline{1-5}
\multirow{2}{1em} {2.0}  & 90.48           & 89.98            & \textbf{90.99}  & 90.43           &                   \\
                         & $\pm$ 0.88      & $\pm$ 0.88       & $\pm$ 0.88      & $\pm$ 0.88      &                   \\
\hline
\end{tabular}
\label{table:training}
\end{center}
\end{table}

\subsection{Ablation study}

\autoref{table:ablation} shows the result of the ablation study, examining how much Grad-CAM consistency along affected the training procedure. For the baseline in this ablation study, we used the combination of supervision loss $ L_{S} $ and pseudo-label loss $ L_{P} $.

To note on the baseline, as we used the loss function of cross-entropy loss, the result was worse than using only supervision loss. Nevertheless, the comparison states the exploitation of Grad-CAM loss guides the model appropriately. We are planning to apply different loss function for the pseudo-label loss.

The best results in the second column of the table shows the best results by using all three losses containing Grad-CAM loss $ L_{G} $ among all different ratios and target layers, showing that the use of Grad-CAM loss results in the improvement.

\section{Conclusion}

We have showed that the improvement can be achieved by using Grad-CAM consistency loss comparing to the baseline model or consistency training with only using pseudo-label. With trends in the result shows, the model showed better improvements with more unlabeled data used.

Unlike the latest studies \cite{ECT} \cite{cam_ct} , we could develop better use-case of the training method with respect to the adjustability. Here, we address some discussions on how the model could get generalized and how our method can be applied for training different data with enhanced adjustability.

\subsection{On generalization}
With little examination, computing Grad-CAM can be viewed as a weakly-supervised semantic segmentation. Thus, the consistency loss with Grad-CAM result may work as slightly different version of the loss used in \cite{consistent-semantic}; this work presents a method of consistency training for semantic segmentation task. Thus, Grad-CAM producing task would have worked as an auxiliary task, resulting better generalization on the image classification task also.

\subsection{On adjustability}
Our hypothesis on how targeting different layer matters is: as the former layers learn features from local regions, while the latter layers learn features from larger regions, training model to produces similar attention on the latter layer would affect the model to produce similar results in global scale. In other words, we believe when the label classes are hugely different (i.e., cat vs. dog) targeting latter layers are efficient, while when the label classes are locally different (i.e., leopard vs. cheetah) targeting front layers will be appropriate.

Although this hypothesis remains to be ensured with more experiments, we suggests our method to be used in variety, within different experimentation such as on different dataset having different label distributions.

\begin{table}[t]
\begin{center}
\caption{Grad-CAM consistency training ablation test}
\begin{tabular}{ |c|c|c|c| }
\hline
\multirow{2}{*}{\textbf{Ratio}} & \multicolumn{2}{c|}{\textbf{Accuracy [\%]}} \\\cline{2-3}
& \textbf{Best} & \textbf{Ablation}  \\
\hline
0.5 & 89.89 & 89.19 $\pm$ 1.16 \\
\hline
1.0 & 90.41 & 89.64 $\pm$ 1.57 \\
\hline
2.0 & 90.99 & 90.56 $\pm$ 1.44 \\
\hline
\end{tabular}
\end{center}
\label{table:ablation}
\end{table}

\bibliographystyle{ms}