\section{Introduction}
\label{sec:intro}
A great surge in neural rendering has emerged in the last few years.
Specifically, the Neural Radiance Fields~\cite{mildenhall2020nerf} (NeRF) has shown remarkable performances in the novel view synthesis.
NeRF leverages a fully connected network to implicitly encode a 3D scene as a continuous signal and renders novel views through a differentiable volume rendering.
However, when the rendering of NeRF performs, a large number of inferences are inevitable, making the computational burden of training and evaluation heavier.

Aware of this problem, related works have circumvented the shortcomings by introducing grid-based approaches~\cite{liu2020nsvf,yu2021plenoctrees,2021bakingnerf,Sun22dvgo,wu2021diver,yu2021plenoxels,2022relufields}, which store view direction-independent representations in dense grids.
While these methods explicitly encode the whole scene simultaneously, they face a trade-off between the computational cost of the model size and its performance.
Therefore, delicate training strategies such as pruning or distillation are often required to preserve the view-synthesis quality and reduce the model size.
Recently, Instant-NGP~\cite{mueller2022instant} addressed these problems by proposing multi-resolution hash encoding for positional encoding, which combines a multi-resolution decomposition with a lightweight hash grid.
The multi-resolution hash encoding achieved state-of-the-art performance and the fastest convergence speed of NeRF.

\input{Figures/fig1.tex}

Despite the impressive performance of multi-resolution hash encoding, the volume rendering procedure (emission-absorption ray casting~\cite{1984raycasting}) used in Instant-NGP depends largely on the accurate camera poses.
This method samples the points along the ray defined by direction and origin, which are determined by the camera pose.
However, obtaining accurate camera poses in real-world scenarios might be unavailable,
so most existing works utilize an off-the-shelf algorithm such as Structure-from-Motion (SfM), or COLMAP~\cite{2016sfm}.
The previous works~\cite{wang2021nerfmm,SCNeRF2021,lin2021barf} have attempted to resolve this issue by jointly optimizing camera poses and scene representations with the original NeRF.
However, applying this approach to multi-resolution hash encoding leads to severe deteriorations in pose refinement and scene representation.

Based on the gradient analysis of the na\"ive joint optimization of pose parameters and multi-resolution hash encodings,
we demonstrate that the non-differentiability of the hash function and the discontinuity of the $d$-linear weights as a function of the input coordinate leads to the fluctuation in the Jacobian of the multi-resolution hash encodings.
We investigate a novel learning strategy for jointly optimizing the camera pose parameters and the other parameters when the camera poses are noisy or unknown, utilizing the outstanding performance of multi-resolution hash encoding.

Given that, we propose to use a non-linear activation function in our straight-through estimator for smooth gradients in the backward pass, consistently maintaining the $d$-linear interpolation in the forward pass (\textit{ref.} ~Figure~\ref{fig:fig1}).
Moreover, we propose the multi-level learning rate scheduling that regulates the convergence speed of each level-wise encoding.
We also empirically show that a small decoder compared to the size of the hash table~\cite{mueller2022instant} converges to suboptimal when the camera poses are noisy.
The ablation studies on the depth and wide of the decoding networks and the core components of the proposed learning framework firmly validate our proposed method for robust camera pose refinement for multi-resolution hash encoding.

In summary, our contributions are three-fold:
\begin{itemize}
    \item[\textbullet] We analyze the derivative of the multi-resolution hash encoding, and empirically show that the gradient fluctuation negatively affects the pose refinement.
    \item[\textbullet] We propose an efficient learning strategy jointly optimizing multi-resolution hash encoding and camera poses, leveraging the smooth gradient and the curriculum learning for coarse-to-fine adaptive convergences.
    \item[\textbullet] Our method achieves state-of-the-art performance in pose refinement and novel-view synthesis with  a faster learning speed than competitive methods.
\end{itemize}
\section{Related Work}
\label{sec:2}
\subsection{Neural Rendering}
\label{subsec:2.1}
\citet{mildenhall2020nerf} first introduced the Neural Radiance Fields (NeRF) which parameterizes 3D scenes using neural networks.
They employed a fully differentiable volume rendering procedure and a sinusoidal encoding to reconstruct high-fidelity details of the scene representations.
The necessity of sinusoidal encoding was examined from the perspectives of kernel regression~\cite{tancik2020fourier}, or the hierarchical structure of a natural scene reconstruction task~\cite{2022pins}.

Subsequently, in order to improve the reconstruction quality of NeRF, various modifications have been proposed such as replacing the ray casting with anti-aliased cone tracing~\cite{barron2021mipnerf}, disentangling foreground and background models through non-linear sampling algorithms~\cite{2020nerfpp,neff2021donerf,barron2022mipnerf360}, or learning implicit surface instead of the volume density field, \textit{e.g.}, signed distance function~\cite{Oechsle2021unisurf,wang2021neus,yariv2021volume}.
Also, there are several applicative studies with decomposition of NeRF~\cite{pumarola2020dnerf,martinbrualla2020nerfw,Pratul2021nerv,boss2021nerd,rebain2021derf,park2021nerfies}, composition with generative works~\cite{2021GRAF,Niemeyer2020GIRAFFE,2022clipnerf,jain2021dreamfields}, or few-shot learning~\cite{yu2021pixelnerf,jain2021dietnerf,rebain2022lolnerf,2022sinnerf,chibane2021SRF,wei2021nerfingmvs,chen2021mvsnerf}.

\subsection{Accelerating NeRF}
\label{subsec:2.2.}
One crucial drawback of NeRF is its slow convergence and rendering speed.
To accelerate the training speed of NeRF, previous works have combined grid-based approaches which store view-direction-independent information on voxel grids.
\citet{liu2020nsvf} introduce a dense feature grid to reduce the computation burden of NeRF and progressively prunes the dense grids.
The other works pre-compute and store a trained NeRF to the voxel grid, increasing rendering speed~\cite{yu2021plenoctrees,2021bakingnerf}.
On the other hand, rather than distilling the trained NeRF to voxel grids, direct learning of features on the voxel has been proposed~\cite{Sun22dvgo,wu2021diver,2022fplen, yu2021plenoxels}.

While these methods have been successful in achieving near real-time neural rendering, they also come with drawbacks such as the increased model size and lower reconstruction quality caused by pre-storing the scene representation.
To overcome these limitations, \citet{mueller2022instant} recently proposed Instant-NGP, which utilizes spatial hash functions and multi-resolution grids to approximate dense grid features and maximizes the hierarchical properties of 3D scenes.
This approach allows for state-of-the-art performance and the fastest convergence speed simultaneously.

\subsection{NeRF with Pose Refinement}
\label{subsec:2.3.}
For the majority of neural rendering, it is crucial to have accurate camera intrinsic and extrinsic parameters.
In an effort to address this issue, \citet{2020inerf} proposed a method for combining pose estimation and NeRFs by utilizing an inverted trained NeRF as an image-to-camera pose model.
Subsequently, various methods for jointly optimizing camera pose parameters and 3D scene reconstruction have been proposed.
\citet{wang2021nerfmm} proposed a joint optimization problem in which the camera pose is represented as a 6-degree-of-freedom (DoF) matrix and optimized using a photometric loss.
Building upon this, \citet{2022sinerf} proposed a method that replaces ReLU-based multi-layer perceptrons (MLPs) with sine-based MLPs and employs an efficient ray batch sampling.

In addition to directly optimizing camera parameters, geometric-based approaches~\cite{SCNeRF2021, lin2021barf,2022garf} have also been suggested.
For example, \citet{lin2021barf} proposed BARF, which optimizes the warping matrix of the camera pose with a standard error back-propagation algorithm, utilizing curriculum training to adjust the spectral bias of the scene representation.

Unlike previous methods based on the original NeRF structure, our method is designed for grid-based approaches, especially for multi-resolution hash encoding, which shows outstanding performance in novel-view synthesis and its training speed.
The common NeRF structure and its variants are prone to slowly converge, but our method can be converged significantly faster with state-of-the-art reconstruction performance under the circumstance of noisy or unknown camera poses.

\subsection{Multi-Resolution Hash Encoding}
\label{sec:3.1.}
This section describes the multi-resolution hash encoding presented by \citet{mueller2022instant}, which we focus on.

\subsubsection{Multi-Resolution Hash Encoding}
As the combination of the multi-resolution decomposition and the grid-based approach with hashing mechanism, multi-resolution hash encoding is defined as a learnable mapping of input coordinate $\mathbf{x} \in \mathbb{R}^{d}$ to a higher dimension.
The trainable encoding is learned as multi-level feature tables independent of each other.

The feature tables $\mathcal{H} = \left \{ \mathcal{H}_{l} \ | \ l \in \left \{ 1, \dots, L \right \} \right \}$ are assigned to the $L$ levels and each table contains $T$ trainable feature vectors with the dimensionality of $F$.
Each level consists of the $d$-dimensional grids where each dimension has $N_{l}$ sizes considering multi-resolution.
The number of grids for each size exponentially grows from the coarsest $N_{\text{min}}$ to the finest resolutions $N_{\text{max}}$.
Therefore, $N_l$ is defined as follows:
\begin{align}
\label{eq:grid}
        b
        \vcentcolon &=
        \exp
        \left(
            \frac{\ln N_{\text{max}} - \ln N_{\text{min}}}{L-1}
        \right)
        \\
        N_{l}
        \vcentcolon &=
        \lfloor N_{\text{min}} \cdot b^{l-1} \rfloor.
\end{align}

For given a specific level $l$, an input coordinate $\mathbf{x}$ is scaled by $N_{l}$ and then a grid spans to a unit hypercube where $\lfloor \mathbf{x}_{l} \rfloor \vcentcolon = \lfloor \mathbf{x}_{l} \cdot N_{l} \rfloor$ and $\lceil \mathbf{x}_{l} \rceil \vcentcolon = \lceil \mathbf{x}_{l} \cdot N_{l} \rceil$ are the vertices of the diagonal.
Then, each vertex is mapped into an entry in the level’s respective feature table.
Notice that, for coarse levels where the number of the total vertices of the grid is fewer than $T$,
each vertex corresponds one-to-one to the table entry.
Otherwise, each vertex corresponds to the element of the $l^{\text{th}}$ table $\mathcal{H}_{l}$, whose table index is the output of the following spatial hash function~\cite{2003hash}:
\begin{equation}
\label{eq:spatial_hashing}
    h(x)
    =
    \left (
        \bigoplus_{i=1}^{d} x_{i} \pi_{i}
    \right )
    \quad \text{mod } T,
\end{equation}
where $\bigoplus$ denotes bitwise XOR and $\pi_{i}$ are unique and large prime numbers.
In each level, the $2^d$ feature vectors of the hypercube are $d$-linearly interpolated according to the relative position of $\mathbf{x}$.
However, the interpolation enables us to get the gradient of the table entry since the interpolating weights are the function of $\mathbf{x}$.
We will revisit this in the following section for analysis.

The output $\mathbf{y}$ of the multi-resolution hash encoding is the concatenation of the entire level-wise interpolated features and its dimensionality is $L \times F$.
For simplicity, we denote as $\mathbf{y} = f(\mathbf{x}; \theta)$ with its trainable parameter $\theta$.
Similar to the other neural renderings using differentiable volume rendering (emission-absorption ray casting), the decoding MLP $m(\mathbf y; \phi)$ predicts the density and the non-Lambertian color along the ray.
All trainable parameters are updated via photometric loss $\mathcal{L}$ between the rendered ray and the ground-truth color.

\subsubsection{Derivative of Multi-Resolution Hash Encoding}
For the gradient analysis, we derive the derivative of the multi-resolution hash encoding with respect to $\mathbf{x}$.
Let $\mathbf{c}_{i,l}(\mathbf{x})$ denote the corner $i$ of the level $l$ resolution grid in which $\mathbf{x}_{l}$ is located, and let $h_{l}(\cdot)$ represent the hash function for the $l$-level, as defined in \cref{eq:spatial_hashing}.

Next, consider a function $\mathbf{h}_{l}: \mathbb{R}^{d} \rightarrow \mathbb{R}^{F}$, whose output is a $l^{\text{th}}$ interpolated feature vector with $2^{d}$ corners, as given by,
\begin{equation}
\label{eq:trilinear}
    \begin{split}
        \mathbf{h}_{l} (\mathbf{x})
        =
        \sum_{i=1}^{2^{d}} w_{i, l}
        \cdot
        \mathcal{H}_{l}\big(h_{l}\big(\mathbf{c}_{i,l}(\mathbf{x}) \big) \big),
    \end{split}
\end{equation}
where $w_{i,l}$ denotes the $d$-linear weight, which is defined by the \textit{opposite} volume in a unit hypercube with the relative position of $\mathbf{x}$:
\begin{equation}
\label{eq:trilinear_weight}
    \begin{split}
        w_{i,l}
        =
        \prod_{j=1}^{d}
        \left(
            1 - | \mathbf{x}_{l} - \mathbf{c}_{i,l}(\mathbf{x}) |_j
        \right )
    \end{split}
\end{equation}
where the index $j$ indicates the $j$-th dimension in the vector.
We can redefine the multi-resolution hash encoding vector $\mathbf{y}$ as follows:
\begin{equation}
\label{eq:hash_enc}
    \begin{split}
        \mathbf{y}
        =
        f(\mathbf{x}; \theta)
        =
        \big[
            \mathbf{h}_1(\mathbf{x}); \dots; \mathbf{h}_L(\mathbf{x})
        \big]
        \in \mathbb{R}^{F'}
    \end{split}
\end{equation}
where the dimension of the output vector is $F' = L \times F$ after the concatenation.

The Jacobian $\nabla_{\mathbf{x}}\mathbf{h}_{l}(\mathbf{x}) \in \mathbb{R}^{F \times d}$ of the $l^{\text{th}}$ interpolated feature vector $\mathbf{h}_l (\mathbf{x})$ with respect to the $\mathbf{x}$ can be derived using the chain-rule as follows:
\begin{equation}
\label{eq:grad}
    \begin{split}
        \nabla_{\mathbf{x}}\mathbf{h}_{l}(\mathbf{x})
        &=
        \left[
            \frac{\partial {\mathbf{h}_{l}}(\mathbf{x})}{\partial {x}_1},
            \dots,
            \frac{\partial {\mathbf{h}_{l}}(\mathbf{x})}{\partial {x}_d}
        \right]
        \\
        &=
        \sum_{i=1}^{2^{d}}
        \mathcal{H}_{l}\big( h_{l} \big( \mathbf{c}_{i,l} (\mathbf{x}) \big) \big)
        \cdot
        \nabla_{\mathbf{x}}{w_{i,l}},
    \end{split}
\end{equation}
where $\mathcal{H}_{l}(g_{l}(\mathbf{c}_{l}(\mathbf{x}) ))$ is not differentiable with respect to $\mathbf{x}$ and the $k$-th element of $\nabla_{\mathbf{x}}{w_{i,l}} \in \mathbb{R}^{1 \times d}$ is defined by,
\begin{equation}
\label{eq:weight_grad}
    \begin{split}
        \frac{\partial {{w}_{i,l}}(\mathbf{x})}{\partial {x}_k}

        &=

        s_k
        \cdot
        \prod_{j \neq k}
        \left(
            1 - | \mathbf{x}_{l} - \mathbf{c}_{i,l}(\mathbf{x}) |_j
        \right ),
    \end{split}
\end{equation}
where $s_k$ denotes $s_k = \textit{sign}\big(|\mathbf{c}_{i,l}(\mathbf{x}) - \mathbf{x}_{l}|_k \big)$.

As seen in \cref{eq:grad}, the Jacobian $\nabla_{\mathbf{x}}\mathbf{h}_{l}(\mathbf{x})$ is the weighted sum of the hash table entries corresponding to the nearby corners of $\mathbf{x}$.
However, the gradient $\nabla_{\mathbf{x}_k}{w_{i,l}}$ is not continuous at the corners due to the variable $s_k$, causing the direction of the gradient to flip.
This oscillation of the gradient $\nabla_{\mathbf{x}}{w_{j,l}}$ is the source of gradient fluctuation, independently from $\mathcal{H}$.
For a detailed analysis of the derivatives and further discussion, please refer to \cref{sec:a.1.}.
\subsection{Camera Pose Refinement}
\label{sec:3.2.}
Camera pose can be represented as a transformation matrix from the camera coordinate to the world coordinate.
Let us denote the camera-to-world transformation matrix as $\begin{bmatrix} \mathbf{R} |   \mathbf{t} \end{bmatrix} \in \text{SE}(3)$, where $\mathbf{R} \in \text{SO}(3)$ and $\mathbf{t} \in \mathbb{R}^{3 \times 1}$ are rotation matrix and translation vector, respectively.

\subsubsection{Pose Refinement with the Sinusoidal Encoding}
\label{sec:3.2.1.}
The pose refinement using error back-propagation in neural rendering is jointly optimizing the 6 DoF pose parameters and neural scene representation through the differentiable volume rendering:
\begin{equation}
\label{eq:joint_opt}
    \begin{split}
        \phi^{*}, \psi^{*}
        =
        \arg \min_{\phi, \psi} \mathcal{L}(\mathcal{I}, \hat{\mathcal{I}}; \phi, \psi),
    \end{split}
\end{equation}
where $\phi$ and $\psi$ denote model parameters and trainable camera parameters, $\hat{\mathcal{I}}$ and $\mathcal{I}$ denote reconstructed color and its ground-truth color respectively.

Note that, to our knowledge, all previous works~\cite{2020inerf,wang2021nerfmm,2022sinerf,SCNeRF2021,lin2021barf,2022garf} of pose refinement in neural rendering utilize fully differentiable encoding with respect to the input coordinate (\textit{e.g.}, sinusoidal or identity).
However, they have limited performance compared to multi-resolution hash encoding~\cite{mueller2022instant}.

\subsubsection{Pose Refinement with Multi-Resolution Hash Encoding}
\label{sec:3.2.2.}
Now, we present the optimization problem of pose refinement with multi-resolution hash encoding.
Based on the \cref{eq:joint_opt}, we also directly optimize the camera pose parameters with multi-resolution hash encoding,
\begin{equation}
\label{eq:joint_opt2}
    \begin{split}
        \phi^{*}, \theta^{*}, \psi^{*}
        =
        \arg \min_{\phi, \theta, \psi} \mathcal{L}(\mathcal{I}, \hat{\mathcal{I}}; \phi, \theta, \psi),
    \end{split}
\end{equation}
where $\theta$ is a trainable parameter for multi-resolution hash encoding, \textit{i.e.}, the entries of the hash tables $\mathcal{H}_l$.
However, we observe that the pose refinement and reconstruction quality from the above optimization problem is much worse than the previous works (Refer to (e) of Table~\ref{tab:ablation}).

To explain the poor performance, we assume that the gradient fluctuation of \cref{eq:joint_opt2}, or \cref{eq:weight_grad}, negatively affects pose refinement.
Since the input coordinate $\mathbf{x}$ is defined as a rigid transformation of the camera pose $\begin{bmatrix} \mathbf{R} |   \mathbf{t} \end{bmatrix} $ and image coordinate (projected in homogeneous space $z=-1$), the gradient fluctuation propagates through the gradient-based updates of the camera poses.
We speculate that this fluctuation makes the joint optimization of the pose refinement and the scene reconstruction difficult.
In Appendix~\ref{sec:a.2.}, we present more details of the camera pose refinement with the gradient-based optimization.

\subsection{Non-linear Interpolation for Smooth Gradient}
\label{sec:3.3.}
To mitigate the gradient fluctuation,
we propose to use a \textit{smooth gradient} for the interpolation weight $w_{i,l} \in [0, 1]$ maintaining forwarding pass, inspired by the straight-through estimator~\cite{Bengio2013stochastic}.

For the {smooth gradient}, we use the activation function $\delta({w}_{i,l})$ whose derivative is zero at the corners of the hypercube, and ${w}_{i,l} \in [0, 1]$,
\begin{equation}
\label{eq:smooth_weighting}
    \begin{split}
        \delta(w_{i,l}) =
        \frac{1- \cos(\pi w_{i,l})}{2},
    \end{split}
\end{equation}
where the activation value $\delta(w_{i,l})$ is ranged in $[0, 1]$.

As a result, the gradient of $\delta({w}_{i,l})$ with respect to $\mathbf{x}$ is derived as follows:
\begin{equation}
\label{eq:smooth_grad}
    \begin{split}
        \nabla_{\mathbf{x}}\delta({w}_{i,l})
        &=
        \frac{\pi}{2} \sin (\pi {w}_{i,l})
        \cdot
        \nabla_{\mathbf{x}}{w_{i,l}}.
    \end{split}
\end{equation}
Remind that $\nabla_{\mathbf{x}}{w_{i,l}}$ is not continuous and flipped through the boundary of a hypercube.
In \cref{eq:smooth_grad}, the weighting by the sine function effectively makes the gradient smooth and continuous (\textit{ref.} ~Figure \ref{fig:fig2}b).
Moreover, the gradient of $\mathbf{x}$ near the boundary is relatively shrunk compared to the middle of the grids, which may prevent frequent back-and-forth across the boundary after camera pose updates.

However, we do not directly use this in the interpolation forward pass.
The cosine function in \cref{eq:smooth_weighting} unintentionally scatters the sampled points in a line toward the edges of the grids.
This phenomenon, which we refer to as the ``zigzag problem,'' can be addressed by the \textit{straight-through} estimator~\cite{Bengio2013stochastic}.
It maintains the results of the linear interpolation in the forward pass by the cancel-out of the last two terms in \cref{eq:straight_through}, and \emph{partially} uses the activation value $\delta({w}_{i,l})$ in the backward pass as follows:
\begin{equation}
\label{eq:straight_through}
    \begin{split}
        \hat{w}_{i,l} = w_{i,l} + \lambda \delta({w}_{i,l}) -  \lambda \tilde{\delta}({w}_{i,l}),
    \end{split}
\end{equation}
where $\lambda$ is a hyperparameter that adjusts the smooth gradient and the zigzag problem, $\tilde{\delta}$ denotes the detached variable from the computational graph.
The steps involved in the straight-through estimator are illustrated in Figure~\ref{fig:fig1}.
\input{Figures/fig2.tex}

For an additional discussion, we present an illustration of the zigzag problem in Appendix~\ref{sec:a.3.} (See Figure~\ref{fig:zig}).
Although this straight-through estimator does not perfectly make the gradient smooth and continuous with the addition in \cref{eq:straight_through}, it is empirically more effective than other mixing variants (see Appendix~\ref{sec:a.3.} and Table~\ref{tab:lambda}).

\subsection{Curriculum Scheduling}
\label{sec:3.4.}
As argued by ~\citet{tancik2020fourier,2022pins}, NeRFs exhibit a hierarchical structure, \textit{i.e.},
the coordinate-based MLPs can suffer from spectral bias issues, in which different frequencies converge at different rates.
\citet{lin2021barf} further address this issue in the pose refinement.
The research showed that the Jacobian of the $k^{\text{th}}$ positional encoding amplifies pose noise, making the na\"ive application of positional encoding inappropriate for pose refinement.

We observe that the multi-resolution hash encoding, which leverages the multi-level decomposition of scenes, exhibits a similar problem.
To resolve the problem, we propose a curriculum scheduling strategy to regulate the convergence rate of the level-wise encoding.
We weight the learning rates $\eta_{l}$ of the $l^{\text{th}}$ multi-resolution hash encoding $\mathbf{h}_{l}$ by
\begin{equation}
\label{eq:curr}
    \begin{split}
        \tilde{\eta}_{l} = r_{l}(t) \cdot \eta_{l},
    \end{split}
\end{equation}
where the weight of learning rate $r_{l}(t)$ is defined as
\begin{equation}
\label{eq:curr_weight}
    \begin{split}
        r_{l}(t) =
        \begin{cases}
            0 & \alpha(t) < l \\
            \frac{1-\cos((\alpha(t) - l) \pi)}{2} & 0 \le \alpha(t) - l < 1 \\
            1 & \text{otherwise},
        \end{cases}
    \end{split}
\end{equation}
and $\alpha(t) = L \cdot \frac{t-t_s}{t_e - t_s} \in [0, L]$ is proportional to the number of iterations $t$ in the scheduling interval $[t_s, t_e]$.

This weighting function is similar to the coarse-to-fine method proposed by \citet{park2021nerfies} and \citet{lin2021barf}.
However, in contrast to these previous works, we apply this weighting to the learning rate of the level-wise hash table $\mathcal{H}_{l}$.
This allows the decoding network receives the encodings from all levels, while high-level encodings are more slowly updated than the coarse levels.
We empirically found this multi-level learning rate scheduling effective in multi-resolution hash encoding.

\section{Method}
\label{sec:method}
As mentioned in Section~\ref{sec:intro},
we observed that a na\"ive error back-propagation for the camera pose refinement with multi-resolution hash encoding leads to inferior results compared to the use of sinusoidal encoding, \citep[\textit{e.g.},][]{SCNeRF2021,lin2021barf}.
To further understand the observation, we analyze the derivative of the multi-resolution hash encoding (Section~\ref{sec:3.1.}).
We point out that the gradient fluctuation of the multi-resolution hash encoding makes it difficult to learn the pose refinement and scene reconstruction jointly (Section~\ref{sec:3.2.}).
To address these, we propose a method for calibrating inaccurate camera poses in multi-resolution hash encoding (Section~\ref{sec:3.3.}).
Additionally, we find that the multi-level decomposition of a scene induces the different convergence rates of multi-level encoding, which results in limited camera pose registration (Section~\ref{sec:3.4.}).

\input{3_1_hash_encoding}
\input{3_2_pose_refine}
\input{3_3_Smooth}
\subsection{Implementation Details}
\label{sec:4.1.}
\subsubsection{Dataset}
\label{sec:4.1.1.}
We evaluate the proposed method against the two previous works,  BARF~\cite{lin2021barf} and GARF~\cite{2022garf}.
Since the implementation of GARF is unavailable, we re-implement GARF.
Our re-implemented GARF has the same structure as BARF except for sinusoidal encoding and Gaussian activation.
Following \citet{lin2021barf} and \citet{2022garf}, we evaluate and compare our method on two public novel-view-synthesis datasets.

\paragraph{NeRF-Synthetic.}
{NeRF-Synthetic}~\cite{mildenhall2020nerf} has 8 synthetic object-centric scenes, which consist of $100$ rendered images with ground-truth camera poses (intrinsic and extrinsic) for each scene.
Following \citet{lin2021barf}, we utilize this dataset for the noisy camera pose scenario.
To simulate the scenario of imperfect camera poses, we adopt the approach in \citet{lin2021barf} synthetically perturbing the camera poses with additive Gaussian noise, $\delta \psi \sim \mathcal{N}(\mathbf{0}, \ 0.15 \mathbf{I})$.

\paragraph{LLFF.}
{LLFF}~\cite{mildenhall2019llff} has 8 forward-facing scenes captured by a hand-held camera, including RGB images and camera poses that have been estimated using the off-the-shelf algorithm~\cite{2016sfm}.
Following previous works, we utilize this dataset for the \textit{unknown} camera pose scenario.
Unlike the synthetic datasets, we initialize all camera poses with the \textit{identity} matrix.
Note that, the camera poses provided by \textbf{LLFF} are the estimations obtained using the COLMAP algorithm~\cite{2016sfm}.
As such, the pose error measured in our quantitative results only indicates the agreement between the learned pose and the estimated pose using the classical geometry-based approach.

\subsubsection{Implementation Details}
\label{sec:4.1.2.}
For the multi-resolution hash encoding, we follow the approach of Instant-NGP~\cite{mueller2022instant}, which uses a table size of $T = 2^{19}$ and a dimensionality of $F=2$ for each level feature.
Each feature table is initialized with a uniform distribution $\mathcal{U}[0,\ 1e-4]$.
Note that we reproduce the entire training pipeline in PyTorch for pose refinement instead of using the original C$++$ \& CUDA implementation of Instant-NGP for fair comparison~\footnote{While our re-implementation performs almost the same with the original, it takes slightly longer training time due to PyTorch's execution latency. The performance of our re-implemented Instant-NGP is reported in Appendix~\ref{sec:b.1.}.}.

The decoding network consists of 4-layer MLPs with ReLU~\cite{2011relu} activation and 256 hidden dimensions, including density network branch and color.
We utilize the tiny-cuda-nn (tcnn)~\cite{2022tcnn} framework for the decoding network.
We present the other implementation details in Appendix~\ref{sec:b.1.}.
While we set $\lambda = 1$ by default for the \textit{straight-through estimator}, the other options are explored in Appendix~\ref{sec:a.3.}.

\subsubsection{Evaluation Criteria}
\label{sec:4.1.3.}
In conformity with previous studies~\cite{lin2021barf,2022garf}, we evaluate the performance of our experiments in two ways: 1) the quality of view-synthesis for the 3D scene representation and 2) the accuracy of camera pose registration.
We measure the PSNR, SSIM, and LPIPS scores for view-synthesis quality, as employed in the original NeRF~\cite{mildenhall2020nerf}.
The rotation and translation errors are defined as follows:
\begin{align}
\label{eq:pose_error}
    &E(\mathbf{R}) = \cos^{-1}\Big(\frac{\textit{tr}(\mathbf{R}'\cdot \mathbf{R}^{\rm T})-1}{2}\Big),
    \\
    &E(\mathbf{t}) \ = |\mathbf{t}' - \mathbf{t}|^2_2,
\end{align}
where $\begin{bmatrix} \mathbf{R}' |   \mathbf{t}' \end{bmatrix} \in \text{SE}(3)$ denotes the ground-truth camera-to-world transformation matrix and $\textit{tr}(\cdot)$ denotes trace operator.
Like the \citet{lin2021barf}, all the metrics are measured after the pre-alignment stage using the Procrustes analysis.
In experiments, all the camera poses $\psi$ are parameterized by the $\mathfrak{se}(3) $ ~Lie algebra with known intrinsics.
\subsection{Quantitative Results}
\input{Tables/synthetic}
\input{Tables/llff}
\subsubsection{Synthetic Objects in NeRF-Synthetic}
\label{sec:4.2.1}
Table~\ref{tab:synthetic} demonstrates the quantitative results of the NeRF-Synthetic.
In Table~\ref{tab:synthetic}, the proposed method achieves state-of-the-art performances in both pose registration and reconstruction fidelity across all scenes.
The results align with \citet{mueller2022instant} showing impressive performance on the scenes with high geometric details.

On the other hand, \citet{mueller2022instant} previously demonstrated that multi-resolution hash encoding is limited to the scenes with complex and view-dependent reflections, \textit{i.e.}, \textit{Materials}.
Although they attributed this limitation to their shallow decoding networks, we observed similar performance when utilizing deeper decoding networks.
We hypothesize that frequency-based encodings, such as sinusoidal or spherical harmonics, might be more appropriate for addressing complex and view-dependent reflections.
We will further investigate this issue in future work.

\subsubsection{Real-World Scenes in LLFF}
\label{sec:4.2.2}
We report the quantitative results of the LLFF dataset in Table~\ref{tab:llff}.
Note that GARF utilizes 6-layer decoding networks for this dataset.
In Table~\ref{tab:llff},  the proposed method outperforms the previous methods regarding reconstruction fidelity and pose recovery, especially for translation.
These results suggest that the learned pose from our method is closely related to that of the classical geometric algorithm, indicating that our proposed method can learn camera poses from scratch using the multi-resolution hash encoding.

In terms of rotation angle registration, our method outperforms BARF, achieving comparable performance to GARF.
Still, notice that our method achieves the best view-synthesis quality compared to the other methods.
Also, in Table~\ref{tab:llff_colmap}, we investigate the interaction with the COLMAP camera pose initialization and our method.
Please refer to \cref{sec:b.2.} for the details.
Here, the underbar denotes runners-up.

\subsection{Ablation Study}
\label{sec:4.2.3}
We present additional ablation studies to examine the proposed method's effectiveness.
Similar to the Instant-NGP~\cite{mueller2022instant}, all the following experiments are conducted on the \textit{Hotdog} in the NeRF-Synthetic dataset for comparison.
Note that other scenes behave similarly.

\input{Tables/ablations.tex}
\input{Tables/llff_colmap.tex}
\subsubsection{Component Analysis}
In Table~\ref{tab:ablation}, we perform the ablation study for our method to examine the role of each element.
As shown in row (b) compared with (c), the \textit{smooth gradient} significantly helps with pose refinements, resulting in more accurate pose registration and higher view-synthesis quality.
Also, from (a) and (b), we observe that the \textit{straight-through estimator} prevents unintentional jittering from the non-linear weighting showing outperformance.
Lastly, as shown in (a) and (d), our proposed multi-level learning rate scheduling reasonably enhances pose estimation and scene reconstruction qualities.

\subsubsection{Time Complexity}
In Figure~\ref{fig:fig3}, we visualize the comparison of the training speed between the proposed method and the previous works~\cite{lin2021barf,2022garf}.
By utilizing fast convergence of multi-resolution hash encoding, the proposed method achieves more than $20\times$ faster training speed compared to the previous works.
Remind that the proposed method outperforms previous methods both in pose registration and view synthesis.
\input{Figures/fig3.tex}

\subsubsection{Decoder Size}
Here, we examine the design criteria for decoding networks $m(\mathbf y; \phi)$ in terms of model capacity.
The original implementation of Instant-NGP~\cite{mueller2022instant} utilizes shallow decoding networks, resulting in the feature table $\mathcal{H}$ having a relatively larger number of learnable parameters than the decoding networks, \textit{i.e.}, $|\theta| \gg |\phi|$.
We find that this often leads to the suboptimal convergence of both the multi-resolution hash encoding and the camera pose registration.

Figure~\ref{fig:fig4} presents the view-synthesis quality with respect to varying model sizes of the decoding network.
Unlike the findings of \citet{mueller2022instant}, who did not observe improvement with deeper decoder MLPs (as shown by the dashed line in the Figure), we observe that the decoder size heavily impacts both the view synthesis and the pose registration.
Therefore, in cases where the camera pose is inaccurate, we assume a sufficient number of parameters in the decoder is necessary.
Informed by this analysis, we employ deeper and wider decoding networks than the original Instant-NGP: 4-layer MLPs with 256 neurons.
Note that competitive methods~\cite{lin2021barf,2022garf} utilize a deeper decoder network with 8-layer MLPs with 256 neurons having more parameters.
\input{Figures/fig4.tex}
\subsection{Qualitative Results}
\label{subsubsec:4.3.}
We present the qualitative results of our method compared with competitive methods. Please refer to Appendix~\ref{sec:b.4.}.
\section{Experiment}
\label{sec:4}
In this section, we validate our proposed method using the multi-resolution hash encoding~\cite{mueller2022instant} with inaccurate or unknown camera poses.

\input{4_1_Implementation_Details}
\input{4_2_Quantitative_Results}
\input{4_3_Qualitative_Results}
\section{Conclusion}
We investigate the joint optimization of camera poses and scene reconstruction using multi-resolution hash encoding.
Based on the careful analysis of the gradient fluctuation of the hash encoding, we propose a simple yet effective method of the straight-through estimator for gradient smoothing.
Additionally, we consider the spectral bias of multi-level decomposition and adopt a multi-level learning rate scheduling varying convergence rates of the multi-level encodings.
Our extensive experiments show that the proposed method successfully recovers the camera poses with state-of-the-art performance on NeRF-Synthetic and LLFF.
However, it shows limited performance in the scenes with complex reflections, inherited from the multi-resolution hash encoding~\cite{mueller2022instant}.
Nevertheless, with its state-of-the-art performance and fast convergence, we believe our method is a reasonable choice for real-world problems with imperfect or unknown camera poses.

\section{The Derivative of Multi-Resolution Hash Encoding}
\label{sec:a}
We describe the details of the gradient-based optimization of the camera poses in the multi-resolution hash encoding.

\subsection{Jacobian of the Interpolated Feature Vector}
\label{sec:a.1.}
Borrowed from the \cref{eq:grad} in the manuscript, the Jacobian $\nabla_{\mathbf{x}}\mathbf{h}_{l}(\mathbf{x}) \in \mathbb{R}^{F \times d}$ of the $l^{\text{th}}$ interpolated feature vector $\mathbf{h}_l (\mathbf{x})$ with respect to $\mathbf{x}$ can be derived using the chain-rule as follows:
\begin{equation}
\label{eq:grad_re}
    \begin{split}
        \nabla_{\mathbf{x}}\mathbf{h}_{l}(\mathbf{x})
        &=
        \left[
            \frac{\partial {\mathbf{h}_{l}}(\mathbf{x})}{\partial {x}_1},
            \dots,
            \frac{\partial {\mathbf{h}_{l}}(\mathbf{x})}{\partial {x}_d}
        \right]
        \\
        &=
        \sum_{i=1}^{2^{d}}
        \mathcal{H}_{l}\big( h_{l} \big( \mathbf{c}_{i,l} (\mathbf{x}) \big) \big)
        \cdot
        \left[
            \frac{\partial {{w}_{i,l}}(\mathbf{x})}{\partial {x}_1},
            \dots,
            \frac{\partial {{w}_{i,l}}(\mathbf{x})}{\partial {x}_d}
        \right].
    \end{split}
\end{equation}

Let $\bar{i}$ be one of the nearest corner indices from $\mathbf{c}_{i,l}$ in a unit hypercube, where $\mathbf{c}_{i,l}$ and $\mathbf{c}_{\bar{i},l}$ make an edge of the unit hypercube.
Among the $2^d$ corners, we have $2^{d-1}$ pairs like that.
Then, we have the relation for $w_{\bar{i}_k,l}$ as follows:
\begin{equation}
\label{eq:conjugate}
    \begin{split}
        \frac{\partial {{w}_{\bar{i}_k,l}}(\mathbf{x})}{\partial {x}_k}
        =
        - \frac{\partial {{w}_{i,l}}(\mathbf{x})}{\partial {x}_k},
    \end{split}
\end{equation}
which can be inferred from \cref{eq:weight_grad} since the relative positions of $\mathbf{x}$ are different for the two cases.

Using this relation and the appropriate choice of the indices, the $k^{\text{th}}$ element of Jacobian $\nabla_{\mathbf{x}}\mathbf{h}_{l}(\mathbf{x})$ can be rewritten as follows:
\begin{equation}
\label{eq:grad2}
    \begin{split}
        \frac{\partial {\mathbf{h}_{l}}(\mathbf{x})}{\partial {x}_k}
        &=
        \sum_{i=1}^{2^{d}}
        \mathcal{H}_{l}\big( h_{l} \big( \mathbf{c}_{i,l} (\mathbf{x}) \big) \big)
        \cdot
        \frac{\partial {{w}_{i,l}}(\mathbf{x})}{\partial {x}_k}
        \\
        &=
        \sum_{i=1}^{2^{d-1}}
        \left(
        \mathcal{H}_{l}\big( h_{l} \big( \mathbf{c}_{i,l} (\mathbf{x}) \big) \big)
        -
        \mathcal{H}_{l}\big( h_{l} \big( \mathbf{c}_{\bar{i}_k,l} (\mathbf{x}) \big) \big)
        \right)
        \cdot
        \frac{\partial {{w}_{i,l}}(\mathbf{x})}{\partial {x}_k}
        \\
        &=
        \sum_{i=1}^{2^{d-1}}
        \left(
        \mathcal{H}_{l}\big( h_{l} \big( \mathbf{c}_{i,l} (\mathbf{x}) \big) \big)
        -
        \mathcal{H}_{l}\big( h_{l} \big( \mathbf{c}_{\bar{i}_k,l} (\mathbf{x}) \big) \big)
        \right)
        \cdot
        \prod_{j \neq k}
        \left(
            1 - | \mathbf{x}_{l} - \mathbf{c}_{i,l}(\mathbf{x}) |_j
        \right ),

    \end{split}
\end{equation}
where $\prod_{j \neq k} \left( 1 - | \mathbf{x}_{l} - \mathbf{c}_{i,l}(\mathbf{x}) |_j \right )$ and the differences between the hash table entries are constant to the $x_k$, which make ${\partial {\mathbf{h}_{l}}(\mathbf{x})} / {\partial {x}_k}$ is constant along with the $k^{\text{th}}$ axis of the unit hypercube.
Notice that the last term can be seen as the weights defined as:
\begin{equation}
    \sum_{i=1}^{2^{d-1}} \prod_{j \neq k} \left( 1 - | \mathbf{x}_{l} - \mathbf{c}_{i,l}(\mathbf{x}) |_j \right ) = 1,
\end{equation}
where ${\partial {\mathbf{h}_{l}}(\mathbf{x})} / {\partial {x}_k}$ is the convex combination of the differences between two hash table entries.

Our speculation from this analysis is that ${\partial {\mathbf{h}_{l}}(\mathbf{x})} / {\partial {x}_k}$ is the linear slope defined by the differences among the hash table entries consist of each grid.
However, the slope is sharply changed when crossing the boundary of the grid.
For the one-dimensional case, this is a kind of \textit{triangular wave}~\footnote{\url{https://en.wikipedia.org/wiki/Triangle_wave}}, which is a periodic, piecewise linear, continuous real function.
This function has sharp local minima and maxima at the vertices of the wave, hindering appropriate gradient steps for accurate pose refinement.

Our method is proposed to remedy this problem.
We empirically show that the smaller gradient for the positions near the boundaries compared with the other positions is significantly helpful for accurate pose refinement.

\subsection{Camera-to-World Transformation}
\label{sec:a.2.}
This section shows that the gradient fluctuation of the Jacobian $\nabla_{\mathbf{x}}\mathbf{h}_{l}(\mathbf{x})$ also affects the gradient-based optimization of the camera poses by the chain-rule.

Remind that we denote the camera-to-world transformation matrix as $\begin{bmatrix} \mathbf{R} |   \mathbf{t} \end{bmatrix} \in \text{SE}(3)$, where $\mathbf{R} \in \text{SO}(3)$ and $\mathbf{t} \in \mathbb{R}^{3 \times 1}$, which are rotation matrix and translation vector, respectively.
Typically, the translation vector $\mathbf{t}$ is expressed as a 3D vector in Euclidean space.
Whereas for the rotation matrix on $\text{SO}(3)$ space, previous works adopt the 3 DoF axis-angle representation with the Rodrigues’ formula or the 6D continuous representation~\cite{zhou2018rot,SCNeRF2021}.

Consider an input coordinate of the image space as $\mathbf{x}_{I} \in \mathbb{R}^2$.
To transform the image coordinate to the world coordinate, the image coordinate is projected to its homogeneous space $z = -1$.
Then, we get the corresponding world coordinate $\mathbf{x} \in \mathbb{R}^{3 \times 1}$ by the rigid transformation of the camera pose as follows:
\begin{equation}
\label{eq:pose_transform}
    \begin{split}
        \mathbf{x} = \mathbf{R} \ { \begin{bmatrix} \mathbf{x}_{I} \\ -1  \end{bmatrix}} + \mathbf{t}.
    \end{split}
\end{equation}
In volume rendering, the emission-absorption ray casting utilizes the transformed coordinate $\mathbf{x}$ to sample the points $\mathbf{x'}$ in a casted ray.
Therefore, the derivative of camera pose parameters with respect to the $\mathbf{x'}$ is derived as follows:
\begin{align}
\label{eq:cam_grad}
    & \nabla_{\mathbf{R}}\mathbf{h}_{l}(\mathbf{x'})
    =
    \frac{\partial \mathbf{h}_l (\mathbf{x})}{\partial \mathbf{x'}}
    \cdot
    \frac{\partial \mathbf{x'}}{\partial \mathbf{R}},

    \\
    & \nabla_{\mathbf{t}}\mathbf{h}_{l}(\mathbf{x'})
    =
    \frac{\partial \mathbf{h}_l (\mathbf{x'})}{\partial \mathbf{x'}}
    \cdot
    \frac{\partial \mathbf{x'}}{\partial \mathbf{t}}.

\end{align}
As shown in the above equation, the relations between $\nabla_{\mathbf{x'}}\mathbf{h}_{l}(\mathbf{x'})$ and, the Jacobian of $\nabla_{\mathbf{R}}\mathbf{h}_{l}(\mathbf{x'})$ and $\nabla_{\mathbf{R}}\mathbf{h}_{l}(\mathbf{x})$ are linear.
Therefore, the gradient fluctuation of the $\nabla_{\mathbf{x'}}\mathbf{h}_{l}(\mathbf{x'})$ directly propagates to the gradient with respect to the camera poses, resulting in inappropriate updates for the pose refinement.

\paragraph{Comparison with Sinusoidal Encoding.}
The main difference of the pose refinement using multi-resolution hash encoding from sinusoidal encoding is \textit{unsmoothness}.
In the case of the sinusoidal encoding~\cite{lin2021barf,SCNeRF2021}, the encoding output of the \cref{eq:trilinear} is replaced with a differentiable encoding function $\gamma(\mathbf{x})$.
Rather than linearly approximating the feature of $\mathbf{x}$ within a unit hypercube, it is directly encoded using the sinusoidal encoding as follows:
\begin{equation}
\label{eq:sinusoidal}
    \begin{split}
        \mathbf{h}_{l}(\mathbf{x})
        = \gamma_{l}(\mathbf{x})
        =
        \left[
            \cos(2^l \pi \mathbf{x}),\ \sin(2^l \pi \mathbf{x})
        \right].
    \end{split}
\end{equation}
Unlike the multi-resolution hash encoding, it induces the smooth gradient with respect to the input coordinate $\mathbf{x}$.

\subsection{Zigzag Problem and the Straight-Through Estimator}
\label{sec:a.3.}
As discussed in the manuscript, if we use smooth interpolation weight directly, the cosine function in \cref{eq:smooth_weighting} unintentionally scatters the sampled points, which we refer to as the zigzag problem.
We present an illustration of the zigzag problem in Figure~\ref{fig:zig}.
As shown on the right side of the figure, if we directly use the smooth interpolation in a hypercube, the sampled points get closer to the edges. \newline \newline
To remedy this problem, we leverage the straight-through estimator where the hyperparameter $\lambda$ adjusts the ``zigzag problem'' between the original weight and the activation value.
We explore the $\lambda$ in Table~\ref{tab:lambda} to investigate this impact.

\input{Figures/zigzag.tex}
\input{Tables/lambda.tex}
\section{Additional Experiments}
\label{sec:b}

\subsection{Re-implementation of Instant-NGP}
\label{sec:b.1.}
\subsubsection{Implementation Details}
We follow the implementation settings from the original Instant-NGP ~\cite{mueller2022instant} and BARF~\cite{lin2021barf} except for the number of layers (Refer to Section~\ref{sec:4.2.3}).
For the NeRF-Synthetic and the LLFF, we resize the images for each dataset to $400 \times 400$ and $480 \times 640$, respectively.
We use the sinusoidal encoding with $4$ levels for view-direction encoding, employed by \citet{mildenhall2020nerf}.
For ray casting, we randomly sample $1024$ rays and $N=128$ samples per ray at each optimization step.
Notice that the sum of all weights normalizes all the reparameterized weights to ensure that their sum equals one.
We use the Adam optimizer and train all models for 200K iterations, with a learning rate of $5\times 10^{-4}$ that exponentially decays to $1\times 10^{-4}$.
For multi-level learning rate scheduling, we set the scheduling interval $[t_s, t_e] = [20\text{K}, 100\text{K}]$, which is between 10$\%$ to 50$\%$ progress of the entire training.

\subsubsection{Quantitative Results}
This section presents the performance of our re-implemented version of multi-resolution hash encodings.
The following experiments were conducted with ground-truth camera poses to compare it with the original Instant-NGP~\cite{mueller2022instant}.
In Table~\ref{tab:ngp_synthetic} and Table~\ref{tab:ngp_llff}, we confirm that our implemented version gets similar results compared with the original.
\input{Tables/ngp_synthetic.tex}
\input{Tables/ngp_llff.tex}

\subsection{Real World Scenes from COLMAP Initialization}
\label{sec:b.2.}
The manuscript investigates the challenging problem of jointly optimizing unknown camera poses and scene reconstruction using multi-resolution hash encoding.
In this section, we conduct additional experiments on the LLFF dataset with the COLMAP~\cite{2016sfm} camera pose initialization.
Given that the estimated poses from COLMAP are inaccurate, we further refine the provided poses through the joint optimization of camera poses and scene reconstructions.
\input{Tables/llff_colmap_appendix.tex}

Table~\ref{tab:llff_colmap} shows the quantitative results on the LLFF dataset with the COLMAP-initialized poses and the further pose refinement using the proposed method.
Here, '\textit{w/} Pose Refinement' denotes the joint optimization of camera poses and scene reconstructions by the proposed method.

As shown in row (b) compared with (a), the results show that the proposed method achieves better view synthesis quality for some scenes, such as \textit{Fortress, Leaves, Room}.
This supports that pose registration (from scratch) can often align more accurate poses for scene reconstruction, even better than the classical geometry-based approaches~\cite{2016sfm}.
Additionally, comparing rows (a) and (c) in the Table, it is observed that the pose registration from the COLMAP initialization significantly outperforms the COLMAP (fixed) or the from-scratch.
Informed by this analysis, since the entire pipeline is lighter than the other vanilla NeRF architectures, the proposed method is expected to be useful in real-world problems of novel-view synthesis with imperfect camera poses, even for quite accurate ones from the COLMAP.

\subsection{Qualitative Results}
\label{sec:b.4.}
In Figure~\ref{fig:qual}, we present the qualitative results of the proposed method in the fourth column.
For comparison, the ground-truth image and the rendering of two previous works, BARF~\cite{lin2021barf} and GARF~\cite{2022garf}, are also presented in the first, second, and third columns, respectively.
The ground-truth is shown at the beginning of each row.
The qualitative results generally show that our method gives the finer-grained novel-view synthesis with the camera pose refinement.

Figure~\ref{fig:qual} is shown in the next page.

\input{Figures/qual.tex}

\section*{Acknowledgements}
Most of this work was done while Hwan Heo and Soohyun Kim were research interns at NAVER AI Lab.
The NAVER Smart Machine Learning (NSML) platform~\cite{NSML} has been used in the experiments.


\section{Introduction}

Exemplar-based image translation is a subdivision of image translation, which aims to generate a image with exemplar image style, while maintaining the content information of the input image. Such methods have a wide range of promising applications, such as style transfer, face editing, scene transformation, etc.

Previous extensive studies \cite{ma2018exemplar,wang2019example,huang2017arbitrary,park2019semantic,zhang2020cross,zhou2021cocosnet,zhan2021unbalanced} have made great progress in exemplar-based image translation. However, current methods mostly focus on learning the correspondence between the two input domains and ignore the potentially useful information interactions within each domain. In this paper, we propose a more general approach that combines the two domain features of the inputs into a whole, and then uses a transformer-based model to learn both the inter-domain correspondence and the information interactions within each domain.

Recently, a number of transformer-based models \cite{dosovitskiy2020vit,carion2020end,wang2020axial,liu2021swin} have been applied to vision tasks. These works outperform CNN-based methods in their respective fields. Nevertheless, training a pure transformer model is time-consuming and highly dependent on the amount of data. Therefore, we utilize a hybrid structure of CNN and transformer to generate better results with limited amount of data. We first use CNN-based encoders to obtain features of the input image and the exemplar image and concatenate them into a whole. Then we propose a novel Cross-domain Feature Fusion Transformer (CFFT) to fuse the whole feature. The CFFT consists of a Feature Fusion Network and a Hierarchical Transformer Network. The former is used for feature fusion and the latter for learning fine-grained spatial alignment at multiple scales. By the improved transformer-based network, CFFT is able to continuously learn self-attention and cross-attention between two input features, so that the model can learn better inter-domain correspondence and potentially beneficial correlation information within each domain.

Based on the CFFT model, we propose CFFT-GAN, a general exemplar-based image translation method. It decouples and fuses features from different domains and leverages generative adversarial network to learn image translation. In particular, we employ encoders to extract features from different image domains and perform feature fusion through CFFT. Then, the fused feature is fed into the spatially-adaptive generator to synthesize the translation results. In addition, our CFFT model is a pluggable design, and we can achieve multi-domain image translation by cascading CFFT modules.

We validate the different types of exemplar-based image translation capabilities of our method on many high-resolution image datasets, including pose-to-image, edge-to-image, sketch-to-image, and mask-to-image. These adequate and diverse experiments have demonstrated the validity of our model and show that our approach has better performance than state-of-the-art methods. Ablation studies and feature visualization results indicate the importance of our proposed CFFT module. Extended multi-domain experiments reflect the potential of our approach.

Our contributions can be summarized as follows:
\begin{itemize}

    \item We present a general exemplar-based image translation method, CFFT-GAN, which using Cross-domain Feature Fusion Transformer module to learn both intra-domain and inter-domain correlation for more effective image translation.

    \item We propose a Hierarchical Transformer (Hiformer) network in CFFT for fine-grained spatial alignment at multiple scales.

    \item Benefiting from the pluggable nature of CFFT, we utilize cascaded CFFT modules to achieve multi-domain image translation.

    \item Rich validation and comparison experiments show that CFFT-GAN can achieve remarkable results in various image translation tasks and also demonstrate the superiority of our approach over SOTA methods.

\end{itemize}

\begin{figure*}[t]
\centering
\includegraphics[width=0.96\linewidth]{model.pdf}
\caption{Overall of our CFFT-GAN model for exemplar-based image translation. Two Encoders are used to extract different domain features. The CFFT module first concatenates different domain features as a whole feature, and performs feature fusion by FFN, and then learns fine-grained spatial alignment of the feature by Hiformer network. Finally, the output feature from CFFT is input to the spatially-adaptive generator to synthesize translation result.}

\label{fig:model}
\end{figure*}

\section{Related Work}

\subsection{Image Translation}
Image translation aims to learn the transformation between different image domains by training a conditional generative adversarial network. Some methods \cite{isola2017image,wang2018high,park2019semantic} use paired data for explicit supervision, while others \cite{zhu2017unpaired,choi2018stargan,choi2020stargan,yi2017dualgan,park2020contrastive} implement unsupervised training without paired data. Our method works well under both conditions. In addition image translation has been used in a variety of scenarios, such as image editing \cite{yu2018generative,wu2020cascade,shen2020interpreting}, domain adaptation \cite{liu2021source,li2020model}, pose transfer \cite{siarohin2018deformable,li2019dense,ren2020deep,ma2021must}. Recently,
exemplar-based image translation \cite{ma2018exemplar,huang2018multimodal,wang2019example,saito2020coco,zhang2020cross,zhou2021cocosnet,zhan2021unbalanced} has received more and more attention because of its flexibility and applicability. Usually exemplar-based image translation refers to the transfer of stylistic features from a exemplar image to some type of label image, such as edges \cite{zhu2017toward,lee2018diverse}, key points \cite{ma2017pose,men2020controllable,ma2021must}, segmentation masks \cite{wang2018high,park2019semantic}, etc. Some approaches \cite{huang2018multimodal,men2020controllable,ma2021must} use AdaIN \cite{huang2017arbitrary} to transfer style information from the exemplar image into the label image. More recently, CoCosNet v1 \cite{zhang2020cross} uses correlation matrices to learn cross-domain correspondences. UNITE \cite{zhan2021unbalanced} introduces optimal transport for proper
feature alignment and faithful style control in image translation. CoCosNet v2 \cite{zhou2021cocosnet} learn a full-resolution correspondence with a hierarchical GRU-assisted PatchMatch \cite{barnes2009patchmatch}. In this paper, we propose a new method using the transformer-based network to learn cross-domain feature fusion.

\subsection{Transformer}

Transformer was first proposed by \cite{vaswani2017attention} for machine translation. Early research on applying transformer to vision tasks has not received extensive attention, while the recent proposal of ViT \cite{dosovitskiy2020vit} model has set off a wave of research on visual transformer models. The idea of ViT is very straightforward, which is to transfer the original transformer model as invariant as possible to the image classification task. After ViT, more and more vision tasks started to use the transformer model and its variants. DETR \cite{carion2020end} applies transformer to object detection task with remarkable results. Axial-deeplab \cite{wang2020axial} applies axial attention to image segmentation. ViViT \cite{arnab2021vivit} proposes a video classification model based purely on visual transformer. Swin transformer \cite{liu2021swin} improves the performance of transformer for various visual tasks by further improving the computation of image patch window. Although transformer has achieved good results in various classical vision tasks, there are relatively few studies related to the use of transformer in dense estimation tasks like image generation. TransGAN \cite{jiang2021transgan} uses a pure transformer structure to design the generator and discriminator to train the adversarial model, but the resolution of the generated images is small. Recently, ViTGAN \cite{lee2021vitgan} combines ViT and GAN and designs a new regularization technique to achieve comparable results to CNN-based GAN models on many standard image generation benchmarks. These studies demonstrate that transformer models can also play an effective role in image generation. However, the mainstream image translation approaches are still using CNN-based networks. Because image translation requires that the generated image has accurate semantic information about the conditional input as well as style information about the exemplar image, which demands learning the correspondence between different image domains. In this paper, we propose a cross-domain feature fusion transformer model for exemplar-based image translation. This model can better learn the correspondences between different domains and the potential connections within each domain.

\section{Approach}
Exemplar-based image translation refers to transfer an input image $x_{A}$ from domain $A$ to domain $B$, and make it have the stylistic texture of exemplar $y_{B}$, while maintaining the content of $x_{A}$. Our proposed CFFT-GAN first transform $x_{A}$ and $y_{B}$ into feature space $F$ by two CNN-based encoders, then fuse them using Cross-domain Feature Fusion Transformer, and generate the translation result via a SPADE-based \cite{park2019semantic} residual generator. The overall of our method is illustrated in Figure \ref{fig:model}.

\subsection{Encoders}
\label{subsection1}
For exemplar-based image translation, the inputs are two different types of image domains, content input domain $A$ and style exemplar domain $B$. There is large semantic gaps between them, which is not conducive to the feature fusion later. To learn correspondence better, we need to map the two image domains into the same latent feature space $F$. Specifically, we let $\mathcal{E}_{A \rightarrow S}$ and $\mathcal{E}_{B \rightarrow S}$ represent the encoders of two domains, then the domain aligned latent features can be expressed as:
\begin{equation}
\begin{aligned}
&x_{F}=\mathcal{E}_{A \rightarrow F}\left(x_{A} ; \theta_{\mathcal{E_{A}}}\right), \\
&y_{F}=\mathcal{E}_{B \rightarrow F}\left(y_{B} ; \theta_{\mathcal{E_{B}}}\right),
\end{aligned}
\end{equation}
where $\theta$ is the learnable parameters for each encoder, and $x_{F}$ and $y_{S} \in \mathbb{R}^{C \times H \times W}$ represent the learned domain aligned features, which are used for later feature fusion. The domain alignment of features is trained by Equation \ref{domain_align}.

\subsection{Cross-domain Feature Fusion Transformer}
We propose a novel Cross-domain Feature Fusion Transformer (CFFT) model, shown on the Figure \ref{fig:model}. The CFFT is designed to learn not only inter-domain correspondences, but also feature correlations within domains. Therefore, we can more fully utilize the features of both domains for image translation. Cross-domain feature pairs are first flattened into feature vectors. Then we concatenate the flattened features into a whole along the channel dimension. There are a total of $L=H \times W$ tokens, and the dimension of each token is $1 \times 2C$. The purpose of this operation is that the transformer-based network can learn the correlation information within each feature and also indirectly learn the correspondence between different domain features.

\noindent \textbf{Feature Fusion Network.}
After adding the position embedding, we then use Feature Fusion Network (FFN) to learn feature fusion and semantic coarse matching. FFN consists of two Transformer Encoders (TE) \cite{vaswani2017attention}, each TE has a module layer number N=3. The two TEs are connected through a MLP layer for compressing feature dimension from 2C to C. In this way, FFN achieves a feature fusion role, and based on the transformer's self-attention mechanism, the module is able to learn both the inter-domain and intra-domain correlations within the total feature.

\noindent \textbf{Hierarchical Transformer.}
To decouple and match the two features more finely, we propose a Hierarchical Transformer (Hiformer) after FFN. It presents a stepped structure that continuously splits features in spatial dimension and performs self-learning optimization for each spatial part of the feature layer by layer. For each TE in Hiformer, we use a module layer number N=2. This approach implements a transformer-based feature learning process from coarse to fine. The number of feature splitting levels is adjustable, and we use 3 levels in this paper. We then gradually fuse the local features by same structure.

With the above network structure, we can obtain the output feature that have been decoupled and spatially aligned, as shown in the feature visualizations of Figure \ref{fig:model}.

\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{multi_domain.pdf}
\caption{Our method can achieve multi-domain image translation by cascading CFFT modules.}
\label{fig:model2}
\end{figure}

\subsection{Spatially-adaptive Generator}
The role of the generator network is to synthesize $\hat{x}_{B}$ with the content of input image and the style of exemplar image. In the previous step, we have obtained the fused and aligned feature named $\mathbf{f}$, we just need to feed it into the generator network. Therefore, we introduce a spatially-adaptive denormalization (SPADE) \cite{park2019semantic} structure as the generator. We concatenate the feature $\mathbf{f}$ with the scaled input image into $\mathbf{w}$, where scaled input image can help the initial training of the model. Then we feed $\mathbf{w}$ into the SPADE-based residual blocks as modulation parameters layer by layer. Specifically, for a particular feature layer $F^{i} \in \mathbb{R}^{C_{i} \times H_{i} \times W_{i}}$ in the generator, we have the following formula,
\begin{equation}
F_{c, h, w}^{i} = \gamma_{h, w}^{i}(\mathbf{w}_{i}) \times \frac{F_{c, h, w}^{i}-\mu_{h, w}^{i}}{\sigma_{h, w}^{i}}+\beta_{h, w}^{i}(\mathbf{w}_{i}),
\end{equation}
where $\mu_{h, w}^{i}$ and $\sigma_{h, w}^{i}$ are the calculated mean and standard deviation across channel dimension, $\gamma_{h, w}^{i}$ and $\beta_{h, w}^{i}$ are learnable weight networks, $\mathbf{w}_{i}$ is a copy of $\mathbf{w}$. Finally, the generated result of the image translation can be expressed as:
\begin{equation}
\hat{x}_{B}=\mathcal{G}\left(\mathbf{w}; \theta_{\mathcal{G}}\right),
\end{equation}
where $\theta_{\mathcal{G}}$ is the learnable parameters of the generator $\mathcal{G}$.

\subsection{Multi-domain Image Translation}
\label{CFFT-cascade}

Our proposed CFFT is a pluggable module design. When the inputs have more than two image domains, we can also easily decouple and fuse information from multiple domains by cascading CFFT modules. As shown in schematic Figure \ref{fig:model2}, we first learn a feature fusion using CFFT1 for domains A and B, then introduce a third domain C and jointly learn a feature fusion for all three domains by CFFT2. This way of learning facilitates the progressive fusion of features, allowing for better correspondence learning between several different domains. As shown in Figure \ref{fig:animation}, our method can introduce a third domain (pose) on top of the dual-domain image translation to achieve both image translation and face animation.

\subsection{Loss Functions}
We train the CFFT-GAN model end-to-end. Usually, semantically aligned data for different domains is accessible, for example $x_{A}$ and $x_{B}$, whereas triplets data $(x_{A}$,$y_{B};x_{B})$ may not be accessible. We therefore construct the pseudo-tuple $(x_{A},\widetilde{x}_{B};x_{B})$, where $\widetilde{x}_{B}=\mathcal{A}(x_{B})$, $\mathcal{A}$ denotes a collection of data augmentation operations, such as horizontal flip, geometric deformation, random crop, etc. The following loss functions are used for the training process.

\noindent \textbf{Domain Alignment Loss.} We need the features $x_{F}$ and $y_{F}$ learned by the encoders to be in the same feature domain, as explained in subsection "Encoders". Therefore, we let different domain features under the same semantic be aligned.
\begin{equation}
\mathcal{L}_{\text {align}}=\left\|\mathcal{E}_{A \rightarrow S}\left(x_{A}\right)-\mathcal{E}_{B \rightarrow S}\left(x_{B}\right)\right\|_{1}.
\label{domain_align}
\end{equation}

\noindent \textbf{Feature Matching Loss.} Following previous work \cite{johnson2016perceptual,zhang2020cross}, we used the pre-trained VGG-19 network to extract the features of the generated translation result and the corresponding Ground Truth separately, and let the features at each layer match.
\begin{equation}
\mathcal{L}_{\text {match}}=\sum_{l} \mu_{l}\left\|\phi_{l}\left(\hat{x}_{B}\right)-\phi_{l}\left(x_{B}\right)\right\|_{1},
\end{equation}
where $\phi_{l}$ is the output features of layer $l$ in the VGG-19 network, and $\mu_{l}$ is the corresponding weight for each layer.

\noindent \textbf{Translation Loss.} We want the model to output translation image that has semantic content of the input image and the stylized texture of the exemplar image. Therefore, We utilize perceptual loss to reduce semantic differences and contextual loss \cite{mechrez2018contextual} to constrain the stylistic consistency. Specifically, the perceptual loss is shown as follows,
\begin{equation}
\mathcal{L}_{\text {perc}}=\left\|\phi_{h}\left(\hat{x}_{B}\right)-\phi_{h}\left(x_{B}\right)\right\|_{2},
\end{equation}
where $\phi_{h}$ represents the output of a high-level layer in the pre-trained VGG-19 network, and this layer contains mainly high-level semantic information about the input image. The contextual loss (CX) is written as follows,
\begin{equation}
\mathcal{L}_{CX}=\sum_{l} \omega_{l}\left[-\log \left(C X\left(\phi_{l}\left(\hat{x}_{B}\right), \phi_{l}\left(y_{B}\right)\right)\right)\right],
\end{equation}
where CX stands for contextual similarity, $\phi_{l}$ is the $l$th layer of low-level features in the pre-trained VGG-19 network, which contain mainly the rich stylized texture of the image, and $\omega_{l}$ is the weight coefficients of the different layers.

\noindent \textbf{Adversarial Loss.} We also use a discriminator network to distinguish between the real samples and the generated samples, and train it together with our image translation model. The adversarial loss function is defined as,
\begin{equation}
\begin{aligned}
\mathcal{L}_{a d v}^{\mathcal{D}} &=-\mathbb{E}\left[h\left(\mathcal{D}\left(y_{B}\right)\right)\right]-\mathbb{E}\left[h\left(-\mathcal{D}\left(\mathcal{G}\left(x_{A}, y_{B}\right)\right)\right)\right], \\
\mathcal{L}_{a d v}^{\mathcal{G}} &=-\mathbb{E}\left[\mathcal{D}\left(\mathcal{G}\left(x_{A}, y_{B}\right)\right)\right],
\end{aligned}
\end{equation}
where $h(t)=min(0,-1+t)$ represents the hinge function to regularize the discriminator.

\noindent \textbf{Total Loss.} In conclusion, we optimize the following overall objective.
\begin{equation}
\begin{aligned}
\mathcal{L} &= \lambda_{\text{align}} \mathcal{L}_{\text {align }}+\lambda_{\text{match}} \mathcal{L}_{\text {match}}+\lambda_{\text{perc}} \mathcal{L}_{\text{perc}} \\
&+\lambda_{\text{CX}}\mathcal{L}_{\text {CX}}+\lambda_{\text{adv}}\left(\mathcal{L}_{\text {adv }}^{\mathcal{D}}+\mathcal{L}_{\text {adv }}^{\mathcal{G}}\right),
\end{aligned}
\end{equation}
where weights $\lambda$ are used to balance the individual objectives.

\begin{table*}[t]
\centering
\begin{tabular}{l|ccc|cc|cc|cc}
\hline
& \multicolumn{3}{c|}{Deepfashion} & \multicolumn{2}{|c|}{CelebA-HQ} & \multicolumn{2}{|c|}{Metfaces} & \multicolumn{2}{|c}{ADE20K}\\
\cline{2-10}
& \text { FID $\downarrow$ } & \text { SWD $\downarrow$ } & \text { LPIPS $\downarrow$ }  & \text { FID $\downarrow$ } & \text { SWD $\downarrow$ } & \text { FID $\downarrow$ } & \text { SWD $\downarrow$ } & \text { FID $\downarrow$ } & \text { SWD $\downarrow$ } \\
\hline
{CoCosNet v1}  & {14.4} & {29.0} & {0.2386} & {19.4} & {22.3} & {25.6} & { 24.3} & {26.4} & {10.5} \\
{UNITE}        & {13.1} & {26.6} & {0.2371} & {18.3} & {21.6} & {24.1} & {23.5} & \textbf{25.1} & {10.1}\\
{CoCosNet v2}  & {12.2} & {24.6} & {0.2245} & {16.2} & {19.7} & {23.3} & {22.4} & {25.2} & \textbf{9.9} \\
{Ours}   & \textbf{8.8} & \textbf{19.0} & {\textbf{0.2070}} & \textbf{12.4} & \textbf{18.1} & \textbf{22.6} & \textbf{21.3} & {25.4} & {10.1}\\
\hline
\end{tabular}
\caption{Quantitative comparison results of image quality between our method and SOTA methods.}
\label{table:metrics1}
\end{table*}

\section{Experiments}
\textbf{Datasets.} To verify the generality of our approach, we perform experiments on several widely used public datasets.
\begin{itemize}
    \item Deepfashion \cite{liu2016deepfashion} contains 52,712 human images with various appearances and poses. Human image paired data is used in our experiments. We also utilize Openpose \cite{cao2017realtime} to extract the key points of the human image as input pose.
    \item CelebA-HQ \cite{liu2015deep} is comprised of 30,000 high resolution images of faces in a variety of styles. We connect its face landmarks to form the face edge, and use the Canny edge detection algorithm to get the background edge, and finally combine them as the input.
    \item MetFaces \cite{karras2020training} is a high resolution art-style portrait dataset, which contains 1,336 images of human faces. We use HED \cite{xie2015holistically} to extract the sketch of the face images.
    \item ADE20K \cite{zhou2017scene} contains 25,574 scene images with 150-class semantic segmentation annotation. We use this dataset to validate the image translation from mask to scene.
    \item AFHQ \cite{choi2020stargan} is a animal faces dataset consisting of 15,000 high-quality images. We also use HED to extract the sketch of the animal faces.
\end{itemize}
For better comparison with the SOTA methods and to demonstrate the good performance of our method, all datasets are experimented with 512 resolution images, except for ADE20K where 256 resolution images are used.

\noindent \textbf{Baselines.} We compare our method with SOTA exemplar-based image translation methods. 1) CoCosNet v1 \cite{zhang2020cross} summarized various types of image translation tasks into exemplar-based image translation for the first time. It proposes a cross-domain correspondence learning method, which deforms the exemplar image by calculating the correlation matrix of two domain features. 2) UNITE \cite{zhan2021unbalanced} introduces optimal transport for proper feature alignment and faithful style control in image translation. 3) CoCosNet v2 \cite{zhou2021cocosnet} learn a full-resolution correspondence, and purpose a hierarchical GRU-assisted PatchMatch method for efficient correspondence computation. Because we propose a general image translation method, there is no comparison with task-specific methods. Besides, we do not compare with the approaches that directly learn image translation and fail to use an exemplar.

\noindent \textbf{Evaluation Metrics.} We adopt the commonly used image generation evaluation metrics to assess the performance of the various methods. Fréchet Inception Score (FID) \cite{heusel2017gans} is the metric used to evaluate the quality and diversity of generated images. Sliced Wasserstein distance (SWD) \cite{karras2017progressive} is applied to measure the statistical distance of low-level patch distribution between synthesized images and real images. Learned Perceptual Image Patch Similarity (LPIPS) is a semantic similarity evaluation metric, and we use it to evaluate the decoupling and semantic alignment of the models on Deepfashion dataset, since only Deepfashion has paired data to compute this metric.

For exemplar-based image translation, the generated image should have the same semantic content as the input and have the consistent style and texture as the exemplar. We therefore calculate two additional metrics for evaluating semantic consistency and style similarity. We utilize the pre-trained VGG model on ImageNet and use its high-level feature maps, $relu3\_ 2$, $relu4\_ 2$ and $relu5\_ 2$, to represent high-level semantics. We then calculate the high-level semantics average cosine similarity between the generated and content input images as a semantic consistency metric. Similarly, the style similarity metric can be calculated using the low-level feature maps, $relu1\_ 2$, $relu2\_ 2$.

\begin{table}[t]
\centering
\setlength{\tabcolsep}{0.4mm}{
\begin{tabular}{l|c|c|c|c}
\hline
& {Deepfashion} & {CelebA-HQ} & {Metfaces} & {ADE20K}\\

\hline
{CoCosNet v1}  & {0.924} & {0.945} & {0.941} & {0.862}  \\
{UNITE}        & {0.944} & {0.948} & {0.956} & {0.839} \\
{CoCosNet v2}  & {0.959} & {0.955} & {0.963} & {0.877}  \\
{Ours}         & \textbf{0.975} & \textbf{0.961} & \textbf{0.968} & \textbf{0.878}\\
\hline
\end{tabular}}
\caption{Quantitative evaluation of semantic consistency. The higher the score, the better the evaluation metric.}
\label{table:metrics2}
\end{table}

\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{compare.pdf}
\caption{Qualitative comparison results of our method with SOTAs on Deepfashion dataset, CelebA-HQ dataset, MetFaces dataset, and ADE20K dataset, respectively.}
\label{fig:compare}
\end{figure}

\begin{table}[t]
\centering
\setlength{\tabcolsep}{0.4mm}{
\begin{tabular}{l|c|c|c|c}
\hline
& {Deepfashion} & {CelebA-HQ} & {Metfaces} & {ADE20K}\\
\hline
{CoCosNet v1}  & {0.959} & {0.947} & {0.944} & {0.952} \\
{UNITE}        & {0.966} & {0.953} & {0.965} & {0.955} \\
{CoCosNet v2}  & {0.974} & {0.966} & {0.964} & \textbf{0.959} \\
{Ours}         & \textbf{0.982} & \textbf{0.974} & \textbf{0.971} & 0.954\\
\hline
\end{tabular}}
\caption{Quantitative evaluation of style similarity. The higher the score, the better the evaluation metric.}
\label{table:metrics3}
\end{table}

\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{animation.pdf}
\caption{Multi-domain image translation results. Through the figure \ref{fig:model2} structure, input edge and exemplar we can get translation result, and then input head pose we can get face animation result based on the translation result.}
\label{fig:animation}
\end{figure}

\begin{figure}[t]
\centering
\includegraphics[width=0.98\linewidth]{cross.pdf}
\caption{Portrait stylization. Our method can transfer image styles from the MetFaces to the faces of the CelebA-HQ. The model is only trained on the
MetFaces dataset. The model input is the edge image of input face.}
\label{fig:cross}
\end{figure}

\begin{figure}[t]
\centering
\includegraphics[width=0.98\linewidth]{vis_feature.pdf}
\caption{Visualization results of CFFT's input and output feature maps, which demonstrate the decoupling and semantic alignment capabilities of CFFT.}
\label{fig:hotmap}
\end{figure}

\noindent \textbf{Implementation Details.}
We use the TTUR \cite{heusel2017gans} learning strategy with a generator and discriminator learning rate of 1e-4 and 4e-4, respectively. We utilize Adam solver with $\beta_1 = 0$ and $\beta_2 = 0.999$. Our experiments are carried out on 8 32GB Tesla V100 GPUs. The input and output feature scales for the CFFT module are $(C,H,W)=(64,64,64)$. The weights of the total loss are $\lambda_{\text{align}}=10$, $\lambda_{\text{match}}=10$, $\lambda_{\text{perc}}=0.001$, $\lambda_{\text{CX}}=10$, $\lambda_{\text{adv}}=10$.

\begin{table}[t]
\centering
\begin{tabular}{l|c|c|c}
\hline
& {FID} & {SWD} & {Params} \\
\hline
{CFFT$\rightarrow$ CM} & {14.3} & {28.8} & {4.8M}  \\
{CFFT$\rightarrow$ AdaIN}  & {16.3} & {30.2} & {0.02M}  \\
{CFFT w/o Hiformer}  & {11.3} & {25.0} & {1.37M} \\
{CFFT 32$\times$32}  & {9.5} & {23.7} &  {2.22M}  \\
{Ours}               & \textbf{8.8} & \textbf{19.0} & 2.62M \\
\hline
\end{tabular}
\caption{Ablation studies of our method. Evaluate the FID and SWD metrics on the Deepfashion dataset, and compute the parameters of each module.}
\label{table:ablation}
\end{table}

\subsection{Experimental Results}

\noindent \textbf{Quantitative Evaluation.} For the quantitative comparison experiments, we first used FID and SWD to assess the quality of the generated images for each model, and use LPIPS to evaluate the model decoupling and semantic alignment ability of each model on the Deepfashion dataset. As shown in Table \ref{table:metrics1}, our approach outperforms the SOTA methods in most datasets. This is due to the fact that our proposed CFFT module is able to better learn the correspondence between different domains and the potentially beneficial information within each domain, allowing the generator to obtain richer information.

In addition to evaluating the quality of the generated images, we quantitatively assess the semantic consistency of the generated images and content inputs and the style similarity to the exemplar images, as shown in Table \ref{table:metrics2} and \ref{table:metrics3}. Compared with the SOTA methods, our model also achieves higher performance.

\noindent \textbf{Qualitative Comparison.} We make a qualitative comparison with SOTA methods, as shown in Figure \ref{fig:compare}. Our method can better fuse the style of the exemplar and the semantic content of the input. Our approach can also obtain more refined results when the semantic difference between exemplar and input is large, as shown in the third row. In addition to the comparative results, Figure \ref{fig:show} also show the more diverse generative results of our method. As can be seen from the figures our approach shows strong performance in a wide range of image translation tasks and the diversity of the generated results reflects the generalizability of our model.

\noindent \textbf{Applications.} Our method has good potential for application. Firstly, we can decouple and fuse information from multiple domains by cascading CFFT modules. We test a specific task, face animation. As shown in Figure \ref{fig:animation}, we first learn exemplar-based image translation between edge input and RGB exemplar, and then add head pose to learn the correspondence between three domains. The head pose is extracted via DECA \cite{feng2021learning}. The whole training process does not use paired data, and it is self-supervised training. Our generated results have the shape, style, and pose corresponding to the each input domain. By cascading CFFT modules, our approach can progressively learn the feature fusion from multiple image domains to achieve more complex image translation. Secondly, we test the portrait stylization experiment, as shown in Figure \ref{fig:cross}. We demonstrate the generalizability and excellent performance of our method through this cross-dataset experiment.

More comparative results and visualizations are presented in the supplementary material.

\subsection{Ablation Study}
We perform the ablation studies of CFFT, the key module of our model, on the Deepfashion dataset. The quantitative results are shown in Table \ref{table:ablation}. The first row of the table represents that we replace the CFFT module with the Correlation Matrix in CoCosNet. This method requires a larger amount of parameters and does not achieve good performance. In the second row of the table we replace the CFFT with the AdaIN \cite{huang2017arbitrary} structure, which is a very common method of feature transfer. Although it has few parameters, its performance is low for complex correspondence learning. In the third row, we remove the Hiformer in the CFFT, and the performance of the model has dropped significantly. In the fourth row, we decrease the spatial scale of the latent features in CFFT, from 64$\times$64 to 32$\times$32. But the model is still able to maintain good performance. In addition, we visualise the input and output features of the CFFT module, as shown in Figure \ref{fig:hotmap}. The results show that the CFFT is capable of good feature decoupling and semantic alignment.

\section{Conclusion}
This paper proposes CFFT-GAN, an exemplar-based image translation that learns the feature fusion of different domains by transformer-based module CFFT, which is able to not only learn the correspondence between domains better but also extract potential beneficial information within each domain. Our proposed CFFT has strong generality, and we can achieve multi-domain image translation by cascading CFFT. Quantitative and qualitative experiments demonstrate the superiority of our approach over the SOTA methods, and our ablation studies on the key module CFFT also reflect its core effect. There is still room for improvement in our approach, e.g., our model still has a performance deficit in the task of mask-to-image. We will optimize this task in future work.

\section{Acknowledgments}
This work is supported by the National Key Research and Development Program of China under Grant No. 2021YFC3320103.