\section{Introduction}
\noindent
Image super-resolution (SR) aims to estimate a perceptually plausible high-resolution (HR) image from a low-resolution (LR) input image ~\cite{yang2014single}. In general this problem is ill-posed due to the inherent information deficit between LR and HR images. Classic super-resolution image processing~\cite{pandey2018compendious} and deep learning based approaches~\cite{wang2020deep} result in visual artefacts for large up-scaling factors ($\times 4$).
To overcome this limitation recent research has introduced the sub-problem of reference image super-resolution (RefSR)~\cite{boominathan2014improving,yue2013landmark,zheng2018crossnet}. Given an input LR image and a similar HR reference image, RefSR approaches estimate a SR image. Reference super-resolution with a single reference image has been demonstrated to improve performances over general SR methods achieving large up-scaling with reduced visual artefacts.
\begin{table}[t]
\centering
\resizebox{1.\linewidth}{!}{\input{amrsr_arxive/image_files/intro2}}
 \vspace{-1.5em}
 \captionof{figure}{The proposed network exploits $N_M=4$ reference images (top left) to super-resolved the LR input (top right). Its SR output has the best visual quality compared to other state-of-the-art methods.}
 \vspace{-1.7em}
 \label{fig:intro}
\end{table}
\\In this paper we generalise reference super-resolution to use multiple reference images giving a pool of image features, and propose a novel adaptive sampling approach to learn the perceptual similarity between reference image features and the LR input.
The proposed adaptive multiple-reference super-resolution network (AMRSR) is designed to allow multiple HR reference images whilst maintaining memory efficiency by introducing a hierarchical attention-based adaptive sampling of the HR reference images. This allows training with multiple HR reference images which would not be possible with a naive extension of existing single-reference super-resolution methods without a significant increase in memory footprint.
Figure~\ref{fig:intro} qualitatively illustrates the performance of the proposed AMRSR approach against state-of-the-art single-image super-resolution (CSNLN~\cite{mei2020image}, RSRGAN~\cite{zhang2019ranksrgan}) and RefSR (SRNTT~\cite{ZhangSRNTT2019}) approaches. Given $N_M$ reference images, AMRSR produces a $\times4$ super-resolved image which is perceptually plausible and has a similar level of detail to the ground-truth HR image.
Primary contributions of the AMRSR approach presented in this paper are:

\begin{itemize}[noitemsep]
    \item Generalisation of reference super-resolution to multiple reference images whilst improving memory efficiency thanks to a part-based mechanism.
    \item Hierarchical attention-based adaptive sampling for perceptual similarity learning between low-resolution image features and multiple HR reference images.
    \item Improved quantitative and perceptual performance for image super-resolution compared with state-of-the-art single-image RefSR.
\end{itemize}

AMRSR is applied to both image and 3D model texture map super-resolution where multiple HR reference images are available.
The proposed approach is evaluated on 4 benchmark datasets and demonstrated to achieve significantly improved performance.
We introduce 3 new multiple reference super-resolution datasets which will be made available to benchmark future super-resolution approaches.

\section{Related work}
\subsection{Single-image super-resolution (SISR)}
A significant breakthrough in the SISR task was achieved when Dong \etal~\cite{dong2014learning} tackled the problem with a convolutional neural network (CNN). From this work, the application of deep learning progressively replaced classic SR computer vision methods~\cite{wang2020deep}. The pioneer work of Dong \etal~\cite{dong2014learning} belongs to a group of SR methods that use mean square error (MSE) as their objective function. VDSR~\cite{kim2016accurate} shows the importance of a deep layer architecture while SRResNet~\cite{ledig2017photo} and EDSR~\cite{lim2017enhanced} demonstrate the benefit of using residual block~\cite{he2016deep} to alleviate the training. Afterwards, several modifications of the residual structure such as skip connections~\cite{tong2017image}, recursive structures~\cite{tai2017image} and channel-attention~\cite{zhang2018image} further improved the accuracy of SISR. The state-of-the-art CSNLN~\cite{mei2020image} integrates a cross-scale non-local attention module to learn dependencies between the LR and HR couple. Other works propose lightweight networks to alleviate computational cost~\cite{muqeet2020ultra,liu2020residual}.
These residual networks ignore the human perception and only aim to high values of PSNR and SSIM~\cite{wang2020deep}, producing blurry SR images. Other SR networks (GANs) aim to enhance the perceptual quality of the SR images and were introduce in the SR task by Ledig \etal with SRGAN~\cite{ledig2017photo}.
The performances of SRGAN were improved by ESRGAN~\cite{ledig2017photo}, which replaces the adversarial loss with a relativistic adversarial loss. RSRGAN~\cite{zhang2019ranksrgan} develops a rank-content loss by training a ranker to obtain state-of-the-art visual results.
\\\indent \textbf{3D appearance super-resolution: }There are only two deep learning works that super-resolve texture maps to enhance the appearance of 3D objects.
The method proposed by Li \etal~\cite{li2019_3dappearance} processes, with a modified version of EDSR~\cite{lim2017enhanced}, LR texture maps and their normal maps to incorporate geometric information of the model in the learning. The pre-process to create normal maps introduces heavy computational cost.
In the second work~\cite{richard2019learned}, a redundancy-based encoder generates a blurry texture map from LR images that is then deblurred by a SISR decoder. Its main objective is not the super-resolution but the creation of texture maps from a set of LR multi-view images.
\subsection{Reference-based super-resolution (RefSR)}
GANs were introduced to solve the problems of the residual networks by focusing on the perceptual quality of the image. However, their generative nature leads to the creation of unnatural textures and artefacts in the SR image. RefSR approaches were applied to eliminate these artefacts by learning more accurate details from reference images.
One of the first RefSR networks is CrossNet~\cite{zheng2018crossnet}, which uses optical flow to align input and reference, limiting the matching of long distance correspondences. CrossNet was improved with two-stage cross-scale warping modules, adding to the optical flow alignment a further warping stage~\cite{tan2020crossnet++}. The optical flow introduces artefacts when misaligned references must be handled.
The "patch-match" approach correlates the reference and input images by matching similar patches. In an early, non deep learning framework~\cite{boominathan2014improving}, patches of downsampled reference image are matched with gradient features of the LR image.
Zheng \etal~\cite{zheng2017learning} adapted Boominathan \etal~\cite{boominathan2014improving} work to a CNN to perform semantic matching as well as to synthesise SR features through a SISR method. More recently, AugTT~\cite{ople2020super} introduces a similarity score to condition an augmentation network to take semantic similar patches from the references. Zhang \etal proposed SRNTT~\cite{ZhangSRNTT2019}, which swaps the most similar features of the reference and the LR image through convolutional layers. TTSR~\cite{yang2020learning} refines the matching mechanism by selecting and transferring only relevant textures from the reference to the LR image. SSEN~\cite{shim2020robust} performs the patch-match through deformable convolution layers using an offset estimator.
CIMR~\cite{cui2020towards} is the only method in the literature that exploits multiple references. It selects a smaller subset among all the features of generic reference images without performing any comparisons with the LR input, neglecting similar textures of the references that are not in the subset.
Our approach utilises a hierarchical patch-match based method to search for relevant texture information among all the feature vectors of multiple references, increasing the possibility to find more similar high-quality textures. It locates the attention on the similarity between sub-vectors of the LR input and the references to limit the number of local minima and ease the possibility of finding the most similar features. Finally, it significantly reduces the GPU usage of the patch-match approaches, facilitating the reproducibility of RefSR studies.

\section{Adaptive Multi-Reference Super-Resolution}

In this section we present the proposed AMRSR network, designed to exploit multiple HR reference images for training and inference whilst maintaining memory efficiency.  A hierarchical adaptive attention-based approach is introduced for image feature matching from the LR input to the HR reference images using a perceptual loss.
The problem of multiple-reference super-resolution can be stated as follows:
given a LR input $I_{LR}$ and a set of HR reference images $\{I^m_{ref}\}_{m=1}^{N_{M}}$, estimate a spatially coherent SR output $I_{SR}$ with the structure of $I_{LR}$ and the appearance detail resolution of the multiple-reference images $\{I^m_{ref}\}$.
The SR output should contain perceptually plausible high-resolution appearance detail without the introduction of visual artefacts such as blur, ringing or unnatural discontinuities observed with previous super-resolution approaches for large up-scaling factors ($>\times2$).
\subsection{Overview of Approach}

Figure \ref{fig:network} presents an overview of the proposed approach, which comprises the following stages.
\\\indent\textbf{Feature Extraction:} to reduce GPU memory consumption with multiple reference images, the LR input $I_{LR}$ and HR reference images   $\{I^m_{ref}\}^{N_M}_{m=1}$ are divided into $N_I$ and $N_{R}$ sub-parts, respectively. Image features are extracted from these parts using a pre-trained VGG-19 network~\cite{simonyan2014very}.
\\\indent\textbf{Hierarchical Attention-based Similarity:} map the features of the LR input to the most similar features of the references. The similarity $s_k$ is inferred between the feature vector of the LR input and the feature vectors of every reference image. The multiple reference images are then sampled based on the most similar features between the LR input and reference image features. This process is executed following a hierarchical structure with $l=N_L$ levels with an attention-based similarity mapping of the input feature vector. The output $O$ is a feature vector containing the most similar reference features to the input features.

\\\indent\textbf{Image Super-resolution:} Given the similarity feature mapping, a convolution network super-resolves the LR input $I_{LR}$ to obtain the SR output $I_{SR}$ which maintains the spatial coherence of the input with the HR appearance detail of the reference images.
\noindent
\\In contrast to the patch-match approach adopted by previous RefSR approaches \cite{ZhangSRNTT2019,yang2020learning,li2020survey,shim2020robust}, AMRSR performs feature similarity matching between sub-vectors of the LR input and reference images to focus the learning attention on the input features. A feature vector contains $N$ features of the input image, each of which is a matrix of values that represents a specific feature of the image. Instead of processing the whole matrix, we divide it into submatrices and perform the similarity mapping with these. The number of local minima and thus, the possibilities where to find the most similar values in each submatrix are reduced. The learning stage is eased and the performances are improved. These are not deteriorated by the lower number of patch-match comparisons because of the high similarity between neighbour values in submatrices of the same feature.

\begin{figure}[t]
    \centering
    \resizebox{0.9\linewidth}{!}{
    \includegraphics[width=0.85in]{amrsr_arxive/images/network2.pdf}}
    \vspace{-1.0em}
    \caption{AMRSR inference overview. Given a LR input image and $N_M$ reference images AMRSR comprises two modules: hierarchical attention-based similarity; and image super-resolution sampling. The resulting output is a HR reconstruction of the LR input image.}
    \vspace{-1.5em}
    \label{fig:network}

\end{figure}

\subsection{Feature extraction}
To conduct the input-references matching in the neural domain, feature vectors of the LR input and references must be retrieved. A problem of the RefSR approaches that apply a patch-match mechanism is the heavy memory consumption. To tackle this, we divide the input and reference images into $N_I$ and $N_{R}$ sub-parts, respectively. Feature vectors of these parts are extracted with a VGG-19 network~\cite{simonyan2014very} as shown in the first part of Figure~\ref{fig:network}. The feature vectors of each part of the input are then divided into $N_C$ sub-vectors $\{\{\phi^c_i(I_{LR})\}_{i=1}^{N_I}\}_{c=1}^{N_C}$ to perform an attention-based similarity on the LR input. For simplicity, we assume that $N_I=1$ (the input image is not divided into parts) and we express the set as $\phi^c(I_{LR})$ without loss of generality. In the case that $N_I>1$, the algorithm is repeated sequentially for each part and the outputs are concatenated as last step.
W.r.t. the references, $N_M \times N_R$ feature vectors are retrieved $\{\phi^r(I_{Ref}^m)_{m=1}^{N_M}\}_{r=1}^{N_{HR}}$ (expressed as $\phi^r(I_{Ref}^m)$ without loss of generality). Dividing the input and reference images into parts and inferring the similarity between them in a hierarchical order, establishes an efficient mechanism that reduces GPU memory requirements, which is important for practical implementation of multi-reference super-resolution.

\subsection{Hierarchical attention-based similarity}

\begin{figure}[h]
    \centering
    \resizebox{0.9\linewidth}{!}{
    \includegraphics[width=1.in]{amrsr_arxive/images/new_methodmod2.pdf}}
    \vspace{-1.em}
    \caption{3 levels hierarchical attention-based similarity. In $l=1$, similarity mapping is performed between input subvectors and every part of every reference, in $l=2$ between input subvectors and all the parts of a single reference, in $l=3$ between input subvectors and all the references.}
    \vspace{-1.7em}
    \label{fig:network2}

\end{figure}
The objective of this stage is to map the features of the LR input to the most similar features of the references. The output is a feature vector that contains the values of these most similar reference features.
To this purpose, a hierarchical approach of similarity mapping is performed over $l=N_L$ levels.
For every level $l$ of the hierarchy, a similarity map between LR input subvectors and reference features is computed. The most similar features are then retrieved considering the maximum values of the similarity map. A new feature vector is created with these features and used to compute the similarity map and the feature vector in the next level of the hierarchy, repeating the process.
\\The similarity map is computed with a convolution between the sub-vectors $\phi^c(I_{LR})$ of the LR input and $O^{l-1,r,m}_{ref}$, which is either the input reference feature vectors $\phi^r(I^m_{ref})$ if $l=1$ or new vectors created in the level $l=l-1$ (which contain features of the references $ref$):
\begin{equation}
\label{eq:1}
    s^{l}_{k}=\phi^c(I_{LR})*\frac {P_{k}(O^{l-1,r,m}_{ref})}{||P_{k}(O^{l-1,r,m}_{ref})||} \tag{1}
\end{equation}
$k=c$ if $l=1$, $k=r$ or $k=m$ otherwise. $P$ is the patch derived from the application of the patch-match approach: patches of $O^{l-1,r,m}_{ref}$ are convoluted with $\phi^c(I_{LR})$ to compute the correlation among them.
\\When the similarity map $s^{l}_{k}$ is retrieved, a vector $O^{l}_{ref}$ containing the most similar features of $O^{l-1,r,m}_{ref}$ is created by applying either one of two distinct approaches:
\begin{enumerate}[noitemsep]
    \item \textbf{Input attention mapping}: if $l=1$ (first time the similarity mapping is executed), a feature vector is created by maximising over every sub-vector of the input:
    \begin{gather}
        \label{eq:2}
        O^{1,r,m}_{ref}(x,y)=P_{k^*}(\phi^r(I^m_{ref}))(x,y) \tag{2}\\ k^*=\argmax_{k=c}s^{1}_{k}(x,y)\notag
    \end{gather}
    $O^{1,r,m}_{ref}(x,y)$ represents a single value in the $(x,y)$ position of the created feature vector $O^{1,r,m}_{ref}$. This value corresponds to the $(x,y)$ value of the $k^*$ patch $P(\phi^r(I^m_{ref}))$ whose $s^1$ is the highest among all the similarity values $s^{1}_{k}(x,y)$ for each sub-vector of the LR input feature vector.
    \item \textbf{Reference attention mapping}: for subsequent levels of the hierarchy, a feature vector is created by maximising a new similarity $s^{l}_{k}$ map over the feature vector created in the previous level.
    \begin{gather}
        \label{eq:3}
        O^{l,k}_{ref}(x,y)=P_{k^*}(O^{l-1,k^*}_{ref})(x,y) \tag{3}\\
        k^*=\argmax_{k}s^{l}_{k}(x,y)\notag
    \end{gather}
    $k=r$ or $ k=m$ depending on which level is processed. The value of $O^{l,k}_{ref}$ in the $(x,y)$ position is the value of $O^{l-1,k}_{ref}$ with the highest $s^{l}$ among all the $s^{l}_{k}(x,y)$ of $O^{l-1,k}_{ref}$.
\end{enumerate}
Both these mapping processes are repeated in a multi-scale fashion with three feature extractor levels to guarantee robustness to the variance of colour and illumination~\cite{ZhangSRNTT2019}.
\\The final output, obtained when the similarity mapping is performed for all the levels of the hierarchy, is a feature vector $O=O^{N_L}_{ref}$ which contains the features of the reference images that are most similar to every feature of the LR input.
\\When the final level of the hierarchy is processed, $N_K$ sets of weights $W_k$ are computed as the maximum of the scalar product between $\phi^c(I_{LR})$ and the $O^{l-1,k}_{ref}$ vector produced in the previous level.
\begin{equation}
\label{eq:4}
    W_k=max(\phi^c(I_{LR}) \cdot O^{l-1,k}_{Ref})\tag{4}
\end{equation}
The final set of weights $W$ is then retrieved from these sets: the weight in position $(x,y)$ of $W$ has the same value as the weight in position $(x,y)$ of the $k^*$-th set $W_{k^*}$ with $k^*$ obtained applying Equation~\ref{eq:3} with $l=N_L$: $W(x,y)=W_{k^*}(x,y)$
\\In our case there are $N_L=3$ levels of hierarchy:
\begin{enumerate}[noitemsep]
    \item $\bf l=1$: feature similarity mapping between every subvector of the input vector and every part of every reference. \textit{Input:} $\phi^c(I_{LR})$, $\phi^r(I_{Ref}^m)$, $k=c$. \textit{Output:} $\{\{O^{1,m,r}_{ref}\}_{m=1}^{N_{M}}\}_{r=1}^{N_{R}}$.
    \item $\bf l=2$: feature similarity mapping between the input subvectors and all the $N_R$ parts of a single reference, repeated for every reference. \textit{Input:} $\phi^c(I_{LR})$, $\{\{O^{1,m,r}_{ref}\}_{m=1}^{N_{M}}\}_{r=1}^{N_{R}}$, $k=r$.  \textit{Output:} $\{O^{2,m}_{ref}\}_{m=1}^{N_{M}}$.
    \item $\bf l=3$: feature similarity mapping between the input subvectors and all the references. \textit{Input:} $\phi^c(I_{LR})$, $\{O^{2,m}_{ref}\}_{m=1}^{N_{M}}$, $k=m$. \textit{Output}: $O=O^3_{ref}$.
\end{enumerate}

\subsection{Image super-resolution: }In the last stage, $I_{LR}$ is super-resolved with a generative network that exploits the information of the vectors obtained with the hierarchical similarity mapping whilst maintaining the spatial coherence of $I_{LR}$. These vectors are embedded to the input feature vector through channel-wise concatenation in different layers $h$ of the network.
We modified the architecture of the generator used by Zhang \etal~\cite{ZhangSRNTT2019} by eliminating the batch normalization layers since they can deteriorate the accuracy for dense pixel value predictions~\cite{yu2018wide} as well as reducing the number of residual blocks. More details are explained in the supplementary materials.
A texture loss is defined to enforce the effect of the texture swapping between $I_{LR}$ and the obtained $O$:
\begin{equation}
\label{eq:5}
    L_{tex}=\sum_{h}||Gr(\phi_h(I_{SR}\cdot W_h))-Gr(O_h\cdot W_h))|| \tag{5}
\end{equation}
where $Gr(\cdot)$ computes the Gram matrix.
Differently from~\cite{ZhangSRNTT2019}, the weighting map $W_h$ is computed among the $N_M$ references. The network aims to further minimise:
\begin{itemize}[noitemsep]
    \item The adversarial loss $L_{adv}$ to enhance the visual quality of the SR output. To stabilize the training, we use the WGAN-GP~\cite{gulrajani2017improved} for its gradient penalization feature.
    \item The reconstruction $L_1$ loss, since it has been demonstrated to be sharper for performance than $L_2$ loss~\cite{yang2020learning}.
    \item The perceptual loss~\cite{johnson2016perceptual} $L_{per}$ to enhance the similarity between the prediction and the target in feature space.
\end{itemize}

\section{Dataset}
\label{sec:dataset}
To the best of our knowledge, no datasets with multiple references related to the LR images are available for public usage (only with a single reference~\cite{ZhangSRNTT2019}).
To achieve our objective of multi-reference SR, we build three datasets:
\begin{enumerate}[noitemsep]
    \item \textbf{CU4REF}: this dataset is built from the single reference image dataset CUFED5~\cite{ZhangSRNTT2019}. Starting from the CUFED dataset~\cite{wang2016event}, four groups of images are defined, each with a different similarity level from the LR input images. We use the images in these groups as our references. The training set contains 3957 groups of LR and reference images while the testing set contains 126 groups (4 references for every LR image).
    \item \textbf{HUMAP:} to create the references of 67 synthetic human texture maps downloaded from several websites (\cite{renderpeople},\cite{cgtrader},\cite{turbosquid},\cite{free3d}), we import their 3D models in Blender~\cite{blender} and 8 reference images for each subject are captured by a virtual set of 8 cameras. Two realistic human texture maps retrieved from 16 multi-view images are added. Due to the low number of data, we augment the dataset by cropping the texture maps into patches (256x256 size) as done in~\cite{li2019_3dappearance}. The same references of the non-patched texture maps are associated to their patches. The training dataset consists of 5505 groups of patches and references. The testing dataset comprises 336 groups created from 5 texture maps of 6 subjects (3 captured by 16 video-cameras, 2 using a 5x5 FLIR Grasshooper3 cameras array) and a texture map of two people~\cite{collet2015high}. The references of two texture maps are captured by a different system of 64 DSLR cameras while the other references are the multi-view images used to retrieve the texture maps.
    \item \textbf{GEMAP:} it consists of generic LR texture map associated with 8 references. The texture maps are taken from the 3DASR dataset~\cite{li2019_3dappearance}, created from the multi-view images and 3D point clouds from other datasets (\cite{ley2016syb3r},\cite{seitz2006comparison},\cite{schops2017multi},\cite{goldlucke2014super},\cite{zhou2014color},\cite{zollhofer2015shading}). The reference images for the texture maps of~\cite{ley2016syb3r} are created with the same approach applied for HUMAP. For the other texture maps, the HR multi-view images captured by DSLR cameras are taken as references. The LR texture maps are cropped like in HUMAP. The training dataset contains 2032 groups while the testing one 290 groups.
\end{enumerate}
To evaluate the generalization capacity of AMRSR on RGB images, we test it on Sun80 dataset~\cite{sun2012super}, which has 80 natural images accompanied by a series of web-searching references that significantly differ from the input images.
\section{Experiments}
\label{sec:exp}
This section illustrates how AMRSR outperforms other state-of-the-art methods with quantitative and qualitative comparisons. Two ablation studies on the network configuration and on the advantage of multiple references are then presented. The GPU memory requirement of the state-of-the-art RefSR approaches is finally compared, confirming the efficiency of AMRSR.
The LR inputs are obtained by bicubic downscaling ($\times4$) from their HR counterparts and the SR results are evaluated on PSNR and SSIM on Y channel of YCbCr space. AMRSR parameters are set as: $N_M=4$,$N_I=1$,$N_R=1$ for CU4REF dataset, $N_R=16$ for the others (reason explained in subsection~\ref{ssec:gpu}). To integrate the input subvectors with the architecture structure, the value of $N_C$ is fixed:
\begin{equation}
    \label{eq:6}
    N_C(x)=\frac{length(\phi(I_{LR})(relu3\_1))}{length(\phi(I_{LR})(relux\_1)_{x=1}^{x=3})/4}\tag{6}
\end{equation}
where x indicates the three different layers used for the multi-scale fashion approach.
Further evaluations and results are presented in the supplementary materials.
\subsection{Comparison with state-of-the-art approaches}
\begin{table*}[!h]
\centering
\resizebox{0.85\linewidth}{!}{\input{amrsr_arxive/image_files/comparison}}
 \vspace{-1.0em}
 \captionof{figure}{Qualitative comparison among the state-of-the-art SR approaches on CU4REF (first two examples), Sun80 (third and fourth), GEMAP (fifth) and HUMAP (last). The references are shown for each example. The top left or the most left reference was used in the single-reference SR approaches.}
 \vspace{-1.5em}
 \label{fig:qualit}
\end{table*}
\begin{table}[!]
 \centering
\resizebox{0.45\textwidth}{!}{\input{table/SR2}}
\vspace{-1em}
\caption{PSNR/SSIM values of different SR approaches. The top part shows the results for the PSNR-oriented approaches while the bottom part for the Visual-oriented ones. * indicates that the reference images are downscaled of a factor of 2 (reason explained in Section\ref{ssec:gpu}).}
\vspace{-1.2em}
\label{tbl:quant}
\end{table}
Qualitative and quantitative comparisons are performed with state-of-the-art SISR and RefSR approaches. The SISR methods are both the PSNR-oriented EDSR~\cite{lim2017enhanced}, MDSR~\cite{lim2017enhanced}, RRDBNet~\cite{wang2018esrgan}, SRResNet~\cite{ledig2017photo}, RCAN~\cite{zhang2018image}, NHR~\cite{li2019_3dappearance}, NLR~\cite{li2019_3dappearance}, CSNLN~\cite{mei2020image}, MAFFSRN~\cite{muqeet2020ultra} and the visual-oriented SRGAN~\cite{ledig2017photo}, ESRGAN~\cite{wang2018esrgan}, RSRGAN~\cite{zhang2019ranksrgan}. The RefSR approaches are CrossNet~\cite{zheng2018crossnet}, SSEN~\cite{shim2020robust}, SRNTT~\cite{ZhangSRNTT2019} and TTSR~\cite{yang2020learning}.
For SR methods,
We train each network with the training datasets presented in Section~\ref{sec:dataset} with the same training configurations. Training with adversarial loss usually deteriorates the quantitative results. For a fair comparison with the PSNR-oriented methods, we train our model, SRNTT and TTSR only on reconstruction loss for 100 epochs (named with the suffix "\textit{$l_2$}").
NHR and NLR are tested on HUMAP and GEMAP since they required normal maps (retrieved with Blender~\cite{blender}).
\\\indent\textbf{Quantitative comparison: }The PSNR and SSIM values of the considered approaches are presented in Table~\ref{tbl:quant}, which is divided in two parts: the results of the PSNR-oriented networks are in the upper part while the ones of GANs and RefSR approaches (visual-oriented) are in the bottom part. The highest scores are highlighted in red while the second highest scores are blue. The bold red figures are the highest among all the results (within both the parts of the table). AMRSR-\textit{$l_2$} achieves the highest values of PSNR and SSIM in the PSNR-oriented case and among all the results for the four datasets while AMRSR figures confirm the superior perfomances among the visual-oriented approaches.
\\\indent\textbf{Qualitative comparison: }
Figure~\ref{fig:qualit} shows visual SR examples of the most relevant approaches considered in our evaluation. The SR outputs produced by the PSNR-oriented methods (RCAN, CSNLN) are blurrier and the details are less sharped. The results produced by the visual-oriented approaches (RSRGAN, SRNTT, TTSR) present unpleasant artefacts such as ringing and unnatural discontinuities.
The SR outputs of AMRSR are less blurry with sharper and finer details as shown in the zoomed patches of the examples, whose quality is higher than the other results.
\\\indent\textbf{User study evaluation: }
To further evaluate the visual quality of the SR outputs, we conduct a user study, where AMRSR is compared with five approaches.
50 random SR output pairs were shown to 60 subjects. Each pair consists of an image of AMRSR and the counterpart generated by one of the other approaches. The users were asked to pick the image with the highest resolution and best visual quality. The values on Y-axis of Figure~\ref{fig:chart} illustrates the percentage of the subjects that select AMRSR outputs over the other methods. Our approach significantly outperforms the other methods with over 90\% of users voting for AMRSR.
\begin{table}[!]
    \centering
    \resizebox{0.45\textwidth}{!}{\input{table/simlevel}}
    \vspace{-1em}
    \caption{Quantitative comparison between AMRSR and RefSR methods using single reference with different levels of similarity to the LR input.}
     \vspace{-1.5em}
    \label{tab:difref}
\end{table}
\begin{figure}[!]
\centering
\resizebox{0.85\linewidth}{!}{\input{amrsr_arxive/image_files/chart}}
\vspace{-1.5em}
\caption{Percentage of users that prefer AMRSR over 5 approaches in the user study. The error bars indicate the $95\%$ confidence interval.}
\vspace{-2em}
\label{fig:chart}
\end{figure}
\begin{table*}[!]
    \centering
    \resizebox{0.85\textwidth}{!}{\input{table/sim}}
    \vspace{-1.0em}
    \caption{PSNR/SSIM values with multiple references dissimilar to the LR inputs on CU4REF dataset}
    \vspace{-1.2em}
    \label{tab:sim}
\end{table*}
\begin{table*}[!]
 \centering
\resizebox{0.9\textwidth}{!}{\input{table/attention}}
\vspace{-1em}
\caption{PSNR/SSIM values of different configurations of AMRSR. The "no attention" configuration do not have any attention mechanism, the "ref" executes an attention mechanism on the reference image vectors while the "both" performs attention on both the reference and the LR input vectors.}
\vspace{-2em}
\label{tbl:attention}
\end{table*}
\begin{table}[!]
    \centering
    \resizebox{0.45\textwidth}{!}{\input{table/cimr}}
    \vspace{-1em}
    \caption{Quantitative comparison between AMRSR and CIMR exploiting single reference images with different levels of similarity with the LR inputs. The results of CIMR are taken from~\cite{cui2020towards}}
    \vspace{-1.2em}
    \label{tbl:cimr}
\end{table}
\begin{table}[!]
\centering
\resizebox{1.\linewidth}{!}{\input{amrsr_arxive/image_files/sim}}
 \vspace{-1.0em}
 \captionof{figure}{Visual comparison of SR outputs obtained with multiple references dissimilar to the LR inputs.}
 \vspace{-1.5em}
 \label{fig:aim}
\end{table}
\begin{table}[!]
    \centering
    \resizebox{0.45\textwidth}{!}{\input{table/cimr2}}
    \vspace{-1em}
    \caption{Quantitative comparison between AMRSR and CIMR exploiting multiple reference images. The results of CIMR are taken from~\cite{cui2020towards}}
    \vspace{-1.7em}
    \label{tbl:cimr2}
\end{table}
\label{ssec:attention}
\\\indent\textbf{Influence of dissimilar references: }Similarity between LR and Ref images highly influences the performances of RefSR methods~\cite{ZhangSRNTT2019}. Following the setting in~\cite{ZhangSRNTT2019}, we evaluate the effect of having dissimilar reference images to the LR input testing on four similarity levels (from L1 to L4) of the CUFED5 test set. The highest figures of PSNR and SSIM (Table~\ref{tab:difref}) are obtained by our network when a single reference is used even though its level of similarity has decreased, confirming the higher efficiency of AMRSR when the references are not similar to the LR input.
\\To further prove the efficiency of leveraging multiple references, we evaluate the RefSR approaches on the CU4REF dataset by swapping the references of an image with the ones of other images in the dataset. The quantitative results of Table~\ref{fig:aim} and the visual comparison in Figure~\ref{tab:sim} demonstrates the benefits of using multiple dissimilar references compared to a single one. AMRSR is able to find more similar patches within multiple references even if these are very dissimilar to the LR input. AMRSR output presents more aliasing and artefacts compared to when similar references are used, but it has a better quality as well as higher values of PSNR and SSIM than other approaches and than when fewer references are used (AMRSR 1ref, AMRSR 2ref).
\\\indent\textbf{Comparison with CIMR~\cite{cui2020towards}: }we compare AMRSR with CIMR~\cite{cui2020towards}, the other multi-reference super-resolution approach. We trained AMRSR following the settings in~\cite{cui2020towards} on CUFED5 dataset (13,761 images). We randomly associated $N_M$ references taken from Outdoor Scene (OST) dataset~\cite{wang2018recovering} to each LR input image.
In~\cite{cui2020towards} two testing settings are adopted: (i) test on content-similar references using single reference images with different similarity levels; (ii) test on content-indipendent references using a reference pool to select a subset of feature vectors from 300 reference images.
The PSNR and SSIM values obtained by testing AMRSR following the first setting are shown in Table~\ref{tbl:cimr}. The highest results of AMRSR for all the different levels of similarity prove that it outperforms CIMR when only a single reference is used.
Table~\ref{tbl:cimr2} illustrates the results obtained by evaluating AMRSR following the second setting. We randomly associated to the LR input images $N_M$ reference images taken from the OST dataset~\cite{wang2018recovering}. AMRSR outperforms CIMR also in the multiple reference case.
\subsection{Ablation study 1: Attention mapping}
We evaluate the effect of processing subvectors of the LR input in the similarity mapping by comparing the actual configuration of AMRSR with three others: (i) without any attention mapping (processing the whole input vector); (ii) with an attention mapping in the reference feature vectors (dividing them into subvectors); (iii) with an attention mapping in both the input and reference feature vectors.
The obtained values of PSNR and SSIM are shown in Table~\ref{tbl:attention}. The worst performance are achieved without any attention for most of the datasets.
The effectiveness of the input attention mapping is proved by both the quantitative and qualitative evaluation, with a significant boost in the performance and higher quality of the SR examples of Figure~\ref{fig:atten}.
\subsection{Ablation study 2: number of reference images}

\begin{table}[!]
\centering
\resizebox{1.\linewidth}{!}{\input{amrsr_arxive/image_files/attention}}
 \vspace{-1.0em}
 \captionof{figure}{Visual comparison of the results obtained by changing configurations of the first module of AMRSR.}
 \vspace{-1.5em}
 \label{fig:atten}
\end{table}
\begin{table*}[!]
 \centering
\resizebox{0.9\textwidth}{!}{\input{table/references}}
\vspace{-1em}
\caption{PSNR/SSIM values obtained with different numbers of reference images. The bottom row shows the figures for the second-best approaches.}
\vspace{-1.2em}
\label{tbl:ref}
\end{table*}
\begin{table*}[!h]
\centering
\resizebox{0.85\linewidth}{!}{\input{amrsr_arxive/image_files/reference}}
 \vspace{-1.0em}
 \captionof{figure}{Visual comparison of the results obtained by changing the number of references. The first is an example of HUMAP while the other is of Sun80.}
 \vspace{-1.2em}
 \label{fig:refere}
\end{table*}
A key contribution of our work is the transfer of high-quality textures from multiple references to increase the matching between similar LR input and HR reference features.
To prove this, we test AMRSR by changing the number of references. The results of using $N_M=1, 2, 4$ reference images are compared for the CU4REF and Sun80 datasets. $N_M=8$ references are also considered for HUMAP and GEMAP. Table~\ref{tbl:ref} presents the PSNR and SSIM figures for the different cases, including the second best results of related works ($2^{nd}$ best in the table).
Increasing the number of references leads to higher values of PSNR and SSIM. The highest ones are generally obtained with the maximum number of references. AMRSR outperforms the $2^{nd}$ best techniques also when a single reference is used.
Figure~\ref{fig:refere} confirms the advantage of using multiple references. The hairs of the human subject in the texture map example are more sharped when 4 references are used because they are transferred from the side of the model, which is not visible when only 1 or 2 references are exploited.
Similarly, the window grates are visible only when 4 references are used because the network learns their texture from the third reference (bottom left image in the reference image).
\subsection{Ablation study 3: Part-based mechanism and GPU memory consumption}
\label{ssec:gpu}
\begin{table*}[!]
 \centering
\resizebox{0.95\textwidth}{!}{\input{table/memory}}
\vspace{-1em}
\caption{PSNR/SSIM values, GPU memory usage and processing time of different configuration of AMRSR and the state-of-the-art RefSR approaches. $N_{HR}$ is the number of parts which the reference images are divided in.}
\vspace{-1.5em}
\label{tbl:memory}
\end{table*}
We finally evaluate the effect of the part-based mechanism of AMRSR by computing the required GPU memory of a Quadro RTX 8000 GPU (48 GB) during inference and comparing it with TTSR, SRNTT, AMRSR with different numbers of references and with different values of $N_{R}$. Changing the value of $N_{R}$ leads to a modification of the hierarchical structure: if $N_{R}=1$, the second level of the hierarchy is skipped.
The maximum size of the reference images is: (500x500) for CU4REF, (1024x928) for Sun80, (5184x3840) for HUMAP and (4032x6048) for GEMAP.
Table~\ref{tbl:memory} presents the required GPU memory, the inferring time and the PSNR and SSIM values. Higher is the size of the references, higher is the memory consumption but higher is also the quality of the images and thus, more details can be transferred from them to the SR output. AMRSR requires significant less memory than the single RefSR approaches also when multiple references are used. To test TTSR and SRNTT with the HUMAP and GEMAP datasets, the reference must be downscaled by a factor of 2 to consume less than 48GB.
For a fair comparison, we test our network with 1 and 4 downscaled references ("cut" in the table).
The comparison between different choices of $N_{R}$ shows that, when higher resolution references are exploited with higher values of $N_{R}$, the memory footprint is reduced and the PSNR and SSIM are increased, confirming the effectiveness of the part-based mechanism. For CU4REF, the best performances are achieved with $N_{R}=1$ because the size of its references is much lower. AMRSR is generally slower than the other approaches: the processing time increases when the number and the size of the references increase. The processing time of AMRSR is similar to the one of the other RefSR methods if a single reference is used.
\section{Conclusion}

In this paper we tackle the super-resolution problem with a multiple-reference super-resolution network that is able to transfer more plausible textures from several references to the SR output. Our network focuses the learning attention in the comparison between sub-vectors of the LR input and the reference feature vectors, achieving significant qualitative and quantitative improvements as demonstrated from the conducted experiments.
A hierarchical part-based mechanism is also introduced to remarkable reduce the GPU memory usage, which is extremely heavy if other RefSR approaches are applied with HR reference images.
The main limitation, whose improvement will be addressed as a future work, is the processing time.
In addition, we constructed three datasets to facilitate the research for multiple-reference super-resolution and for 3D appearance super-resolution.
\newpage
\newpage
{\small
\bibliographystyle{ieee_fullname}

\section{Introduction}
\noindent
Image super-resolution (SR) aims to estimate a perceptually plausible high-resolution (HR) image from a low-resolution (LR) input image ~\cite{yang2014single}. This problem is ill-posed due to the inherent information deficit between LR and HR images. Classic super-resolution image processing~\cite{pandey2018compendious} and deep learning based approaches~\cite{wang2020deep} result in visual artefacts for large up-scaling factors ($4\times$).
To overcome this limitation, recent research has introduced the sub-problem of reference image super-resolution (RefSR)~\cite{boominathan2014improving,yue2013landmark,zheng2018crossnet}. Given an input LR image and a similar HR reference image, RefSR approaches estimate a SR image. Reference super-resolution with a single reference image has been demonstrated to improve performances over general SR methods achieving large up-scaling with reduced visual artefacts.
\begin{table}[t]
\centering
\resizebox{1.\linewidth}{!}{\input{amrsr_arxive/image_files/intro2}}
 \vspace{-1.2em}
 \captionof{figure}{The proposed network exploits $N_M=4$ reference images (top left) to super-resolve the LR input (top right). Its SR output has the best visual quality compared to other state-of-the-art methods.}
 \vspace{-1.7em}
 \label{fig:intro}
\end{table}
\\In this paper we generalise reference super-resolution to use multiple reference images giving a pool of image features and propose a novel attention-based sampling approach to learn the perceptual similarity between reference features and the LR input.
The proposed attention-based multiple-reference super-resolution network (AMRSR) is designed to allow multiple HR reference images by introducing a hierarchical attention-based mapping of LR input feature subvectors into HR reference feature vectors, focusing the learning attention on the LR input. This allows training with multiple HR reference images which would not be possible with a naive extension of existing single-reference super-resolution methods without a significant increase in memory footprint.
Figure~\ref{fig:intro} qualitatively illustrates the performance of the proposed AMRSR approach against state-of-the-art single-image super-resolution (CSNLN~\cite{mei2020image}, RSRGAN~\cite{zhang2019ranksrgan}) and RefSR (SRNTT~\cite{ZhangSRNTT2019}) approaches. Given $N_M$ reference images, AMRSR produces a $4\times$ SR image which is perceptually plausible and has a similar level of detail to the ground-truth HR image.
The primary contributions of the AMRSR approach presented in this paper are:
\begin{itemize}[noitemsep]
    \item Generalisation of single reference super-resolution to multiple reference images whilst improving memory efficiency thanks to a part-based mechanism.
    \item Hierarchical attention-based adaptive sampling for perceptual similarity learning between low-resolution image features and multiple HR reference images.
    \item Improved quantitative and perceptual performance for image super-resolution compared with state-of-the-art single-image RefSR.
\end{itemize}
AMRSR is applied to both image and 3D model texture map SR where multiple HR reference images are available. The proposed method is evaluated on benchmark datasets and demonstrated to significantly improve performances.
We introduce 3 new multiple reference SR datasets which will be made available to benchmark future SR approaches.
\section{Related work}
\subsection{Single-image super-resolution (SISR)}
\noindent
A breakthrough in the SISR task was achieved when Dong \etal~\cite{dong2014learning} tackled the problem with a convolutional neural network (CNN). From this work, the application of deep learning progressively replaced classic SR computer vision methods~\cite{wang2020deep}. The pioneer work of Dong \etal~\cite{dong2014learning} belongs to a group of SR methods that use mean squared error (MSE) as their objective function. VDSR~\cite{kim2016accurate} shows the importance of a deep layer architecture while SRResNet~\cite{ledig2017photo} and EDSR~\cite{lim2017enhanced} demonstrate the benefit of using residual block~\cite{he2016deep} to alleviate the training. Several modifications of the residual structure such as skip connections~\cite{tong2017image}, recursive structures~\cite{tai2017image} and channel-attention~\cite{zhang2018image} further improved the accuracy of SISR. The state-of-the-art CSNLN~\cite{mei2020image} integrates a cross-scale non-local attention module to learn dependencies between the LR and HR images. Other works propose lightweight networks to alleviate computational cost~\cite{muqeet2020ultra,liu2020residual}.
These residual networks ignore the human perception and only aim to high values of PSNR and SSIM, producing blurry SR images~\cite{wang2020deep}. Generative adversarial networks (GANs), introduced in the SR task by Ledig \etal with SRGAN~\cite{ledig2017photo}, aim to enhance the perceptual quality of the SR images.
The performances of SRGAN were improved by ESRGAN~\cite{ledig2017photo}, which replaces the adversarial loss with a relativistic adversarial loss. RSRGAN~\cite{zhang2019ranksrgan} develops a rank-content loss by training a ranker to obtain state-of-the-art visual results.
\\\indent \textbf{3D appearance super-resolution: }There are only two deep learning works that super-resolve texture maps to enhance the appearance of 3D objects.
The method proposed by Li \etal~\cite{li2019_3dappearance} processes, with a modified version of EDSR~\cite{lim2017enhanced}, LR texture maps and their normal maps to incorporate geometric information of the model in the learning. The pre-process to create normal maps introduces heavy computational cost.
In the second work~\cite{richard2019learned}, a redundancy-based encoder generates a blurry texture map from LR images that is then deblurred by a SISR decoder. Its main objective is not the super-resolution but the creation of texture maps from a set of LR multi-view images.
\subsection{Reference-based super-resolution (RefSR)}
\noindent
GANs were introduced to solve the problems of the residual networks by focusing on the perceptual quality of the image. However, their generative nature leads to the creation of unnatural textures in the SR image. RefSR approaches were applied to eliminate these artefacts by learning more accurate details from reference images.
One of the first RefSR networks is CrossNet~\cite{zheng2018crossnet}, which uses optical flow to align input and reference, limiting the matching of long distance correspondences. CrossNet was improved with two-stage cross-scale warping modules, adding to the optical flow alignment a further warping stage~\cite{tan2020crossnet++}. The optical flow introduces artefacts when misaligned references must be handled.
The ``patch-match" approach correlates the reference and input images by matching similar patches. In an early, non deep learning framework~\cite{boominathan2014improving}, patches of downsampled reference image are matched with gradient features of the LR image. This work was adapted by Zheng \etal~\cite{zheng2017learning} to perform semantic matching as well as to synthesise SR features through a CNN. More recently, Zhang \etal proposed SRNTT~\cite{ZhangSRNTT2019}, which swaps the most similar features of the reference and the LR image through convolutional layers. TTSR~\cite{yang2020learning} refines the matching mechanism by selecting and transferring only relevant textures from the reference image. SSEN~\cite{shim2020robust} performs the patch-match through deformable convolution layers using an offset estimator. MASA~\cite{lu2021masa} adds a spatial adaptation module to handle large disparity in color or luminance distribution between reference and input images.
CIMR~\cite{cui2020towards} is the only method in the literature that exploits multiple references. It selects a smaller subset from all the features of generic reference images without performing any comparison with the LR input, neglecting similar textures of the references.
Our approach utilises a hierarchical patch-match method to search for relevant textures among all the feature vectors of multiple references, increasing the possibility to find more similar high-quality textures. It performs an attention-based similarity mapping between the references and subvectors of the LR input, improving performances. Finally, it significantly reduces the GPU usage of the patch-match approaches, facilitating the reproducibility of RefSR studies.
\section{Adaptive multi-reference super-resolution}
\noindent
In this section we present the proposed AMRSR network, designed to exploit multiple HR reference images for training and inference whilst maintaining memory efficiency. A hierarchical attention-based approach is introduced for image feature matching from the LR input to the HR reference images using a perceptual loss.
Hierarchical attention allows multiple HR reference images without a significant increase in GPU memory requirements and is demonstrated to improve performance versus a global similarity search (section \ref{ssec:gpu}).
The problem of multiple-reference super-resolution can be stated as follows:
given a LR input $I_{LR}$ and a set of HR reference images $\{I^m_{ref}\}_{m=1}^{N_{M}}$, estimate a spatially coherent SR output $I_{SR}$ with the structure of $I_{LR}$ and the appearance detail resolution of the multiple-reference images.
The SR output should contain perceptually plausible HR appearance detail without the introduction of visual artefacts such as blur, ringing or unnatural discontinuities observed with previous SR approaches for large up-scaling factors ($>2\times$).
\subsection{Overview of approach}
\begin{figure}[!]
    \centering
    \resizebox{1\linewidth}{!}{
    \includegraphics[width=0.85in]{amrsr_arxive/images/parte1_2_v2.pdf}}
    \vspace{-2.0em}
    \caption{AMRSR inference overview. Given a LR input image and $N_M$ reference images, AMRSR comprises three modules: feature extraction; hierarchical attention-based similarity; and image SR sampling. The resulting output is a HR reconstruction of the LR input image.}
    \vspace{-1.8em}
    \label{fig:network}
\end{figure}
\noindent
Figure \ref{fig:network} presents an overview of the proposed approach, which comprises the following stages.
\\
\textbf{Feature Extraction:} to reduce GPU memory consumption with multiple reference images, the LR input $I_{LR}$ and HR reference images $\{I^m_{ref}\}^{N_M}_{m=1}$ are divided into $N_I$ and $N_{R}$ sub-parts, respectively. Image features are extracted from these parts using a pre-trained VGG-19 network~\cite{simonyan2014very}.
\\
\textbf{Hierarchical Attention-based Similarity:} computes a mapping of features from the LR image to the most similar features of the HR reference images. The similarity $s_k$ is inferred between the feature vector of the LR input and of every reference image. The multiple references are then sampled based on the most similar features. This process is executed following a hierarchical structure of $l=N_L$ levels with an attention-based similarity mapping of the input feature vector. The output $O$ is a feature vector containing the most similar reference features to the input features.
\\
\textbf{Image Super-resolution:} given the feature similarity mapping $O$, a convolutional network super-resolves the LR input $I_{LR}$ to obtain the SR output $I_{SR}$ which maintains the spatial coherence of the input with the HR appearance detail of the reference images.
\begin{figure}[!]
    \centering
    \resizebox{0.85\linewidth}{!}{
    \includegraphics[width=1.in]{amrsr_arxive/images/new_methodmod2_v2.pdf}}
    \vspace{-1.em}
    \caption{Hierarchical attention-based similarity with $N_L=3$ levels.}
    \vspace{-1.8em}
    \label{fig:network2}
\end{figure}
\noindent
\\In contrast to the patch-match adopted by previous RefSR approaches \cite{ZhangSRNTT2019,yang2020learning,li2020survey,shim2020robust,lu2021masa}, AMRSR performs feature similarity matching between subvectors of the LR input and reference images to focus the learning attention on the input features. A feature vector contains $N$ features of the input image, each of which is a matrix of values that represents a specific image feature. Instead of processing the whole matrix, we divide it into submatrices and perform the similarity mapping with these.
This improves the learning of the most similar features in each submatrix giving improved performance (Table~\ref{tbl:attention}). We refer the reader to the supplementary material (Table 2) for an outline of the notation.
\subsection{Feature extraction}
\noindent
To conduct the similarity matching in the neural domain, feature vectors of the LR input and references must be retrieved. A problem of previous RefSR approaches is that the patch-match on HR reference images requires high GPU memory usage. To tackle this, we divide the input and reference images into $N_I$ and $N_{R}$ sub-parts, respectively. Feature vectors of these parts are extracted with a VGG-19 network~\cite{simonyan2014very} as shown in the second part of Figure~\ref{fig:network}. The feature vectors of each part of the input are divided into $N_C$ subvectors $\{\{\phi^c_i(I_{LR})\}_{i=1}^{N_I}\}_{c=1}^{N_C}$ to perform an attention-based similarity on the LR input. For simplicity, we assume $N_I=1$ (the input image is not divided into parts) and we express the set as $\phi^c(I_{LR})$ without loss of generality. If $N_I>1$, the algorithm is repeated for each part and the outputs are concatenated. $N_M \times N_R$ feature vectors are retrieved for the references $\{\{\phi^r(I_{ref}^m)\}_{m=1}^{N_M}\}_{r=1}^{N_{R}}$ (expressed as $\phi^r(I_{ref}^m)$). Dividing the input and reference images into parts and inferring the similarity between them in a hierarchical order, establishes an efficient mechanism that improves performances and reduces GPU memory requirements. This is important for practical implementation of multi-reference SR within a fixed GPU memory size.
\subsection{Hierarchical attention-based similarity}
\noindent
The objective of this stage is to map the features of the LR input to the most similar features of the HR reference images. The output is a feature vector that contains the values of these most similar reference features.
A hierarchical approach of similarity mapping is performed over $l=N_L$ levels.
For every level $l$ of the hierarchy, a similarity map between LR input subvectors and reference features is computed. The most similar features are then retrieved considering the maximum values of the similarity map. A new feature vector is created with these features and used to compute the similarity map and the feature vector in the next level of the hierarchy.
\\
The similarity map $s^l_k$ for level $l$ is evaluated by convolution between the subvectors $\phi^c(I_{LR})$ of the LR input and $O^{l-1,r,m}_{ref}$, which is either the input reference feature vectors $\phi^r(I^m_{ref})$ if $l=1$ or new vectors created in the level $l=l-1$ (which contain features of the references $\{I^m_{ref}\}$):
\begin{equation}
\label{eq:1}
    s^{l}_{k}=\phi^c(I_{LR})*\frac {P_{k}(O^{l-1,r,m}_{ref})}{||P_{k}(O^{l-1,r,m}_{ref})||} \tag{1}
\end{equation}
$k=c$ if $l=1$, $k=r$ or $k=m$ otherwise. $P$ is the patch derived from the application of the patch-match approach: patches of $O^{l-1,r,m}_{ref}$ are convoluted with $\phi^c(I_{LR})$ to compute the similarity.
\\
When the similarity map $s^{l}_{k}$ is evaluated, a vector $O^{l}_{ref}$ containing the most similar features of $O^{l-1}_{ref}$ is created by applying either one of two distinct approaches:
\begin{enumerate}[noitemsep]
    \item \textbf{Input attention mapping} ($l=1$): in the first level a feature vector is created by maximising over every subvector of the input:
    \begin{gather}
        \label{eq:2}
        O^{1,r,m}_{ref}(x,y)=P_{k^*}(\phi^r(I^m_{ref}))(x,y) \tag{2}\\ k^*=\argmax_{k=c}s^{1}_{k}(x,y)\notag
    \end{gather}
    $O^{1,r,m}_{ref}(x,y)$ represents a single value in the $(x,y)$ position of the created feature vector $O^{1,r,m}_{ref}$. This value corresponds to the $(x,y)$ value of the $k^*$ patch $P(\phi^r(I^m_{ref}))$ whose $s^1$ is the highest among all the similarity values $s^{1}_{k}(x,y)$ for each subvector of the LR input feature vector.
    \item \textbf{Reference attention mapping ($l>1$)}: for subsequent levels of the hierarchy, a feature vector is created by maximising a new similarity $s^{l}_{k}$ map over the feature vector created in the previous level.
    \begin{gather}
        \label{eq:3}
        O^{l,k}_{ref}(x,y)=O^{l-1,k^*}_{ref}(x,y) \tag{3}\\
        k^*=\argmax_{k}s^{l}_{k}(x,y)\notag
    \end{gather}
    $k=r$ or $ k=m$ depending on which level is processed. The value of $O^{l,k}_{ref}$ in the $(x,y)$ position is the value of $O^{l-1,k}_{ref}$ with the highest $s^{l}$ among all the $s^{l}_{k}(x,y)$ of  $O^{l-1,k}_{ref}$.
\end{enumerate}
Mapping is repeated at multi-scales with three feature extractor levels to achieve robustness to the variance of colour and illumination~\cite{ZhangSRNTT2019}.
The final output, obtained when the similarity mapping is performed for all the levels of the hierarchy, is a feature vector $O=O^{N_L}_{ref}$ which contains the features of the reference images that are most similar to every feature of the LR input.
When the final level of the hierarchy is processed, $N_K$ sets of weights $W_k$ are computed as the maximum of the scalar product between $\phi^c(I_{LR})$ and the $O^{l-1,k}_{ref}$ vector produced in the previous level.
\begin{equation}
\label{eq:4}
    W_k=max(\phi^c(I_{LR}) \cdot O^{l-1,k}_{ref})\tag{4}
\end{equation}
The final set of weights $W$ is then retrieved from these sets: the weight in position $(x,y)$ of $W$ has the same value of the weight in position $(x,y)$ of the $k^*$-th set $W_{k^*}$ with $k^*$ from Equation~\ref{eq:3} with $l=N_L$: $W(x,y)=W_{k^*}(x,y)$.
\\
For $N_L=3$ levels of hierarchy as shown in Figure~\ref{fig:network2}:
   \\ $\bf l=1$: feature similarity mapping between every subvector of the input vector and every part of every reference. \textit{Input:} $\phi^c(I_{LR})$, $\phi^r(I_{ref}^m)$, $k=c$. \textit{Output:} $\{\{O^{1,m,r}_{ref}\}_{m=1}^{N_{M}}\}_{r=1}^{N_{R}}$.
    \\$\bf l=2$: feature similarity mapping between the input subvectors and all the $N_R$ parts of a single reference, repeated for every reference. \textit{Input:} $\phi^c(I_{LR})$, $\{\{O^{1,m,r}_{ref}\}_{m=1}^{N_{M}}\}_{r=1}^{N_{R}}$, $k=r$.  \textit{Output:} $\{O^{2,m}_{ref}\}_{m=1}^{N_{M}}$.
    \\$\bf l=3$: feature similarity mapping between the input subvectors and all the references. \textit{Input:} $\phi^c(I_{LR})$, $\{O^{2,m}_{ref}\}_{m=1}^{N_{M}}$, $k=m$. \textit{Output}: $O=O^3_{ref}$.
\subsection{Image super-resolution}
\noindent
In the last stage, $I_{LR}$ is super-resolved with a generative network that exploits the information of the vectors obtained with the hierarchical similarity mapping whilst maintaining the spatial coherence of $I_{LR}$. These vectors are embedded to the input feature vector through channel-wise concatenation in different layers $h$ of the network.
We modified the architecture of the generator used by Zhang \etal~\cite{ZhangSRNTT2019} by eliminating the batch normalization layers since they can reduce the accuracy for dense pixel value predictions~\cite{yu2018wide}. More details are explained in the supplementary material.
A texture loss is defined to enforce the effect of the texture swapping between $I_{LR}$ and the obtained $O$:
\begin{equation}
\label{eq:5}
    L_{tex}=\sum_{h}||Gr(\phi_h(I_{SR}\cdot W_h))-Gr(O_h\cdot W_h))|| \tag{5}
\end{equation}
where $Gr(\cdot)$ computes the Gram matrix.
Differently from~\cite{ZhangSRNTT2019}, the weighting map $W_h$ is computed among the $N_M$ references. The weight of HR image features more similar to $I_{LR}$ will be higher. Thus the appearance transfer from $\{I_{ref}\}$ to $I_{SR}$ is adaptively enforced based on the references similarity. In addition, the network minimises:
\begin{itemize}[noitemsep]
\item The adversarial loss $L_{adv}$ to enhance the visual quality of the SR output. To stabilize the training, we use the WGAN-GP~\cite{gulrajani2017improved} for its gradient penalization feature.
\item The reconstruction $L_1$ loss, since it has been demonstrated to give sharper performance than $L_2$ loss~\cite{yang2020learning}.
\item The perceptual loss~\cite{johnson2016perceptual} $L_{p}$ to enhance the similarity between the prediction and the target in feature space.
\end{itemize}
\section{Dataset}
\label{sec:dataset}
\noindent
To the best of our knowledge, no multi-reference benchmark datasets are available (only with a single reference~\cite{ZhangSRNTT2019}).
To achieve our objective of multi-reference SR, we introduce three datasets:
\vspace{0.05in}
\\\textbf{1. CU4REF}: this dataset is built from the single reference dataset CUFED5~\cite{ZhangSRNTT2019}. 4 groups of images are defined from the CUFED dataset~\cite{wang2016event}, each with a different similarity level from the LR input images. We use the images in these groups as our references. The training set contains 3957 groups of LR and reference images while the testing set contains 126 groups (4 references for every LR image).
\\\textbf{2. HUMAP:} to create the references of 67 synthetic human texture maps downloaded from several websites \cite{renderpeople,cgtrader,turbosquid,free3d}, we import their 3D models in Blender~\cite{blender} and render 8 camera views as reference images for each subject. Two real human texture maps retrieved from 16 multi-view images are added. Due to the low amount of data, we augment the dataset by cropping the texture maps into patches (256x256 size)~\cite{li2019_3dappearance}.
The training dataset consists of 5505 groups of patches and references. The testing dataset comprises 336 groups created from 5 texture maps of 6 subjects (3 captured by 16 video-cameras, 2 using a 5x5 FLIR Grasshooper3 camera array) and a texture map of 2 people~\cite{collet2015high}.
\\\textbf{3. GEMAP:} consists of generic LR texture maps associated with 8 references. The texture maps are taken from the 3DASR dataset~\cite{li2019_3dappearance}, created from the multi-view images and 3D point clouds from other datasets (\cite{ley2016syb3r},\cite{seitz2006comparison},\cite{schops2017multi},\cite{goldlucke2014super},\cite{zhou2014color},\cite{zollhofer2015shading}). The reference images for the texture maps of~\cite{ley2016syb3r} are created with the same approach applied for HUMAP. For the other texture maps, the HR multi-view images captured by DSLR cameras are taken as references. The LR texture maps are cropped as in HUMAP. The training dataset contains 2032 groups and for testing 290 groups.
    \vspace{-0.7em}
\\To evaluate the generalization capability of AMRSR on RGB images, we test it on Sun80 dataset~\cite{sun2012super}, which has 80 natural images accompanied by a series of web-search references that significantly differ from the input images.
\section{Results and evaluation}
\label{sec:exp}
\noindent
This section illustrates how AMRSR outperforms other state-of-the-art methods with quantitative and qualitative comparisons. Two ablation studies on the network configuration and on the advantage of multiple references are then presented. The GPU memory requirement of the state-of-the-art RefSR approaches is compared confirming the efficiency of AMRSR. The LR inputs are obtained by bicubic downscaling ($4\times$) from their ground-truth HR images and the SR results are evaluated on PSNR and SSIM on the Y channel of YCbCr space. AMRSR parameters are: $N_M=4$, $N_I=1$, $N_R=1$ for CU4REF dataset, $N_R=16$ for all others (see Section~\ref{ssec:gpu}). To integrate the input subvectors with the architecture structure, the value of $N_C$ is:
\begin{equation}
    \label{eq:6}
    N_C(x)=\frac{length(\phi(I_{LR})(relu3\_1))}{length(\phi(I_{LR})(reluq\_1)_{q=1}^{3})/4}\tag{6}
\end{equation}
where $q$ indicates the three different layers used for the multi-scale fashion approach.
Further evaluations and results are presented in the supplementary material.
\subsection{Comparison with state-of-the-art approaches}
\begin{table}[!]
 \centering
\resizebox{0.45\textwidth}{!}{\input{amrsr_arxive/table/SR2_new_v2}}
\vspace{-0.8em}
\caption{PSNR/SSIM values of different SR approaches. * indicates that the references are downscaled of a factor of 2 (see Section\ref{ssec:gpu}).}
\vspace{-1.7em}
\label{tbl:quant}
\end{table}
\begin{table*}[!h]
\centering
\resizebox{0.85\linewidth}{!}{\input{amrsr_arxive/image_files/comparison}}
 \vspace{-1.1em}
 \captionof{figure}{Qualitative comparison among the state-of-the-art SR approaches on CU4REF (first two examples), Sun80 (third and fourth), GEMAP (fifth) and HUMAP (last). The references are shown for each example. The top left or the most left reference was used in the single-reference SR approaches.}
 \vspace{-1.8em}
 \label{fig:qualit}
\end{table*}
\noindent
Qualitative and quantitative comparisons are performed with state-of-the-art SISR and RefSR approaches. The SISR methods are the PSNR-oriented EDSR~\cite{lim2017enhanced}, MDSR~\cite{lim2017enhanced}, RRDBNet~\cite{wang2018esrgan}, SRResNet~\cite{ledig2017photo}, RCAN~\cite{zhang2018image}, NHR~\cite{li2019_3dappearance}, NLR~\cite{li2019_3dappearance}, CSNLN~\cite{mei2020image}, MAFFSRN~\cite{muqeet2020ultra} and the visual-oriented SRGAN~\cite{ledig2017photo}, ESRGAN~\cite{wang2018esrgan}, RSRGAN~\cite{zhang2019ranksrgan}. The RefSR approaches are CrossNet~\cite{zheng2018crossnet}, SSEN~\cite{shim2020robust}, SRNTT~\cite{ZhangSRNTT2019}, TTSR~\cite{yang2020learning} and MASA~\cite{lu2021masa} (published June '21).
We train each network with the datasets presented in Section~\ref{sec:dataset} with the same training configurations. Training with adversarial loss usually deteriorates the quantitative results. For a fair comparison with the PSNR-oriented methods, we train our model, SRNTT, MASA and TTSR only on reconstruction loss (named with the suffix ``\textit{$l_2$}").
NHR and NLR are tested on HUMAP and GEMAP since they require normal maps (retrieved with Blender~\cite{blender}).
\\\indent\textbf{Quantitative comparison: }the PSNR and SSIM values of each method are presented in Table~\ref{tbl:quant}, which is divided into two parts: PSNR-oriented networks in the upper part; visual-oriented GANs and RefSR in the lower part. The highest scores are highlighted in red while the second highest scores are blue. The bold red figures are the highest across both PSNR- and visual-oriented methods. AMRSR-\textit{$l_2$} achieves the highest values of PSNR and SSIM in the PSNR- and visual-oriented methods for all four datasets.
\\\indent\textbf{Qualitative comparison: }
Figure~\ref{fig:qualit} shows SR examples of the most relevant approaches considered in our evaluation. The SR outputs produced by the PSNR-oriented methods (RCAN, CSNLN) are blurrier and the details are less sharp. The results produced by the visual-oriented approaches (RSRGAN, SRNTT, TTSR) present unpleasant artefacts such as ringing and unnatural discontinuities.
The SR outputs of AMRSR are less blurry with sharper and finer details as shown in the zoomed patches of the examples, whose quality is higher than the other results.
\\\indent\textbf{User study evaluation: }
to further evaluate the visual quality of the SR outputs, we conduct a user study, where AMRSR is compared with five approaches.
50 random SR output pairs were shown to 60 subjects. Each pair consists of an image of AMRSR and the counterpart generated by one of the other approaches. The users were asked to pick the image with the best visual quality. The values on the Y-axis of Figure~\ref{fig:chart} illustrate the percentage of the users that select AMRSR outputs. AMRSR significantly outperforms the other methods with over 90\% of users voting for it.
\begin{figure}[!]
\centering
\resizebox{0.8\linewidth}{!}{\input{amrsr_arxive/image_files/chart}}
\vspace{-1.em}
\caption{Percentage of users that prefer AMRSR over 5 approaches in the user study. The error bars indicate the $95\%$ confidence interval.}
\vspace{-1.em}
\label{fig:chart}
\end{figure}
\begin{table}[!]
    \centering
    \resizebox{0.45\textwidth}{!}{\input{amrsr_arxive/table/simlevel_v2}}
    \vspace{-0.8em}
    \caption{Quantitative comparison between AMRSR and RefSR methods using single reference with different levels of similarity to the LR input.}
     \vspace{-2.0em}
    \label{tab:difref}
\end{table}
\\\indent\textbf{Influence of dissimilar references: }similarity between LR and reference images significantly influences the performances of RefSR methods~\cite{ZhangSRNTT2019}. Following the setting in~\cite{ZhangSRNTT2019}, we evaluate the effect of dissimilar reference images to the LR input testing on four similarity levels (from L1 to L4) of the CUFED5 test set, defined by computing the number of best matches of SIFT features between the input image and the references. The references that belong to L1 have the highest number of matches while the L4 ones have the lowest number. The highest figures of PSNR and SSIM (Table~\ref{tab:difref}) are obtained by our network when a single reference is used even though its level of similarity has decreased, confirming the higher efficiency of AMRSR when the references are not similar to the LR input. To show the efficiency of leveraging multiple references, we evaluate the RefSR approaches on the CU4REF dataset by swapping the references of an image with other images in the dataset. The quantitative results of Table~\ref{fig:aim} and the visual comparison in Figure~\ref{tab:sim} demonstrate the benefits of using multiple dissimilar references. AMRSR is able to find more similar patches within multiple references even if these are very dissimilar to the LR input. For dissimilar references, AMRSR achieves higher PSNR and SSIM than other methods and than when a single reference is used (AMRSR 1). This study demonstrates the performance of adaptive sampling of AMRSR even for references with dissimilar features.
\\\indent\textbf{Comparison with CIMR~\cite{cui2020towards}: } we compare AMRSR with CIMR~\cite{cui2020towards}, the only other multi-reference super-resolution approach. AMRSR is trained following the setting in~\cite{cui2020towards} on CUFED5 dataset (13,761 images). We randomly associated $N_M$ references taken from Outdoor Scene (OST) dataset~\cite{wang2018recovering} to each LR input image.
CIMR is evaluated on content-independent references using a reference pool to select a subset of feature vectors from 300 reference images. We evaluate AMRSR by randomly associating to the LR input images $N_M$ reference images from the 300 images.
Results presented in Table~\ref{tbl:cimr2} show that AMRSR outperforms CIMR in the multiple references case.
\begin{table}[!]
    \centering
    \resizebox{0.48\textwidth}{!}{\input{amrsr_arxive/table/sim2_v2}}
    \vspace{-1.0em}
    \caption{PSNR/SSIM values with references dissimilar to the LR input.}
    \vspace{-1.0em}
    \label{tab:sim}
\end{table}
\begin{table}[!]
\centering
\resizebox{0.9\linewidth}{!}{\input{amrsr_arxive/image_files/sim}}
 \vspace{-1.2em}
 \captionof{figure}{Visual comparison of SR outputs obtained with multiple references dissimilar to the LR input.}
 \vspace{-1.0em}
 \label{fig:aim}
\end{table}
\begin{table}[!]
    \centering
    \resizebox{0.45\textwidth}{!}{\input{amrsr_arxive/table/cimr2}}
    \vspace{-0.8em}
    \caption{Quantitative comparison between AMRSR and CIMR exploiting multiple reference images. The results of CIMR are taken from~\cite{cui2020towards}.}
    \vspace{-1.7em}
    \label{tbl:cimr2}
\end{table}
\subsection{Ablation studies}
\label{ssec:gpu}
\indent\textbf{Number of reference images: }a key contribution of our work is the transfer of high-quality textures from multiple references to increase the matching between similar LR input and HR reference features.
To prove this, we test AMRSR by changing the number of references. The results of using $N_M=1, 2, 4$ reference images are compared for CU4REF and Sun80 datasets. $N_M=8$ references are also considered for HUMAP and GEMAP. Table~\ref{tbl:ref} presents the PSNR and SSIM figures for the different cases, including the second best results of related works (``$2^{nd}$ best").
Increasing the number of references leads to higher values of PSNR and SSIM. The highest ones are generally obtained with the maximum number of references. AMRSR outperforms the $2^{nd}$ best techniques also when a single reference is used.
Figure~\ref{fig:refere} confirms the advantage of using multiple references. The hairs of the human subject in the texture map example are sharper when 4 references are used because they are transferred from the side of the model, which is not visible with 1 or 2 references.
Similarly, the window grates are neat only when 4 references are used because the network learns their texture from the third reference.
\indent\textbf{Attention mapping: }
we evaluate the effect of processing subvectors of the LR input in the similarity mapping by comparing the actual configuration of AMRSR with three others: (i) without any attention mapping (processing the whole input vector); (ii) with an attention mapping in the reference feature vectors (dividing them into subvectors); (iii) with an attention mapping in both the input and reference feature vectors.
The obtained values of PSNR and SSIM are shown in Table~\ref{tbl:attention}.
The effectiveness of the input attention mapping is proved by both the quantitative and qualitative evaluation, with a significant boost in the performance and higher quality of the SR examples of Figure~\ref{fig:atten}.
\begin{table}[!]
 \centering
\resizebox{0.45\textwidth}{!}{\input{amrsr_arxive/table/reference_psnr2}}
\vspace{-0.8em}
\caption{PSNR/SSIM values obtained with different numbers of reference images. The bottom row shows the figures for the second-best approaches.}
\vspace{-1.0em}
\label{tbl:ref}
\end{table}
\begin{table}[!]
 \centering
\resizebox{0.45\textwidth}{!}{\input{amrsr_arxive/table/attention_psnr2}}
\vspace{-0.8em}
\caption{PSNR/SSIM values of different configurations of AMRSR obtained by dividing into subvectors the feature vectors of references (ref), of both reference and input (both) or none (no).}
\vspace{-1.0em}
\label{tbl:attention}
\end{table}
\begin{table}[!]
\centering
\resizebox{1.\linewidth}{!}{\input{amrsr_arxive/image_files/attention}}
 \vspace{-1.2em}
 \captionof{figure}{Visual comparison of the results obtained by changing attention.}
 \vspace{-2.0em}
 \label{fig:atten}
\end{table}
\\\indent\textbf{Part-based mechanism and GPU memory usage: }
we finally evaluate the effect of the part-based mechanism of AMRSR by computing the required GPU memory of a Quadro RTX 8000 GPU (48 GB) during inference and comparing it with TTSR, SRNTT, MASA, AMRSR with different numbers of references and with different values of $N_{R}$ (Table~\ref{tbl:memory}). Changing the value of $N_{R}$ leads to a modification of the hierarchical structure: if $N_{R}=1$, the $2^{nd}$ level of the hierarchy is skipped. The maximum size of the reference images is: (500x500) for CU4REF, (1024x928) for Sun80, (5184x3840) for HUMAP and (4032x6048) for GEMAP.
Increase in the reference image size results in increased memory consumption and improve super-resolution quality. AMRSR requires significantly less memory than the single RefSR approaches when multiple references are used. To test TTSR, SRNTT and MASA with HUMAP and GEMAP datasets, the reference must be downscaled ($2\times$) to consume less than 48GB. For a fair comparison, we test our network with 1 and 4 downscaled references (``cut" in the table). The comparison between different choices of $N_{R}$ shows that, when higher resolution references are exploited with higher values of $N_{R}$, the memory footprint is reduced and the PSNR and SSIM are increased. For CU4REF, the best performances are achieved with $N_{R}=1$ because the size of its references is much lower.
\section{Conclusion}
\begin{table}[!]
\centering
\resizebox{1\linewidth}{!}{\input{amrsr_arxive/image_files/reference2}}
 \vspace{-1.2em}
 \captionof{figure}{Visual comparison of the results obtained by changing the number of references. The first example is of HUMAP, the other is of Sun80.}
 \vspace{-1.0em}
 \label{fig:refere}
\end{table}
\begin{table}[!]
 \centering
\resizebox{0.48\textwidth}{!}{\input{amrsr_arxive/table/memory3_v2}}
\vspace{-0.8em}
\caption{PSNR/SSIM values and GPU memory usage (in GB) of different configurations of AMRSR and the state-of-the-art RefSR approaches. $N_{R}$ is the number of parts which the reference images are divided into.}
\vspace{-2em}
\label{tbl:memory}
\end{table}
\noindent
In this paper, we tackle the super-resolution problem with a multiple-reference super-resolution network that is able to transfer more plausible textures from several references to the super-resolution output. Our network focuses the learning attention in the comparison between subvectors of the low-resolution input and the reference feature vectors, achieving significant qualitative and quantitative improvements as demonstrated from the evaluation. A hierarchical part-based mechanism is introduced to reduce the GPU memory usage, which is prohibitive if previous RefSR methods are applied with high-resolution reference images. In addition, we introduce 3 datasets to facilitate the research for multiple-reference and 3D appearance super-resolution.
\\
\noindent{\bf Acknowledgement:} This research was supported by UKRI EPSRC Platform Grant EP/P022529/1.

{\small
\bibliographystyle{ieee_fullname}