\section{Introduction}\label{sec:introduction}

Face super-resolution (SR), also known as face hallucination, refers to generating high resolution (HR) face images from the corresponding low resolution (LR) inputs. Since there exist many low resolution face images (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot, faces in surveillance videos ) and face analysis algorithms (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot, face recognition) often perform poorly on such images, there is a growing interest in face SR.

Different from general image SR, face SR places its emphasis on the recovery of \blank{the} key \blank{face} structures (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, shapes of \blank{face} components and face outline). These structures only account for a very small portion of the image, but are often more difficult to recover as they exhibit larger pixel variations. Training a deep neural network with the commonly used mean square error (MSE) loss, which weights pixels equally, is not very effective in recovering these ``sparse'' structures. \blank{Previous} works \cite{Chen2018CVPR,bulat2017super,yu2018face} proposed incorporating additional task, such as face parsing and landmark detection, to assist the training of face SR networks. \cite{Chen2018CVPR,yu2018face} also used predicted face priors to help face SR. Although joint training with these additional tasks helps enhance the importance of \blank{the} key \blank{face} structures, there are two major drawbacks\blank{, namely} (1) extra effort is needed to label the data for the additional \blank{task,} and (2) predicting face prior from LR inputs is \blank{itself also} a difficult problem.

On the other hand, if we subdivide a face image into many small regions and consider each region as an individual sample, the unbalanced distribution between regions containing key \blank{face} structures (referred to as hard regions) and those not containing key \blank{face} structures (referred to as easy regions) will \blank{resemble} the imbalance between foreground and background samples in object detection. This suggests that we may adopt some techniques similar to {\em bootstrapping} or {\em online hard example mining} (OHEM)~\cite{shrivastavaCVPR16ohem} in object detection to solve our face SR problem.

\begin{figure}[t]
  \includegraphics[width=.99\linewidth]{Fig_intro_1.png}
  \caption{\blank{Super-resolution} result \blank{produced by} SPARNetHD for an old photo of Marie Curie from Solvay conference 1927. Please zoom in to see details.}
  \label{fig:intro_1}
\end{figure}

In this paper, we introduce a carefully designed \blank{Face Attention Unit (FAU)} to construct a \blank{SPatial Attention Residual Network} (SPARNet) for face SR. The key idea is to bootstrap features related to the key \blank{face} structures \blank{using} a 2D spatial attention map. Instead of hard selection, the spatial attention map assigns a score between $0$ and $1$ to each spatial location of the feature map. This allows learning \blank{the prediction of} the spatial attention map through gradient descent. Spatial attention maps in different FAUs of the network \blank{can learn} to focus on different \blank{face} structures. For example, attention maps in \blank{deeper layers} focus more on coarse structures such as eyes and mouth, while those in \blank{shallower layers} focus more on detailed textures such as hairs. Considering that most \blank{of the} existing face SR methods can only \blank{produce} $128\times128$ outputs, we further extend SPARNet, referred to as SPARNetHD, to generate \blank{high resolution outputs (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, $512\times512$)}. Specifically, we enlarge the output resolution of SPARNet from \blank{$128\times128$ to $512\times512$} and adopt \blank{a} multi-scale discriminator loss similar to Pix2PixHD~\cite{wang2018pix2pixHD} to generate more realistic textures. Experiments show that SPARNetHD trained with synthetic LR data is pretty robust with natural LR inputs, while models without the proposed spatial attention mechanism produce undesirable artifacts. We show an example result of SPARNetHD on an old photo in \blank{Fig.~\ref{fig:intro_1}}. We can see that \blank{SPARNetHD} can restore key \blank{face} components very well and also generate high resolution and realistic textures.

\blank{The} key contributions of this paper can be summarized as follows:

\begin{enumerate}
  \item We propose an efficient framework \blank{named} SPARNet for face super-resolution. Without relying on any extra supervisions (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot, face parsing maps and landmarks), it achieves state-of-the-art performance on various kinds of metrics, including PSNR, SSIM, identity similarity, and landmark detection.
  \item We show that the proposed FAU\blank{, the basic building block of SPARNet,} can bootstrap the key \blank{face} structures (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, \blank{face} components and face outline) and significantly improve the performance of face super-resolution.
  \item By repeating FAUs in SPARNet, the spatial attention maps in different FAUs can learn to focus on different \blank{face} structures and further \blank{improve} the performance of SPARNet.
  \item We introduce SPARNetHD to \blank{generate} high resolution \blank{face} images (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, $512\times512$), and the model trained with synthetic data works \blank{equally} well on natural LR images.
\end{enumerate}

\begin{figure*}[t]
  \centering
  \captionsetup{justification=centering}
  \includegraphics[width=1.\linewidth]{Fig_arch.pdf}
  \caption{Architecture of the proposed SPatial Attention Residual Network (SPARNet).}
  \label{fig:arch}
\end{figure*}

\section{Related Works}

In this section, we briefly review the literature on face super-resolution and recent attention neural networks.

\subsection{Face Super-Resolution}
Face hallucination was pioneered by Baker and Kanade \cite{baker2002limits}, who showed that it is possible to perform high magnification SR for images of a specific category (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot, face and text). Since then many methods were proposed to improve the performance of face hallucination. They can be roughly classified into sub-space based methods \cite{wang2005hallucinating,liu2007face,ma2010hallucinating,liu2001two,liu2005hallucinating} and component based methods \cite{tappen2012bayesian,yang2013structured,song-ijcai17-faceSR}. Sub-space based methods usually \blank{rely} on Principle Component Analysis (PCA), which requires precisely aligned faces. Component based methods require detecting facial landmarks, which is difficult for LR face images. Both kinds of methods fail to produce satisfactory results for face SR with a high upscale factor.

Recently, deep convolutional neural networks (CNNs) have brought remarkable progress to face SR. Zhu \emph{et al}\onedot \cite{zhu2016deep} proposed a cascaded two-branch network to optimize face hallucination and dense correspondence field estimation in a unified framework. Yu \emph{et al}\onedot exploited generative adversarial networks (GAN) \cite{goodfellow2014generative} to directly super-resolve the LR inputs. They further improved their model to handle unaligned faces \cite{yu2017face}, noisy faces \cite{yu2017hallucinating}, and faces with different attributes \cite{yusuper}. Instead of directly inferring HR face images, Huang \emph{et al}\onedot proposed to predict wavelet coefficients from LR images to reconstruct HR images. Latest works employed extra \blank{face} prior supervisions, such as face parsing maps \cite{Chen2018CVPR}, landmark heatmaps \cite{bulat2017super,yu2018face}, and identity information \cite{zhang2018super}, to train their networks. Kim \emph{et al}\onedot \cite{pfsrnet} proposed a facial attention loss which focuses the network on the landmark region. Ma \emph{et al}\onedot \cite{dicnet} introduced an iterative method which predicts SR results and landmarks iteratively. Although extra supervisions help to improve the performance, there are two shortcomings\blank{, namely} (1) extra effort is needed to label the data, and (2) it is \blank{an} indirect way to \blank{direct} the SR network \blank{towards the key face} structures. Different from these methods, our SPARNet learns \blank{to predict} spatial attention maps to bootstrap key \blank{face} structures and \blank{other} feature-rich regions.

\subsection{Attention Networks}
Attention mechanism has been widely used in high level vision tasks, such as image classification \cite{mnih2014recurrent,hu2018senet,wang2017residual,woo2018cbam}, image captioning \cite{xu2015show,chen2017sca}\blank{,} and visual question answering \cite{schwartz2017high,yu2017multi}. The key idea is to \blank{reweight} features using a score map to emphasize \blank{important features and suppress less useful ones} \cite{hu2018senet}. Wang \emph{et al}\onedot \cite{wang2017residual} introduced a trunk-and-mask attention mechanism to a residual network for image classification. He \emph{et al}\onedot \cite{hu2018senet} proposed the Squeeze-and-Excitation network which
\blank{employs} a channel-wise attention mechanism and \blank{demonstrates} significant performance improvements. Woo \emph{et al}\onedot proposed a convolutional block attention module (CBAM) which sequentially infers attention maps along the channel and spatial dimensions separately. Attention mechanism has also been employed in image generation tasks recently. Zhang \emph{et al}\onedot \cite{zhang2018rcan} combined channel attention with a very deep residual network for image SR. Cao \emph{et al}\onedot \cite{Cao2017Attention} proposed an attention aware face SR framework based on reinforcement learning, which sequentially attends to, crops out\blank{,} and super-resolves a patch.

Although closely related, our work is different from \cite{zhang2018rcan} and \cite{Cao2017Attention} in several aspects. First, attention mechanisms in high level tasks \blank{often} employ pooling for extracting semantic information. For face SR, however, this may \blank{result the loss of} important low and middle level features such as edges and shapes. In contrast, our attention mechanism \blank{is designed to take} advantages of multi-scale features. Second, \blank{unlike \cite{zhang2018rcan} which utilizes channel attention, our work considers spatial attention which facilitates region based attention of the key face components.}
Third, \blank{unlike \cite{Cao2017Attention} which takes a patch based approach, we generate spatial attention maps for the entire face.} This allows our network to have a global view of the \blank{face} structures and benefit from the contextual information.

\subsection{High Resolution Image Generation with GAN.} Recently, GAN has demonstrated to be very effective in generating HR images. Wang \emph{et al}\onedot \cite{wang2018pix2pixHD} proposed Pix2PixHD and used a multi-scale generator and discriminator architecture for HR image generation. Karras \emph{et al}\onedot \cite{karras2017progressive,karras2019style} proposed a new progressive training methodology for GAN to generate HR face images from random input vectors, and improved the quality of synthesized faces in \cite{karras2019analyzing}. Park \emph{et al}\onedot \cite{park2019SPADE} proposed the spatially-adaptive normalization layer for generating photo-realistic images given an input semantic layout. We follow the idea of Pix2PixHD and use \blank{a} multi-scale discriminator to generate \blank{high resolution} results. Different from Pix2PixHD, our SPARNetHD does not require progressive training from LR to HR images\blank{,} and our results are much better than Pix2PixHD with the help of the proposed spatial attention mechanism.

\section{SPARNet for Face SR}

\subsection{Overview} \label{sec:overview}
As shown in Fig.~\ref{fig:arch}, our SPARNet consists of three modules, namely the downscale module, the feature extraction module\blank{,} and the upscale module. Each of these modules is composed of a stack of FAUs (see \ref{sec:FAU} for details). Let
$I_{LR} \in \mathbb{R}^{3\times H'\times W'}$, $I_{SR} \in \mathbb{R}^{3\times H\times W}$\blank{,} and $I_{HR} \in \mathbb{R}^{3\times H\times W}$ denote the LR face image (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, input), the super-resolved image (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, output)\blank{,} and the ground truth HR image respectively. $I_{LR}$ is first upsampled to the same spatial dimension as $I_{HR}$ through bicubic interpolation, denoted as $I_{LR-UP} \in \mathbb{R}^{3\times H\times W}$, which is then fed to SPARNet to produce $I_{SR}$. Given a training set with $N$ pairs of LR-HR images, $\left \{ I_{LR}^i, I_{HR}^i\right\}_{i=1}^N$, we optimize SPARNet by minimizing the pixel-level $L_2$ loss given by
\begin{equation}
    \mathcal{L}_{pix}(\Theta) = \frac{1}{N} \blank{\sum_{i=1}^N} \|\mathcal{F}_{SPAR}(I_{LR-UP}^i, \Theta) - I_{HR}^i\blank{\|_2^2}, \label{eq:pix}
\end{equation}
where $\mathcal{F}_{SPAR}$ and $\Theta$ denote \blank{SPARNet and its network parameters} respectively.

\subsection{Face Attention Unit}\label{sec:FAU}
Based on the observation that some \blank{face} parts (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot, key \blank{face} components like eyes, eyebrows, nose, and mouth) are more important than the others (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot, shading of the cheek) in face SR, we propose a spatial attention mechanism to make our network focus more on the important and informative features. The key question is how to produce the attention map and how to integrate it with the convolutional layers. First, \blank{we believe} the spatial attention mechanism should not only have a high level view of the face but should also focus on low level structures. \blank{Note that a} high level view helps the network to learn how faces look, while a low level view makes the network \blank{learn local details better.} \blank{Hence, it would be desirable for the spatial attention mechanism to be able to} learn from multi-scale features. Second, residual blocks have \blank{demonstrated} great success in both general SR task \cite{ledig2017photo,Lim_2017_CVPR_Workshops,zhang2018rcan} and face SR task \cite{Chen2018CVPR,bulat2017super}. It should \blank{therefore} be beneficial to integrate spatial attention mechanism with residual blocks.

\begin{figure}[t]
  \centering
  \includegraphics[width=.9\linewidth]{Fig_SPARB.pdf}
  \caption{Face Attention Unit}
  \label{fig:sparb}
\end{figure}

Based on the above \blank{discussion}, we propose a Face Attention Unit (FAU) which extends the original residual block by introducing a spatial attention branch (see Fig.~\ref{fig:sparb}). By stacking FAUs together, important features for face SR are continuously enhanced. Denote the feature input \blank{of the $j$-th indexed FAU} as $\blank{\mathbf{x}_{j-1}} \in \mathbb{R}^{C_{j-1}\times H_{j-1} \times W_{j-1}}$, the attention map $\blank{{\boldsymbol \alpha}_{j}} \in \mathbb{R}^{1\times H_j \times W_j}$ is computed as
\begin{align}
    \blank{\mathbf{f}_j} &= \mathcal{F}_{feat}(\blank{\mathbf{x}_{j-1}}), \\
    \blank{\boldsymbol{\alpha}_j} &= \sigma(\mathcal{F}_{att}(\blank{\mathbf{f}_j})),
\end{align}
where $\blank{\mathbf{f}_j} \in \mathbb{R}^{C_{j}\times H_{j} \times W_{j}}$ is the output of the feature branch $\mathcal{F}_{feat}$, $\mathcal{F}_{att}$ \blank{denotes} the attention branch, and $\sigma$ is the sigmoid function. Finally, the output of \blank{the $j$-th indexed} FAU is \blank{given by}
\begin{equation}
    \blank{\mathbf{x}_j = \mathbf{x}_{j-1} + {\boldsymbol \alpha}_j \otimes \mathbf{f}_j}, \label{eq:res-out}
\end{equation}
where ``$\otimes$'' denotes element-wise multiplication. Details of \blank{$\mathcal{F}_{feat}$ and $\mathcal{F}_{att}$} are \blank{given in the next two paragraphs.}

\para{Attention Branch} As discussed above, the attention branch should extract multi-scale features. We adopt the hourglass block followed by an extra Conv layer to generate the attention map. The hourglass block is known to be capable of capturing information at multiple scales \cite{newell2016stacked}. It has also shown great performance in face \blank{analysis} tasks, such as face alignment~\cite{bulat2017far} and face parsing~\cite{Chen2018CVPR}. The kernel size and filter number for all Conv layers in the hourglass block are $3\times3$ and $64$ respectively.

\para{Feature Branch} After experimenting with several variants of residual blocks \cite{Lim_2017_CVPR_Workshops,He2016}, we finally choose the pre-activation Residual Unit with PReLU \cite{he2015delving} as our feature branch. Although previous general image SR work \cite{Lim_2017_CVPR_Workshops} argued that networks without batch normalization perform better, we find that pre-activation structure shows slightly better performance with batch normalization. We use PReLU as the activation function after batch normalization to avoid ``dead features'' caused by zero gradients in ReLU, and it shows more stable performance \cite{dong2016accelerating}.
For the residual blocks in downscale and upscale module, we slightly modify the original residual branch using scale Conv (see Fig.~\ref{fig:scale_res}), and (\ref{eq:res-out}) becomes
\begin{equation}
    \blank{\mathbf{x}_j = \mathcal{F}_{scale}(\mathbf{x}_{j-1}) + {\boldsymbol \alpha}_j \otimes \mathbf{f}_j},
\end{equation}
where $\mathcal{F}_{scale}$ denotes the scale Conv layer. Downscale Conv is a normal Conv layer with stride $2$, and upscale Conv is a nearest-neighbor upsampling layer with a normal Conv layer which helps avoid checkerboard artifacts \cite{odena2016deconvolution}.

\begin{figure}[htb]
  \includegraphics[width=1.\linewidth]{Fig_scale_res.pdf}
  \caption{Scale Residual Block}
  \label{fig:scale_res}
\end{figure}

\subsection{Multi-scale Discriminator Network}

We extend \blank{our} SPARNet to SPARNetHD \blank{to generate high resolution} and more realistic SR images. SPARNetHD increases the channel number of SPARNet and adopt multi-scale discriminators similar to Pix2PixHD \cite{wang2018pix2pixHD}. We refer to the discriminators as $D_1, D_2$\blank{,} and $D_3$, which are used to discriminate \blank{SR images at three different scales, namely} $512\times512$, $256\times256$\blank{,} and $128\times128$\blank{, respectively}, from the \blank{ground truth downsampled to the same resolutions}. Using multiple discriminators at different scales can help to improve the quality of the \blank{SR} images.

The loss \blank{functions used in training} SPARNetHD \blank{are composed of the following} four components:

\noindent (1) \textit{Pixel loss}.
\blank{We} use L1-norm as the pixel level loss between $I_{SR}$ and $I_{HR}$. It mainly helps to constrain the low level information in the outputs especially color, and is defined as
\begin{align}
  \blank{\mathcal{L}_{pix}^{h}} &\blank{= \frac{1}{N} \sum_{i=1}^N \|I_{SR}^i - I_{HR}^i\|_1},\\
  \blank{I_{SR}^i} &\blank{= G(I_{LR-UP}^i)},
\end{align}
\blank{where $G$ denotes the SPARNetHD generator.}

\noindent (2) \textit{Adversarial loss}.
\blank{It is the critical loss which helps to make the outputs sharper and generate more realistic textures such as hair. The loss functions of the generator and discriminator are formulated as}
\begin{gather}
  \mathcal{L}^h_{GAN\_G} = \frac{1}{N} \sum_{i=1}^N\sum_{k=1}^3 -D_k(I_{SR}^i),\\
  \begin{aligned}
  \mathcal{L}^h_{GAN\_D} = \frac{1}{N} \sum_{i=1}^N\sum_{k=1}^3 [\max(0, 1 - D_k(I_{HR}^i)) \\ + \max(0, 1 + D_k(I_{SR}^i))],
  \end{aligned}
\end{gather}
where the outputs of $D_k$ are scalars which indicates whether the input images are real ($\geq 1$) or fake ($\leq -1$).

\noindent (3) \textit{Feature matching loss}. This is the feature space loss of the discriminators \cite{wang2018pix2pixHD}. It helps to stabilize the training of GAN. Let \blank{$\mathbf{f}_{D_k}^l$} be \blank{the} feature map of the \blank{$l$-th} layer in \blank{$D_k$}, \blank{$L_k$} be the total number of layers in \blank{$D_k$}, and \blank{$M_k^l$} be the number of elements in \blank{$\mathbf{f}_{D_k}^l$}. \blank{The feature matching loss is then formulated as}
\begin{equation}
  \mathcal{L}^h_{fm} = \blank{\frac{1}{N} \sum_{i=1}^N\sum_{k=1}^3 \sum_{l=1}^{L_k} \frac{1}{M_k^l} \| \mathbf{f}_{D_k}^l(I_{SR}^i) - \mathbf{f}_{D_k}^l(I_{HR}^i)\|_1},\label{eq:fm_loss}
\end{equation}

\noindent (4) \textit{Perceptual loss}. Different from feature matching loss, perceptual loss~\cite{ledig2017photo} is the feature space loss of the pretrained VGG19 network~\cite{simonyan2014very}. It helps to constrain the high level semantics in the outputs. We follow the notation of Eq.~\ref{eq:fm_loss} and denote the perceptual loss as
\begin{equation}
\mathcal{L}^h_{pcp}=\blank{\frac{1}{N} \sum_{i=1}^N\sum_{l=1}^{L_{VGG}} \frac{1}{M_{VGG}^l} \|\mathbf{f}_{VGG}^l(I_{SR}^i) - \mathbf{f}_{VGG}^l(I_{HR}^i)\|_1.}
\end{equation}

\noindent \blank{Finally, the loss functions are defined as}
\begin{gather}
  \mathcal{L}^h_G = \lambda_{pix}\mathcal{L}^h_{pix} + \lambda_{adv}\mathcal{L}^h_{GAN\_G} + \lambda_{fm}\mathcal{L}^h_{fm} + \lambda_{pcp}\mathcal{L}^h_{pcp}\blank{,} \label{eq:loss-hd-final}\\
  \mathcal{L}^h_D = \mathcal{L}^h_{GAN\_D}\blank{,}
\end{gather}
where $\mathcal{L}^h_G$ and $\mathcal{L}^h_D$ are minimized iteratively to train $G$ and $D$, and $\lambda_{pix}, \lambda_{adv}, \lambda_{fm}$, and $\lambda_{pcp}$ are the \blank{weights for each loss item respectively}.

The reason why we use more complicated loss functions to train SPARNetHD is to generate results with high perceptual quality. As pointed out by \cite{blau2018perception}, distortion and perceptual quality are at odds with each other. Methods trained with only $\mathcal{L}_{pix}$ in Eq. \ref{eq:pix} always lead to better PSNR and SSIM scores, but over-smoothed results with bad perceptual quality. Therefore, we train SPARNet with $\mathcal{L}_{pix}$ to make fair comparison on distortion metrics (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, PSNR and SSIM scores) with previous works.

\subsection{Training Details}
For SPARNet, we set the batch size as 64, and fix the learning rate at $2\times10^{-4}$. We \blank{use} Adam~\cite{kingma2014adam} to optimize the model with $\beta_1=0.9$ and $\beta_2=0.99$. For SPARNetHD, we empirically set $\lambda_{pix}=100$, \blank{$\lambda_{adv}=1$}, $\lambda_{fm}=10$, and \blank{$\lambda_{pcp}=1$}. The learning rates of $G$ and $D$ are $1\times10^{-4}$ and $4\times10^{-4}$ respectively. We \blank{use} Adam \blank{to optimize} both $G$ and $D$ with $\beta_1=0.5$ and $\beta_2=0.99$. The batch size \blank{is} set to $2$. Our models \blank{are} implemented in PyTorch and run on a Tesla K40 GPU.

\section{Experiments}

In this section, we conduct experiments for SPARNet and SPARNetHD respectively. First, we \blank{analyze}  the effectiveness of the proposed spatial attention mechanism in SPARNet, and compare SPARNet with previous face SR methods and general SR methods with different evaluation metrics using the same training data. Second, we evaluate the performance of SPARNetHD on real LR faces trained on synthetic datasets.

\subsection{Analysis of SPARNet}

\subsubsection{Datasets and Evaluation Metrics}

\para{Training Data} We use CelebA~\cite{liu2015faceattributes} to train SPARNet. We first detect faces using MTCNN~\cite{zhang2016mtcnn}, and crop out the face regions roughly without any pre-alignment operation. Next, we select images larger than $128\times128$, resize them to $128\times128$ through bicubic interpolation, and use them as the HR training set. \blank{The} LR training set is obtained by downsampling the HR images to $16\times16$. This results in roughly $179$K image pairs. To avoid overfitting, we \blank{carry} out data augmentation by random horizontal flipping, image rescaling (between $1.0$ and $1.3$), and image rotation ($\ang{90}$, $\ang{180}$ and $\ang{270}$).

\para{Testing Data} Following previous works~\cite{zhu2016deep,huang2017wavelet,Chen2018CVPR}, we use the test set of Helen~\cite{Sagonas_2013_CVPR_Workshops} for evaluation of image quality and landmark detection. For identity similarity evaluation, we use the same test set as in \cite{zhang2018super}, which is specifically designed to preserve identity in face SR. This dataset contains randomly selected images of $1,000$ identities from UMD-Face~\cite{bansal2016umdfaces}.

Following the practice of image super-resolution, the Peak Signal-to-Noise Ratio (PSNR) and Structure SIMilarity (SSIM) index calculated on the luminance channel are used as the primary quantitative evaluation metrics. Since two most important applications of face SR are alignment and identification of LR faces, we adopt two further metrics, namely landmark detection and identity similarity, to evaluate face SR performance.

\para{Landmark Detection} As mentioned before, structure recovery is very important in face SR. Similar to \cite{Chen2018CVPR,bulat2017super}, we use landmark detection accuracy for evaluation. Specifically, we use a popular landmark detection model, FAN\footnote{\url{https://github.com/1adrianb/face-alignment}} \cite{bulat2017far}, which is pre-trained on HR faces, to detect landmarks on SR and HR faces. Following \cite{bulat2017far}, we use the \blank{area under the curve} (AUC) of the normalized mean error (NME) to quantify the performance.

\para{Identity Similarity} Identity similarity measures how well identity information is preserved in a super-resolved face. Same as \cite{zhang2018super}, we first extract identity feature vectors for the SR and HR faces using a pre-trained SphereFace model\footnote{\url{https://github.com/clcarwin/sphereface_pytorch}} \cite{Liu_2017_CVPR}, and then compute the identity similarity as the cosine distance between the two feature vectors.

\begin{figure}[t]
  \centering
  \subfigure[Image Quality]{\includegraphics[width=.49\linewidth]{Fig_ablation1_a.pdf}\label{fig:base-spar-quality}}
  \subfigure[Landmark Detection]{\includegraphics[width=.49\linewidth]{Fig_ablation1_b.pdf}\label{fig:base-spar-lmk}}
  \subfigure[Visualization of the PSNR (left) and SSIM (right) error maps. The heatmaps for Baseline and SPARNet are the average 2D PSNR/SSIM error maps (larger value indicates larger error), and the improvement heatmaps show the error reduction (larger value indicates more error reduction).]{\includegraphics[width=.99\linewidth]{Fig_ablation1_c.pdf} \label{fig:score-improve-visual}}
  \caption{Quantitative comparison between Baseline, SPARNet-V1 and SPARNet on the Helen test set.} \label{fig:base-spar}
\end{figure}

\subsubsection{Ablation Study}

We compare the following three variants of our model:

\begin{itemize}
  \item Baseline: residual SR network without any spatial attention branches.
  \item SPARNet-V$N$: To evaluate the effectiveness of using multiple FAU blocks, we keep the feature branch unchanged and vary the numbers of attention branch used in SPARNet \footnote{Reducing the numbers of entire FAU would make the network shallower which would definitely make the results worse, therefore we only vary the number of attention branch}. Considering that decoder parts are supposed to be more sensitive to spatial attention, we gradually increase the number of attention branches from the backend. We denote models with $N$ spatial attention branches as SPARNet-V$N$, where $N \in \{1, 2, 4, 8, 16\}$.
  \item SPARNet-S$M$: To demonstrate that it is crucial for the attention mechanism to learn from multi-scale features, we vary the numbers of downsample/upsample blocks to change the smallest size (scale) of feature maps in the bottleneck. We denote SPARNet with $M\times M$ size of feature maps in the bottleneck of attention branch as SPARNet-S$M$, where $M \in \{2, 4, 8, 16\}$ and smaller $M$ indicates more scales of features are used.
  \item SPARNet: the full model used in this work, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, SPARNet-V16-S4.
\end{itemize}

\para{Effectiveness of Spatial Attention} To evaluate the effectiveness of the proposed spatial attention mechanism, we compare the results of SPARNet-V1 and SPARNet with that of the Baseline model. Fig~\ref{fig:base-spar-quality} shows that, with the proposed spatial attention mechanism, we can generate results with better image quality in terms of both PSNR and SSIM scores. This is expected as the spatial attention mechanism enables our network to focus on \blank{and} better recover the key \blank{face} structures. To visualize the improvement brought by the attention mechanism, we align the faces to a fixed template based on five key facial landmarks (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot two eye centers, nose, and two mouth corners), calculate 2D PSNR and SSIM error maps\footnote{The PSNR error is calculated by mean square error, and the SSIM error is calculated by subtracting the SSIM score from 1.} for each image, and average the error maps over the whole test set. It can be observed from Fig.~\ref{fig:score-improve-visual} that pixels corresponding to key \blank{face} structures are most difficult to recover (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, with larger errors), and as expected, most of the improvement happens around them. Furthermore, we compare the landmark detection results on the super-resolved images. Better recovery of key \blank{face} structures should lead to \blank{higher} landmark detection accuracy. It can be seen in Fig.~\ref{fig:base-spar-lmk} that SPARNet achieves better landmark detection accuracy ($58.51\%$ AUC) than the Baseline ($56.23\%$ AUC). This demonstrates the benefit of the spatial attention mechanism. Fig.~\ref{fig:ablation-result-example} shows some SR results on the Helen test set. It can be observed that SPARNet can produce sharper and clearer \blank{face} structures than the Baseline, especially for the eyes.

\para{Number of FAUs} By comparing the results of SPARNet-V1 with that of SPARNet in Fig.~\ref{fig:base-spar}, we can see that SPARNet, which is composed entirely of FAUs, performs better than SPARNet-V1, which has only one FAU at the end.  Although the attention map of SPARNet-V1 can focus on the key \blank{face} structures, some noises also show up on the face region \blank{and cause} some distraction \blank{(see Fig.~\ref{fig:ablation-result-example} bottom left)}. This suggests that bootstrapping the key \blank{face} structures using a single FAU may be sub-optimal. On the other hand, the \blank{sequence of} FAUs in SPARNet allow the attention maps to gradually focus on different key \blank{face} structures at different stages. For instances, one can see the attention map of the second last FAU focuses cleanly on the outline of the key \blank{face} structures, and that of the last FAU focuses on the general face region \blank{(see Fig.~\ref{fig:ablation-result-example} bottom right). In Fig. \ref{fig:ablation-faus}, we show more results comparison of SPARNet-V$N$. It can be observed that the evaluation PSNR scores of SPARNet-V$N$ show a positive correlation with $N$, which demonstrates the effectiveness of stacking multiple FAUs.}

\para{Multiscale features in FAU} The results of SPARNet-S$M$ are shown in Fig. \ref{fig:ablation-hourglass}. We can see that models with more scales (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, smaller $M$) generally perform better, but the improvement declines when $M$ becomes smaller. The performance of SPARNet-S$4$ is similar to SPARNet-S$2$. Therefore, we set $M=4$ for SPARNet to balance the performance and computation cost.

\begin{figure}[t]
  \centering
  \includegraphics[width=.99\linewidth]{Fig_ablation1_d.pdf}
  \caption{Results on the Helen test set. Top: HR photo and SR results; bottom: spatial attention map in FAU of SPARNet-V1 (left) and last three FAUs of SPARNet (right).} \label{fig:ablation-result-example}
\end{figure}

\begin{figure}[ht]
    \centering
    \centering
    \subfigure[Evaluation PSNR score of SPARNet-V$N$ during training.]{\includegraphics[width=.49\linewidth]{ablation_faus.pdf}\label{fig:ablation-faus}}
    \subfigure[Evaluation PSNR score of SPARNet-S$M$ during training.]{\includegraphics[width=.49\linewidth]{ablation_hourglass.pdf}\label{fig:ablation-hourglass}}
    \caption{Ablation study of SPARNet-V$N$ and SPARNet-S$M$ on Helen testset. We set $M=4$ when training SPARNet-V$N$, and $N=16$ for SPARNet-S$M$.}
    \label{fig:more-sparnet-ablatoin}
\end{figure}

\subsection{Comparisons of SPARNet against Other Methods}

We compare SPARNet with state-of-the-art general image SR method RCAN~\cite{zhang2018rcan}, latest attention network for image classification CBAM~\cite{woo2018cbam}\blank{,} and other face SR methods, including URDGN~\cite{yu2016ultra}, Wavelet-SRNet~\cite{huang2017wavelet}, Attention-FH~\cite{Cao2017Attention}, FSRNet~\cite{Chen2018CVPR}\blank{,} SICNN \cite{zhang2018super}, PFSRNet \cite{pfsrnet} and DICNet \cite{dicnet}. When comparing with RCAN and CBAM, we replace our FAU with the corresponding attention module (borrowed from public released codes) and keep network depth the same for a fair comparison. \emph{All models are trained \blank{using} the same dataset except for FSRNet and SICNN.} Note that FSRNet only provides test code and SICNN relies on a much larger pre-aligned training set.

\subsubsection{Overall Results}
Quantitative comparison with other state-of-the-art methods on the Helen test set is shown in Table~\ref{tab:psnr-ssim}. SPARNet clearly outperforms both generic SR methods and face SR methods in terms of both PSNR and SSIM scores. With the spatial attention mechanism, SPARNet can better recover important \blank{face} structures. One can observe from Fig.~\ref{fig:qual-comp} that while most of the other methods fail to recover the eyes and nose, and the shapes are blurry, SPARNet can generate sharper results with shapes close to the ground truth HR images.

\begin{table*}[ht]
  \caption{Quantitative comparison on the Helen (first 3 rows) and UMD (last row) test \blank{sets} with a $16\times16$ input size and an upscale factor of $8\times$. The AUCs are calculated under a threshold of $10\%$. The results of FSRNet$^*$ are generated using the test model provided by its authors.}
  \label{tab:psnr-ssim}
  \centering
  \begin{tabular}{|c|ccccccc|cc|}
    \hline
    Method & Bicubic & RCAN & CBAM & URDGN & Wavelet & Att-FH & FSRNet$^*$ & Baseline & SPARNet \\ \hline \hline
    PSNR  & 23.52  & 26.40  & \textcolor{blue}{\textbf{26.46}}  & 25.17  & 26.42  & 25.10  & 24.97 & 26.38 & \textcolor{red}{\textbf{26.59}}   \\ \hline
    SSIM  & 0.6408 & 0.7648 & 0.7666 & 0.7140 & \textcolor{blue}{\textbf{0.7711}} & 0.7188 & 0.7091 & 0.7639 & \textcolor{red}{\textbf{0.7716}}  \\ \hline
    AUC($<10\%$) & 4.42\% & 56.53\% & 56.83\% & 41.78\% & \textcolor{blue}{\textbf{58.44}}\% & 46.63\% & 40.42\% & 56.23\% & \textcolor{red}{\textbf{58.51\%}}  \\ \hline \hline
    Identity & 0.1851 & 0.5373 & \textcolor{blue}{\textbf{0.5378}} & 0.3981 & 0.5147 & 0.4731 & 0.4765 & 0.5302 & \textcolor{red}{\textbf{0.5546}}  \\ \hline
  \end{tabular}
\end{table*}

\begin{figure*}[ht]
  \includegraphics[width=.99\linewidth]{qual_comp/3220402975_1.png}
  \includegraphics[width=.99\linewidth]{qual_comp/3214022978_1.png}
  \begin{minipage}[t]{.99\textwidth}
    \footnotesize
    \centering
    \begin{tabular}{*{11}{C{1.4cm}}}
    (a) HR & (b) Bicubic & (c) RCAN~\cite{zhang2018rcan} & (d) CBAM~\cite{woo2018cbam} & (e) URDGN~\cite{yu2016ultra} & (f) Wavelet~\cite{huang2017wavelet} & (g) Att-FH~\cite{Cao2017Attention} & (h) FSRNet~\cite{yu2018face} & (i) Baseline & (j) SPARNet
    \end{tabular}
  \end{minipage}
  \caption{Qualitative comparison with state-of-the-art methods. The resolution of the input is $16\times16$ and the upscale factor is $8\times$. }
  \label{fig:qual-comp}
\end{figure*}

\subsubsection{Detailed Comparisons}

\para{Attention Mechanisms} Among all the methods under comparison, only RCAN, CBAM\blank{,} and Attention-FH have an attention mechanism. RCAN is designed for generic SR task. It applies a channel attention mechanism in the feature space. Although channel attention has been shown to be beneficial to general SR task, spatial structures are more important in very low resolution face super-resolution. This explains why RCAN does not show big improvement over the Baseline in terms of PSNR score ($26.40$ vs $26.38$). Attention-FH adopts a patch level attention and cannot recover details (see Fig~\ref{fig:qual-comp}(g)). CBAM also employs a spatial attention. However, it is manipulated to extract semantic information through pooling operations which cause loss of shape and edge details. We compare the SR \blank{results} and spatial attention maps generated by SPARNet and CBAM in Fig~\ref{fig:cbam-spar}. We can see that the attention maps produced by CBAM are blurry and not able to outline important details such as eyes and mouth, whereas the attention maps of SPARNet, thanks to the multi-scale feature based approach, provide more detailed structure information. Accordingly, the mouth and nose generated by SPARNet are sharper and clearer than that of CBAM.

\begin{figure}[ht]
  \subfigure[SR results of CBAM and SPARNet.]{\includegraphics[width=.99\linewidth]{Fig_cbam_spar_a.pdf} \label{fig:cbam-spar-a}}
  \subfigure[Attention maps of CBAM (top) and SPARNet (bottom).]{\includegraphics[width=.99\linewidth]{Fig_cbam_spar_b.pdf} \label{fig:cbam-spar-b}}
  \caption{SR results and attention maps of CBAM and SPARNet.}
  \label{fig:cbam-spar}
\end{figure}

\para{Comparison with FSRNet~\cite{Chen2018CVPR}} Since FSRNet only provides test code, the results \blank{of FSRNet} reported in Table~\ref{tab:psnr-ssim} \blank{are} not obtained using the same training set as the others. To make a fair comparison, we retrain and test our model on the same dataset used by FSRNet (18K images from CelebA), and show the results in Table~\ref{tab:fsrnet-comp}. Although FSRNet uses face parsing map as an additional supervision, SPARNet still outperforms FSRNet in terms of PSNR and SSIM scores. This demonstrates the superiority of the proposed spatial attention mechanism.

\begin{table}[ht]
  \caption{Comparison between FSRNet and SPARNet on CelebA.} \label{tab:fsrnet-comp}
  \centering
  \begin{tabular}{|c|c|c|c|c|}
  \hline
  Method & FSRNet & SPARNet \\ \hline
  PSNR/SSIM & 26.31/0.7522 &  \textcolor{red}{\textbf{26.68/0.7741}} \\ \hline
  \end{tabular}
\end{table}

\para{Comparison with Wavelet-SRNet~\cite{huang2017wavelet}} Wavelet-SRNet shows closest performance to SPARNet in terms of SSIM score. Nonetheless, SPARNet requires much less parameters and is more computational efficient and flexible than Wavelet-SRNet. The performance and efficiency comparison between Wavelet-SRNet and SPARNet is summarized in Fig.~\ref{fig:wave-spar-comp}. Wavelet-SRNet has $4^n$ parallel subnets with an upscale factor of $2^n$. This implies the network parameters increase quadratically with the upscale factor. Besides, it requires different input size for each upscale factor. Because the output size is usually fixed, a smaller upscale factor needs a larger size of input when training the network. Since a larger input means more convolution operations and requires higher computational power, the GFLOPs of Wavelet-SRNet increase dramatically when the upscale factor is small. In contrast, SPARNet uses the same architecture for different upscale factors, and achieves better performance without increasing its computational complexity.

\begin{figure}[t]
  \includegraphics[width=.99\linewidth]{Fig_wavelet.pdf}
  \caption{Performance and efficiency comparison between Wavelet-SRNet and SPARNet under different upscale factors.}
  \label{fig:wave-spar-comp}
\end{figure}

\para{Comparison with landmark based methods} We compare SPARNet with two recent landmark based methods PFSRNet\footnote{\url{https://github.com/DeokyunKim/Progressive-Face-Super-Resolution}} \cite{pfsrnet} and DICNet\footnote{\url{https://github.com/Maclory/Deep-Iterative-Collaboration}} \cite{dicnet}. To get the best results of the compared method, we directly use their public models instead of retraining them since both of them are trained on CelebA, the same as SPARNet. We evaluate the performance on the Helen test dataset provided by DICNet because our test images are cropped in a different way and DICNet performs bad on it. The quantitative results and visual examples are shown in Table \ref{tab:extra-comp} and Fig. \ref{fig:extra-comp} respectively. As we can observe from Table \ref{tab:extra-comp}, SPARNet shows the best PSNR and SSIM scores in the test dataset. The visual results in Fig. \ref{fig:extra-comp} indicate that SPARNet can recover the key face components, especially the eyes, better than DICNet even without any extra landmark information. We hypothesize that this is because it is too difficult to detect accurate landmarks for low resolution face images, and the multi-stage iterative process in DICNet amplifies the error. As for PFSRNet, we found it performs badly when test faces are not aligned. Since detecting landmarks from $16\times16$ LR face is by itself a difficult task, we find the practical value of PFSRNet is limited compared with DICNet and our SPARNet. We also show the computation complexity in Table \ref{tab:extra-comp}. Compared with state-of-the-art method DICNet, our method produces better results with much less parameters. Moreover, our SPARNet is faster than DICNet which is an iterative framework. Compared with PFSRNet, our method achieves much better performance with similar computation complexity.

\begin{table}[t]
    \centering
    \caption{Quantitative comparison on the Helen test dataset provided by DICNet \cite{dicnet}.}
    \label{tab:extra-comp}
    \begin{tabular}{|c|c|c|c|}
    \hline
      Methods & PFSRNet \cite{pfsrnet} & DICNet \cite{dicnet} & SPARNet \\ \hline
      PSNR & 24.13 & 26.73 & \redb{26.97} \\ \hline
      SSIM & 0.6688 & 0.7955 & \redb{0.8026} \\ \hline \hline
      Params (M) & 8.97 & 22.80 & 9.86 \\ \hline
      Time (s) & 0.045 & 0.065 & 0.051 \\ \hline
    \end{tabular}
\end{table}

\begin{figure}[t]
    \centering
    \includegraphics[width=0.99\linewidth]{extra_compare.pdf}
    \caption{Visual comparison between PFSRNet, DICNet, and SPARNet on Helen test dataset provided by DICNet. All results are generated by public models trained on CelebA. PFSRNet generates bad results for unaligned test faces. SPARNet generates better key face components than DICNet especially in the eyes.}
    \label{fig:extra-comp}
\end{figure}

\para{Comparison with Different Upscale Factors} We also trained models with $4\times$, and $16\times$ upscale factors. The $4\times$ model was trained from scratch, whereas the $16\times$ model was initialized with the pre-trained $8\times$ model. We show the results of $4\times$ and $16\times$ upscale factors in Table~\ref{tab:multi-res}. It can be seen that SPARNet achieves state-of-the-art results for both $4\times$ and $16\times$ upscale factors, especially for \blank{the} $16\times$ upscale factor.

\begin{table}[!thb]
  \caption{Image quality comparison on Helen with upscale factors $4\times$ and $16\times$}
  \label{tab:multi-res}
  \setlength{\tabcolsep}{0.25em}
  \centering
  \begin{tabular}{|c|c|cccc|c|}
  \hline
    & Scale & Bicubic & RCAN & CBAM & Wavelet & SPARNet \\ \hline \hline
  PSNR &  \multirow{2}{*}{$4\times$} & 27.43 & 30.73 & \textbf{\textcolor{blue}{30.76}} & 30.67 & \textbf{\textcolor{red}{30.83}} \\
  SSIM &  & 0.8049 & 0.8857 & 0.8861 & \textbf{\textcolor{red}{0.8888}} & \textbf{\textcolor{blue}{0.8872}} \\ \hline \hline
  PSNR &  \multirow{2}{*}{$16\times$} & 20.22 & 20.71 & \textbf{\textcolor{blue}{20.75}} & 22.44 & \textbf{\textcolor{red}{22.85}}\\
  SSIM &  & 0.5180 & 0.6360 & \textbf{\textcolor{blue}{0.6375}} & 0.6314 & \textcolor{red}{\textbf{0.6426}} \\ \hline
  \end{tabular}
\end{table}

\subsubsection{Landmark Detection Results}
As mentioned previously, the proposed spatial attention mechanism helps recover important \blank{face} structures. Landmark detection can be exploited to evaluate such an ability. From Table~\ref{tab:psnr-ssim}, we can see that SPARNet \blank{achieves} the best performance in landmark detection. Wavelet-SRNet's performance is similar to SPARNet, but SPARNet is more efficient (refer to Fig.~\ref{fig:wave-spar-comp}) and shows much better performance (in terms of PSNR/SSIM scores) for \blank{the} $16\times$ upscale factor (refer to Table~\ref{tab:multi-res}).

\subsubsection{Identity Similarity Results}
We first evaluate the models in Table~\ref{tab:psnr-ssim} on the UMD-Face test set. Note that these models are trained on CelebA without pre-alignment. SPARNet achieves the best performance. We also notice that GAN based methods such as URDGN \blank{are} worse than non-GAN based methods. This is because GAN based methods \blank{target} at producing realistic textures but \blank{they do} not care whether the generated textures \blank{are} consistent with the \blank{HR images}. Therefore, \blank{they} may generate textures which look sharper but disturb the identity information. To further explore how well SPARNet can preserve identity information, we retrain our model using the same dataset as SICNN to allow a fair comparison. We denote this model as SPARNet$^*$. From Table~\ref{tab:id-sim}, we can see that SPARNet$^*$ shows better performance than SICNN even without using any explicit identity supervision. This proves that the proposed spatial attention mechanism is beneficial to preserving identity information in face SR.

\begin{table}[thb]
  \caption{Comparison of identity similarity between SICNN and SPARNet$^*$. (SPARNet$^*$ is trained with the same dataset as SICNN.)}
  \label{tab:id-sim}
  \centering
  \begin{tabular}{|c|c|c|c|}
  \hline
  Method  & SICNN~\cite{zhang2018super} & SPARNet$^*$ \\ \hline \hline
  Identity Similarity & 0.5978 & \textcolor{red}{\textbf{0.6272}} \\ \hline
  \end{tabular}
\end{table}

\subsection{Evaluation of SPARNetHD}

Different from SPARNet which aims to hallucinate very low resolution faces (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, $16\times16$) that are difficult to recognize, the extended SPARNetHD tries to deal with real world low resolution faces. \blank{Typical} application scenarios \blank{include} old photos \blank{and} unclear faces shot by \blank{low-end} devices. These LR faces usually \blank{do} not have fixed upscale factors and are usually noisy and blurry. \blank{Hence}, we need \blank{a} different degradation model \blank{for data synthesis. We} also need high resolution datasets in order to get high resolution outputs.

\subsubsection{Degradation Model}
According to previous works \cite{Li_2018_ECCV,xu2017learning} and common practice in \blank{single image SR} framework, we generate the LR image $I_l^s$ \blank{from the HR image $I_h$ using} the following degradation model:
\begin{equation}
  I_l^s = ((I_h \blank{*} \textbf{k}_\varrho)\downarrow_s + \textbf{n}_\delta)_{JPEG_q}, \label{equ:degrade}
\end{equation}
where \blank{$*$} represents the convolution operation between \blank{$I_h$} and a blur kernel $\textbf{k}_\varrho$ with parameter $\varrho$. $\downarrow_s$ is the downsampling operation with a scale factor $s$. $\textbf{n}_\delta$ denotes the additive white Gaussian noise (AWGN) with a noise level $\delta$. $(\cdot)_{JPEG_q}$ indicates the JPEG compression operation with quality factor $q$. The hyper parameters $\varrho, s, \delta, q$ are randomly selected for each HR image $I_h$, and $I_l^s$ is generated online. We set the hyper parameters as
\begin{itemize}
\item $\textbf{k}_\varrho$ is the blur kernel. We randomly choose one of the following four kernels: Gaussian Blur \blank{($3 \leq \varrho \leq 15$)},  Average Blur \blank{($3 \leq \varrho \leq 15$)}, Median Blur \blank{($3 \leq \varrho \leq 15$)}, Motion Blur \blank{($5 \leq \varrho \leq 25$)};
\item $\downarrow_s$ is the downsample operation. The scale factor $s$ is randomly selected in $[\frac{16}{512}, \frac{128}{512}]$;
\item $\textbf{n}_\delta$ is addictive white gaussian noise (AWGN) with \blank{$0 \leq \delta \leq 0.1\times255$};
\item $JPEG_q$ is the JPEG operation. The compression level is randomly chosen from $[60, 85]$, in which higher means stronger compression level.
\end{itemize}
After \blank{obtaining} $I_l^s$, $I_{LR-UP}=(I_l^s)\uparrow_s$ \blank{serves as the LR input of SPARNetHD}.

\subsubsection{Datasets and Evaluation Metrics}

\para{Training data} We adopt the FFHQ \cite{karras2019style} dataset \blank{for training}. This dataset consists of $70,000$ high-quality images at a size of $1024\times1024$ crawled from the internet. All images are automatically cropped and aligned. We resize the images to $512\times512$ with bilinear downsampling as the ground-truth HR images, and synthesize the LR inputs online with Eq. \ref{equ:degrade}.

\para{Synthetic Test Data} We use the test partition of CelebAHQ dataset~\cite{karras2017progressive} as synthetic test data. CelebAHQ contains $30,000$ HR faces in total which are split into a training set ($24,183$ images), a validation set ($2,993$ images), and a test set ($2,824$ images). We generate the LR dataset with Eq.~\ref{equ:degrade}, denoted as CelebAHQ-Test.

\para{Natural Test Data} We collect $1,020$ faces smaller than $48\times48$ from CelebA. We also collect some old photos from the internet for testing. All images are cropped and aligned in the same manner as FFHQ, and then resized to $512\times512$ using bicubic upsampling. We merge all these images together and create a new dataset containing $1,051$ natural LR faces, denoted as CelebA-TestN.

\para{Evaluation Metrics} For CelebAHQ-Test, we use LPIPS (Learned Perceptual Image Patch Similarity) \cite{zhang2018perceptual} as our evaluation metric \blank{because it} can better represent the visual quality \blank{of} HR images than PSNR and SSIM. For CelebA-TestN, since there are no ground truth HR images, we use FID (Fréchet Inception Distance) \cite{heusel2017gans} to measure the similarity between the SR results and reference datasets containing HR face images. We use the HR version of CelebAHQ-Test as the reference dataset.

\subsubsection{Ablation Study of Loss Functions}
\begin{figure*}[!ht]
  \centering
  \includegraphics[width=.99\textwidth]{ablation_sparnethd.pdf}
  \begin{tabular}{*{6}{C{2.6cm}}}
    LR Input & SPARNetHD & w/o $\mathcal{L}_{pix}$ & w/o $\mathcal{L}_{GAN}$ & w/o $\mathcal{L}_{fm}$ & w/o $\mathcal{L}_{pcp}$
  \end{tabular}
  \caption{Ablation study of losses used in SPARNetHD on the synthetic dataset CelebAHQ-Test (first row) and real dataseet CelebA-TestN (second row). Better zoom in to see the details.}
  \label{fig:sparhd-ablation}
\end{figure*}
To verify the effectiveness of different loss terms in Eq. \ref{eq:loss-hd-final}, we conduct an ablation study by removing each of them separately. Same as SPARNetHD, all models are trained for 100k iterations which take about 3 days on a single GPU. The comparison results are shown in Table \ref{tab:quant-results} and Fig. \ref{fig:sparhd-ablation}. We can observe that $\mathcal{L}_{pix}$ has the least influence on the final performance because it mainly affects the subtle low level details. Comparing the results of second column and third column, the skin color of SPARNetHD (w/o $\mathcal{L}_{pix}$) is mixed with slight background green color compared with full SPARNetHD in the first row, and teeth shape is less natural w/o $\mathcal{L}_{pix}$ in the second row (please zoom in to see the details). From the 4th column, we can see that the network fails to generate sharp edges and realistic details without adversarial loss, and these greatly degrade the quantitative performance. In the 5th column, we can observe that the generated details are less realistic and there are some noise-like artifacts near the hair in the first row. This is mainly because the GAN network trained without $\mathcal{L}_{fm}$ is less stable. In the last column, we can see obvious shape distortions in the eyes and lips since $\mathcal{L}_{pcp}$ mainly helps to constrain mid-level and high-level semantics.

\subsubsection{Comparison with Other Methods}

\begin{figure*}[!ht]
  \centering
  \includegraphics[width=.99\textwidth]{sparnet_celebahq_test.jpg}
  \begin{minipage}[t]{.99\textwidth}
    \centering
    \begin{tabular}{*{6}{C{2.6cm}}}
    (a) Synthetic LR & (b) ESRGAN & (c) Pix2PixHD & (d) SFTGAN & (e) Ours & (f) GT
    \end{tabular}
  \end{minipage}
  \caption{Visual comparison between \blank{results produced by} SPARNetHD and other methods on \blank{the} synthetic dataset CelebAHQ-Test. Better zoom in to see the details.}
  \label{fig:sparhd-comp-synthetic}
\end{figure*}

\begin{figure*}[!ht]
  \centering
  \includegraphics[width=0.99\textwidth]{sparnet_celebatestn.jpg}
  \begin{minipage}[t]{.99\textwidth}
    \centering
    \begin{tabular}{*{5}{C{3.1cm}}}
    (a) Real LR & (b) ESRGAN & (c) Pix2PixHD & (d) SFTGAN & (e) Ours
    \end{tabular}
  \end{minipage}
  \caption{Visual comparison between \blank{results produced by} SPARNetHD and other methods on real LR dataset CelebA-TestN. Better zoom in to see the details.}
  \label{fig:sparhd-comp-natural}
\end{figure*}

We conduct experiments to compare SPARNetHD with other methods on CelebAHQ-Test and CelebA-TestN. \blank{Since very} few previous face SR methods can produce \blank{high resolution} outputs, we mainly compare SPARNetHD with general SR methods and GAN based image-to-image translation methods which \blank{provide} public codes and can be used to generate \blank{high resolution} outputs. Specifically, we compare SPARNetHD with state-of-the-art SR method ESRGAN~\cite{wang2018esrgan}, parsing map based SR method SFTGAN~\cite{wang2018sftgan}, high resolution image translation method Pix2PixHD \cite{wang2018pix2pixHD}, and blind face restoration method based on a reference image GFRNet \cite{Li_2018_ECCV}. We retrain ESRGAN, SFTGAN and Pix2PixHD. We use existing face parsing model to generate face parsing map for SFTGAN. As for GFRNet, we only \blank{carry out a} visual comparison with the results reported in the \blank{original} paper because it needs reference images for training and testing.

\begin{table}[t]
\caption{Quantitative comparison on CelebAHQ-Test and CelebA-TestN. \emph{Note for w/o $\mathcal{L}_{GAN}$$^*$: because $L_{fm}$ depends on the discriminator, we also remove it when $\mathcal{L}_{GAN}$ is not used.}}
\label{tab:quant-results}
\centering
\begin{tabular}{l|c|c}
\hline
Methods & CelebAHQ-Test (LPIPS$\downarrow$) & CelebA-TestN (FID$\downarrow$) \\
\hline\hline
ESRGAN \cite{wang2018esrgan} &  0.49 & 60.67 \\
SFTGAN \cite{wang2018sftgan}   &  0.36 & 37.76 \\
Pix2PixHD \cite{wang2018pix2pixHD} & 0.36 & 43.1  \\
\hline \hline
SPARNetHD & \redb{0.29} & \redb{27.16} \\
 w/o $\mathcal{L}_{pix} $ & 0.31 & 30.63 \\
 w/o $\mathcal{L}_{GAN}$$^*$ & 0.38 & 39.75 \\
 w/o $\mathcal{L}_{fm} $ & 0.32 & 33.52 \\
 w/o $\mathcal{L}_{pcp} $  & 0.31 & 32.97 \\
\hline
\end{tabular}
\end{table}

Table~\ref{tab:quant-results} gives \blank{a} quantitative \blank{comparison} on CelebAHQ-Test and CelebA-TestN. We can observe that SPARNetHD \blank{outperforms the other methods on both datasets}. Pix2PixHD performs better than ESRGAN because of the multi-scale discriminators. \blank{The} fact that the results of SPARNetHD are much better than \blank{that of} Pix2PixHD demonstrates the effectiveness of the proposed attention mechanism\blank{,} which is the key difference between SPARNetHD and Pix2PixHD. SFTGAN shows slightly better performance than Pix2PixHD with the help of face parsing maps. This \blank{shows the advantage of taking parsing map as additional supervision}. Nevertheless, the proposed SPARNetHD outperforms SFTGAN by a large margin. This is because parsing map can only provide coarse semantic level guidance, while the proposed spatial attention mechanism cannot only provide coarse semantic guidance but also give low level texture guidance.

The \blank{qualitative} results in Fig.~\ref{fig:sparhd-comp-synthetic} and Fig~\ref{fig:sparhd-comp-natural} \blank{are consistent} with the observations in Table~\ref{tab:quant-results}. Fig.~\ref{fig:sparhd-comp-synthetic} shows the results on \blank{the} synthetic test dataset CelebAHQ-Test. We can see that ESRGAN fails to generate realistic faces. While the results of Pix2PixHD \blank{look} much better, \blank{undesirable artifacts show up} in the right eyes. SFTGAN does not have such problem\blank{, but it cannot} generate detailed textures\blank{, especially in the hair and teeth}. In contrast, SPARNetHD can restore key \blank{face} components as well as the texture details in the hair and teeth. \blank{Similar} phenomenon can also be observed \blank{in} Fig.~\ref{fig:sparhd-comp-natural}\blank{,} which \blank{shows} the results on natural LR faces. While the competitive methods generate many artifacts for real LR faces, our results are much more robust and \blank{natural. This} illustrates the good generalization ability of SPARNetHD.

We also compare SPARNetHD with a recent blind face restoration method GFRNet. GFRNet takes a LR image and a HR image of the same person as \blank{input} to restore the LR image. We \blank{show some qualitative results} on natural LR faces in Fig.~\ref{fig:sparhd-comp-gfrnet}. It can be observed that SPARNetHD generates much better texture details than GFRNet even without \blank{utilizing} any additional information \blank{in} the training stage.

\begin{figure}[!htb]
  \centering
  \includegraphics[width=.99\linewidth]{sparnet_gfrnet_comp.jpg}
  \begin{minipage}[t]{.99\linewidth}
    \centering
    \begin{tabular}{*{3}{C{2.6cm}}}
    (a) Real LR & (b) GFRNet & (c) Ours
    \end{tabular}
  \end{minipage}
  \caption{Visual comparison between \blank{results produced by} SPARNetHD and GFRNet. Better zoom in to see the details.}
  \label{fig:sparhd-comp-gfrnet}
\end{figure}

\section{Conclusion}
We propose a SPatial Attention Residual Networks (SPARNet) for very low resolution face super-resolution. SPARNet is composed \blank{by stacking} Face Attention Units (FAUs), which extend vanilla residual block with a spatial attention branch. The spatial attention mechanism allows the network to \blank{pay less attention on the less feature-rich regions.}
\blank{This} makes the training of SPARNet more effective and efficient. Extensive experiments with various kinds of metrics illustrate the advantages of SPARNet over current state-of-the-arts. We further extend SPARNet to SPARNetHD with more channel numbers and multi-scale discriminator networks. SPARNetHD trained on synthetic datasets is able to generate realistic and \blank{high resolution} outputs (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, $512\times512$) for LR face images. Quantitative and qualitative comparisons with other methods indicate that the proposed spatial attention mechanism is beneficial to restore texture details of LR face images. We also demonstrate \blank{that} SPARNetHD can \blank{generalize} well to real world LR faces, \blank{making it highly practical and applicable}.

\section*{Acknowledgments}

The authors would like to thank Tencent AI Lab for supporting this research. The work of Kwan-Yee K. Wong was supported by a grant from the Research Grant Council of the Hong Kong (SAR), China, under the Project HKU 17203119.

\ifCLASSOPTIONcaptionsoff
  \newpage
\fi


\section{Introduction}

Weather prediction at a high spatial resolution is valuable and often essential for many applications including flood prediction and water management, renewable energy production, and wildfire forecasting. However, its production using traditional approaches (e.g., numerical weather prediction models) is computationally expensive. Recently, researchers have been exploring the use of machine learning techniques to downscale coarser resolution weather and climate simulations to finer resolution grids. Broadly speaking, four machine learning methods for have been described in the literature: Regression based methods, analog methods, auto-regression methods, and deep neural network super resolution based methods (e.g., \cite{Larraondo2017, Mouatadid2017, Coulibaly2005, He2016, Rodrigues2018, Medina2018, Rebora2006}. The latter method encompasses convolutional and generative adversarial neural networks for super resolution.

The motivation for this work stems from The Jefferson Project at Lake George, a multiyear research initiative based at Lake George, New York, USA that has a goal of understanding the impact of human activity on fresh water, and how to mitigate those effects. A recent discovery (see \cite{AugerSubmitted}) is that hydrodynamic simulations of Lake George are considerably more accurate when forced with high-resolution weather model data (<1 km horizontal resolution). A particularly striking result is that after one month of simulation, the hypolimnetic volume of the lake reduces by 15\% when forced with weather information generated at 0.33 km resolution -- in line with observations -- compared to 3 km. In other words, the livable zone for cold-water fish reduces by 15\%, which has significant implications for understanding the impact of a warming climate on lake biological activity.

Given the apparent importance of high-resolution weather data to accurately model the long-term changes of mid- and small-sized lakes, and given the exorbitant computational expense (and carbon footprint) of generating such high-resolution simulations with a physics-based weather model, we hereby examine the feasibility of using two super-resolution models to generate high-resolution inputs from coarse resolution weather model data. We strive for a resolution increase of up to 9x, more than most recent ML-based downscaling studies (e.g., \cite{Manepalli2020}). Successful implementation of such algorithms will enable the generation of high-resolution weather information at significantly lower computational cost. This, in turn, may spur a deeper, global understanding of local weather impacts on vulnerable ecosystems in a changing climate.

The manuscript presents early results for precipitation only, although other weather variables (wind, downward radiation, humidity) are equally important for coupling with hydrodynamic simulations. It builds on previous attempts at downscaling using tree-based regression models trained at every grid cell (\cite{WatsonLynar2020}), which were found to be inferior to super-resolution algorithms.

\section{Data and Methods}

All precipitation data used in this study was generated by version 3.8.1 of the Advanced Research version of the Weather Research and Forecasting (WRF) Model (\cite{Skamarock08}). Daily, operational weather forecasts for Lake George have been performed for Lake George with four (one-way coupled) domains with horizontal resolutions of 9 km, 3 km, 1 km and 0.33 km (Fig. \ref{fig:Domains}). Each domain has 42 vertical levels (with 14 below 1 km), a 50-hPa model top, and a 5 km-deep upper-level absorbing layer. Further details can be found in \citet{AugerSubmitted}.

\begin{wrapfigure}{r}{0.5\textwidth}
 \begin{center}
 \includegraphics[width=0.48\textwidth]{Domains.png}
 \end{center}
 \caption{WRF model domains in northeast USA (Lake George is outlined in red). This manuscript uses WRF data from the three outer domains with 9, 3 and 1 km grid resolution.}
 \label{fig:Domains}
\end{wrapfigure}

We use hourly WRF data from 2017-01-01 to 2020-03-20 (with an 80\%/20\% split at 2019-08-01) in an effort to reconstruct the simulated 1 km WRF data with 3 km and 9 km WRF data. In other words:

\begin{itemize}

\item Experiment 1: Feature values are 3 km WRF variables; target variable is 1 km WRF precipitation

\item Experiment 2: Feature values are 9 km WRF variables; target variable is 1 km WRF precipitation

\end{itemize}

The WRF variables used in this study are hourly accumulated surface precipitation, and instantaneous 2 m temperature, u- and v- 10 m winds, 2 m specific humidity and surface pressure. All variables are output at hourly intervals.

\subsection{Downscaling with ESRGAN}

The enhanced super-resolution generative adversarial networks (ESRGAN) is a deep neural network approach which was initially created for use with natural images which typically have no inherent resolution. Image super-resolution models have received increased application in the physical sciences, with \citet{Manepalli2020} finding the ESRGAN has encouraging transferable and generalizable properties (and can even reproduce the simulated power spectral density of near-surface winds).

We employed the SRResNet structure with the Residual-in-Residual Dense Block (RRDB) as basic blocks, as proposed by \citet{Wang2019} paper. We used the default training parameters mentioned in ESRGAN, except with 3x- instead of 2x-upsampling blocks. In order to optimize for the prediction accuracy rather than visual quality, the training is PSNR-oriented without adding VGG or GAN loss (as per the recommendation in \citet{Wang2019}).

\subsection{Downscaling with Content Adaptive Resampler}

Recently, \citet{sun2020} proposed a new content adaptive-resampler (CAR) based image downscaling method. Briefly, the resampler network generates content adaptive image resampling kernels which are applied to the original high-resolution input to generate pixels on the downscaled image. This resampler is different to traditional bilinear downscaling as it is dynamic and uses meta-learning that in turn utilises filters similar to dynamic filter networks. We employed the CAR model implemented with Pytorch using the default configurations as proposed by \citet{sun2020}.

\section{Preliminary Results}

The ESRGAN systematically outperformed the CAR model, presenting accurate reconstructions well beyond expectations. The long timeseries of WRF data ensures ample training and testing examples, with the 80/20\% train/test split providing almost 8 months of continuous data for testing the skill of the models. Overall, the mean absolute error (MAE) is shown in Table~\ref{tab:MAE} along with that obtained via standard bicubic interpolation (a simple baseline model).

\begin{table}[h]
    \centering
    \caption{Mean absolute error using different methods. The output grid resolution is 1 km, and the input grid resolution is listed in the table. The ESRGAN model has the best performance, followed by the CAR model. Both of the proposed models outperform the baseline bicubic interpolation.}
    \begin{tabular}{cccc}
        \toprule
          \multirow{2}{*}{Input grid resolution} & \multicolumn{3}{c}{Mean absolute error (mm/hr)} \\
          \cmidrule(r){2-4}
         & Bicubic & ESRGAN & CAR \\
        \midrule
        3 km & 0.83 & 0.040 & 0.384\\
        9 km & 1.98 & 0.078 & 0.931\\
        \bottomrule
    \end{tabular}
    \label{tab:MAE}
\end{table}

Specific events were extremely well captured by the ESRGAN (e.g., Fig. \ref{fig:4panel}), and although it did systematically underestimate precipitation by up to 0.1 mm/hr (not shown), it motivates us to continue investigation with this model. In contrast, the CAR method retained many visual features of the coarser resolution precipitation input; perhaps additional testing with different configurations can improve results.

\begin{figure}[h]
  \centering
  \includegraphics[width=\textwidth]{Figures/4_panel.png}
  \caption{Example of precipitation reconstructions by the ESRGAN and CAR models from 9 km to 1 km horizontal resolution. The panels show accumulated precipitation (mm/hr) in the hour preceding 2019-11-01 04:00 UTC. (a) Original WRF precipitation from the 9 km domain; (b) original WRF precipitation from the 1 km domain (target); reconstruction by ESRGAN to 1 km resolution; and (d) reconstruction by the CAR model to 1 km resolution}
  \label{fig:4panel}
\end{figure}

\section{Discussion and Broader Impacts}

This study has demonstrated the utility of two super-resolution methods to downscale precipitation in the northeast US. The ESRGAN outperformed the CAR method, especially when downscaling from 9 km to 1 km precipitation. The significant 9x super-resolution is particularly encouraging because the WRF model uses different physics to simulate precipitation at 9 km resolution vs 1 km (we used the Grell-Freitas cumulus scheme in the 9 km domain, whereas cumulus-scale precipitation is assumed to be explicitly simulated in the 1 km domain).

We are excited for the potential impact of this work. An accurate and efficient downscaling method reduces time spent running expensive computational code (and thus reduce the carbon footprint) and provides a scalable approach for (global) climate impact assessments.

Further investigation has been identified:

\begin{itemize}

\item Expand analysis to include additional variables, specifically near surface winds, surface humidity and downward short and long wave radiation. These are key inputs to hydrodynamic models.

\item Apply ESRGAN to downscale GFS data (which provides boundary conditions to the WRF model) at ~25 km resolution to the 1 km WRF domain. This is a particularly demanding task given the GFS model has different model parameterizations and dynamical core.

\item How transferable and generalizable is this approach? \citet{Manepalli2020} showed some success in this regard using ESRGAN to downscale winds, and is key to moving to climatic geographies.

\item How much data is required to adequately train the model? Here, we provided over nearly 2.5 years of hourly data for training; a prohibitive amount of simulation for larger geographic areas. Can we leverage any correlations across time to improve the super-resolution (e.g., low resolution sequence in, high resolution sequence out)?

\end{itemize}

Finally, we do not believe the data is biased nor do we expect people to be placed at a disadvantage from this research. That said, flaws in the super-resolution system could result in inaccurate weather and climate predictions with net-negative consequences for policy development and decision making.

\begin{ack}

We gratefully acknowledge assistance by personnel associated with the Jefferson Project at Lake George, a collaboration between Rensselaer Polytechnic Institute, IBM Research and the FUND for Lake George. In addition to funding from the three partners, the sensor network has been partially supported by a NSF MRI grant (\#1655168) and a New York State Higher Education Capital grant (\#7290).

\end{ack}

\medskip

\small


\section{Previous Work and Motivation}
\label{sec-1}

Automated authorship attribution has a long history (starting from the early 20th century \cite{mendenhall1901mechanical}) and has since then been
extensively studied and elaborated upon. The problem of authorship identification is based on the assumption
that there are stylistic features that can help distinguish the real author of a text from any other theoretical author.
This said set of stylistic features was recently defined as a linguistic fingerprint
(or stylome), which can be measured, is largely unconscious, and is constant \cite{van2005new}.
One of the oldest studies to propose an approach to this problem is on the issue of the \emph{Federalist Papers},
in which \cite{mosteller1963inference} try to determine the real author of a few of these papers which have
disputed paternity. This work remains iconic in the field, both for introducing a standard dataset and for
proposing an effective method for distinguishing between the authors' styles, that is still relevant to this day,
based on function words frequencies. Many other types of features have been proposed and successfully used
in subsequent studies to determine the author of a text. These types of features generally contrast with the
content words commonly used in text categorization by topic, and are said to be used unconsciously and harder
to control by the author. Such features are, for example, grammatical structures \cite{baayen1996outside},
part-of-speech n-grams \cite{koppel2003exploiting}, lexical richness \cite{tweedie1998variable}, or even the more general
feature of character n-grams \cite{kevselj2003n,dinu2008authorship}. Having applications that go beyond
finding the real authors of controversial texts, ranging from plagiarism detection to forensics to security,
stylometry has widened its scope into other related subtopics such as author verification (verifying whether a text
was written by a certain author) \cite{koppel2004authorship}, author profiling (e.g. gender or age prediction),
author diarization, or author masking (given a document, paraphrase it so that the original style does not match that of its original author anymore).

In this work we attempt to explore a slightly different issue, strongly related with the stylistic fingerprint of an
author, namely if an author preserves his stylome across different time periods, and across political and cultural environments.
More specifically, we want to see if we can discriminate between texts written by the same author under a communist regime, as compared to under democracy.

For this analysis, we chose the works of Solomon Marcus (1925-2016), one of the most prominent scientists and men of culture of modern Romania,
and a very prolific author whose work spans a large period of time, with a consistent amount of work both before and after the fall of communism in Romania.
As a scientist, he was active in an impressive range of different fields, such as mathematics, computer science, mathematical and computational linguistics,
semiotics etc, and published over 50 books translated in more than 10 languages,
and about 400 research articles in specialized journals in almost all European countries.
He is one of the initiators of mathematical linguistics \cite{marcus1971introduzione} and of mathematical poetics \cite{marcus1973mathematische},
and has been a member of the editorial board of tens of international scientific journals covering all
his domains of interest. As a man of culture,
he has written an equally impressive amount of texts, having as preferred topics mathematical education, culture, science, children, etc.
His wide interests and complex personality have left deep imprints in the Romanian scientific and cultural world.

We chose these works and this period in Romanian history as a particularly interesting study case for stylistic variation.
Marcus' essays \cite{marcus2012rani} were written over a period spanning almost half a century:
22 years of communist regime and 27 years of democracy (after the fall of communism in Romania in 1989).
The significant changes in Romania's cultural life with the fall of communism motivate such an analysis beyond a simple temporal
analysis of texts: in the pre-democracy period, the communist norms demanded an elaborated writing style,
whereas the list of officially approved topics was much more limited.
On the other hand, with the fall of communism, we expect that the author would
would be able to radically change his style to express his ideas freely.

Previous work on the same corpus \cite{dinu-dinu-dumitru:2017:RANLP} focused on a lexical quantitative analysis of Marcus' works,
uncovering some differences in word usage between the communist and the democracy period. We continue this work by performing
more robust analyses using various metrics and classification experiments to show that the texts written during the two periods are indeed
distinguishable with a reasonable accuracy.

\section{The Corpus}
\label{sec-2}

For the purposes of the analysis presented here, we used a collection of Solomon Marcus' non-scientific texts.
The whole collection of Marcus’ non-scientific publications is available in print,
in six volumes entitled "Răni deschise" (\emph{Open wounds}), of which we analyze the first four: two containing texts written
before the fall of communism, and the other two with texts written under democracy.
The first volume comprises 1256 pages of texts, conferences and interviews, from 2002 to 2011 (and a few published before 1990).
The second volume "Cultură sub dictatură" (\emph{Culture under dictatorship}) of 1088 pages and the third "Depun Mărturie" (\emph{I testify}), of 668 pages,
is a collection of texts published between 1967 and 1989. The fourth volume "Dezmeticindu-ne" (\emph{Awaking}), of 1030 pages, contains texts from
 the period immediately following the Romanian Revolution (December 1989) to the year 2011.

Before the texts could be used for automatic analysis, some pre-processing was involved, including extracting the texts from PDF files into
text form, and splitting the corpus into individual essays.
We also parsed the table of contents in order to label each text with its year of publication. Additionally, we excluded some texts
that would interfere with the experiments, such as texts in languages different than Romanian (most of the texts are in Romanian),
and interviews (which also contain lines of the interviewer, that should not be considered when analysing text authored by Solomon Marcus).

\section{Methodology}
\label{sec-3}

In order to analyze the style of Marcus' work and its variation, we perform a series of classification and clustering experiments, as well as a temporal
analysis of the evolution of the author's style. We look at various features to capture both style and semantic content, with a focus on metrics that
that characterize the stylistic fingerprint of the author.

We perform various classification and clustering experiments, as well as visualisations complementing them, for which we experiment first with simple bag of words models.
To capture the stylistic fingerprint of the author, we resort to one of the most commonly accepted indicators of personal style of an author,
namely the distribution of functional words, such as pronouns, determiners, prepositions, etc, which were successfully used in many previous studies to solve authorship attribution problems.
For comparison, we perform a parallel experiment where we use as features
only the content words, assuming they are descriptive of the semantic content of the texts, as opposed to their style.

Aside from these feature sets, we also use more sophisticated features in some additional classification experiments: various stylistic metrics (such as readability or
lexical richness) to capture the stylistic variation, as well as features resulted from topic modelling to capture the semantic content. These features will be described
in more detail in the chapters dedicated specifically to the classification experiments.

\subsection{Clustering Experiments}
\label{sec-3-1}

Processing the text involves extracting all function words from each essay, and representing each text as a vector of frequencies of function words.
We use a curated list of 120 functional Romanian words \cite{dinu2012pastiche}.
On these vectors we then apply similarity metrics in order to cluster texts based on their stylistic profile.
The distance we have chosen is rank distance \cite{dinu2003classification}, \cite{popescu2008rank},
an ordinal distance which was successfully previously used in other problems of authorship \cite{dinu2008authorship}, \cite{dinu2012pastiche}.

For our purposes, we applied a hierarchical clustering algorithm, and plotted the resulted dendrogram, as shown in Figure 1.
Texts from each volume are marked with the prefix \emph{RD<volume\_number>}.
The intention is to see whether texts from the same period are clustered together: in this case, we would expect to see texts
from volumes 1 and 4 (communism) appear close together, and texts from volumes 2 and 3 (post-communism) appear in a separate group.

An additional processing step we performed before applying the clustering algorithm was to group the essays into longer texts, while
keeping them separated into volumes (texts from different volumes/periods were not grouped together). For each volume,
batches of 10 essays were grouped toghether in 1 text.
This results in a fewer number of texts, which can be more easily plotted on a dendrogram, and additionally should contain less
noise than the very short original texts.

\begin{figure}[htb]
\centering
\includegraphics[width=1.0\linewidth]{marcus_stylome.png}
\caption{Style dendrogram: Hierarchical clustering of function word vectors}
\end{figure}

\subsection{Dimensionality Reduction and Visualization}
\label{sec-3-2}

In addition to the clustering experiments, we perform an experiment where we plot the essays in the 4 volumes in 2-D space,
by initially applying dimensionality reduction (principal component analysis) on the texts, represented as function word frequency
vectors, as described above. The resulted plot is shown in Figure 2.

\begin{figure}[htb]
\centering
\includegraphics[width=1.0\linewidth]{marcus_stopwords_PCA_new.png}
\caption{Style space: 2-D visualization of function word vectors}
\end{figure}

In a separate experiment, we take a brief look at the content profile of the texts (as opposed to the stylistic one), by performing the same
analysis on content words instead of function words. We extract all content words from all the (grouped) texts, and represent the texts as
vectors of content words frequencies. We then apply PCA on the resulted vector space, and plot the 2-dimensional representations of
the texts, as shown in Figure 3. Although this is still a very crude analysis, the result should provide an insight into the content similarities and dissimilarities
between the texts in the 4 volumes.

We also perform a version of the clustering experiments on the same vectors of content words, with the same methodology that was applied in the
stylistic analysis: the rank distance similarity is applied on the word vectors, and the distances are used in a hierarchical clustering algorithm,
which results in the "content" dendrogram in Figure 4.

\begin{figure}[htb]
\centering
\includegraphics[width=1.0\linewidth]{marcus_topic_grouped2.png}
\caption{Topic dendrogram: Hierarchical clustering on content word vectors}
\end{figure}

\begin{figure}[htb]
\centering
\includegraphics[width=1.0\linewidth]{marcus_contentwords_PCA_new.png}
\caption{Topic space: 2-D visualization of content word vectors}
\end{figure}

\subsection{Classification Experiments}
\label{sec-3-3}

In order to have a more conclusive result regarding the differences between the texts written in the two epochs and the levels at which
they occur, we also performed some classification experiments where we try to predict for each essay if it was written before or
after the fall of communism, as well as to which specific volume it pertains.

For classifying the texts we used an SVM classifier with a linear kernel, and tested its performance in a series of leave-one-out experiments.

\subsubsection{Features}
\label{sec-3-3-1}

We used various sets of features, in turn specific to style and to semantic content, in order to conclusively characterize the nature of the
evolution of Marcus' writing over time and across the two periods.

As stylistic features, we experimented with 2 setups: first a bag-of-words model using only functional words; and secondly a set of stylistic markers:

\begin{itemize}
\item \textbf{Automated Readability Index}, as defined in \cite{senter1967automated}, computed as:
\end{itemize}
$$ ARI = 4.71\frac{c}{w} + 0.5\frac{w}{s} - 21.43 $$
where \emph{c}, \emph{w} and \emph{s} represent, respectively, total number of characters, words and sentences in the text

\begin{itemize}
\item \textbf{Lexical richness}, computed as the number of unique lemmas divided by the total number of tokens in the text:
\end{itemize}

$$ LR = \frac{number\ of\ unique\ lemmas}{total\ number\ of\ tokens} $$

\begin{itemize}
\item \textbf{Average word length}
\item \textbf{Average sentence length}
\end{itemize}

To measure the variation in semantic content, we performed two classification experiments. In the first experiment we used a simple bag-of-words model, this time using only content words
as features. For the second experiment, we used topic modelling to extract topics in each article, that we then used as features in the classifier. To extract the topics,
we used a Latent Dirichlet Allocation model, configured to generate 3 latent topics for each document.

Results are shown in Table 1 and Table 2: in Table 1 we report the results of trying to classify the texts into the two distinct periods, while Table 2 shows the
results of classifying each text into the specific volume to which it pertains.

\subsection{Temporal variation of stylistic metrics}
\label{sec-3-4}

In order to show that the stylistic variation discovered in the experiments described in the previous chapter can indeed be attributed to the change of
political regime, and not simply to the passing of time and the natural gradual change in the author's style, we also look at how the various stylistic metrics
vary over time at a finer level.

Figures 5 to 8 show the evolution of each stylistic metric described in the previous chapter from year to year. To achieve this, we computed each metric for every article
in the corpus, then for each year we averaged the values obtained for the articles written during that year. To reduce the effect of noise and variation in number
of articles for each year, we smooth the obtained values by
applying a moving average over the articles in each year with a window of 10 years.

In order to confirm the variation in style is more pronounced around the year 1990 (that marks the beginning of democracy in Romania),
we performed a series of experiments in which we attempted temporal splits of the article set into two periods using each year as a separator,
and computed the statistical significance of the difference shown at the level of each metric for each of these splits. We assume that the more significant
the split across a certain year is, the more radical the author's change in style around that year, and compare the effect of the particular year that marked the beginning
of democracy in Romania to all of the other years during which Marcus wrote his essays. We plot, in addition to the actual values of the stylistic
metrics for each year, the p-values corresponding to the statistical significance of the difference for each metric between the articles written before and after each year.
The vertical line over the year 1990 splits the plot into two parts, corresponding to the period before respectively after the fall of communism.

\begin{figure}[htb]
\centering
\includegraphics[width=1.0\linewidth]{readability_evolution.png}
\caption{Evolution of readability over time}
\end{figure}

\begin{figure}[htb]
\centering
\includegraphics[width=1.0\linewidth]{lexical_richness_evolution.png}
\caption{Evolution of lexical richness over time}
\end{figure}

\begin{figure}[htb]
\centering
\includegraphics[width=1.0\linewidth]{avg_wlen_evolution.png}
\caption{Evolution of average word length over time}
\end{figure}

\begin{figure}[htb]
\centering
\includegraphics[width=1.0\linewidth]{avg_senlen_evolution.png}
\caption{Evolution of average sentence length over time}
\end{figure}

\section{Results and Analysis}
\label{sec-4}

The clustering experiments, as well as the 2-dimensional visualizations, show a clear distinction between the works of the two periods.
The style dendrogram in Figure 1 shows that texts tend to be grouped according to the volume they belong to, but moreover,
the volumes belonging to each period tend to be grouped together.
It is noticeable that texts belonging to the first and fourth volumes, written during the post-communist epoch (marked "RD1" and "RD4"),
are grouped together on the lower branch of the dendrogram, and the same happens with texts in volumes 2 and 3 ("RD2" and "RD3"), written
during communism, which are grouped on the upper branch.

The same tendency is visible in the 2-dimensional vector space plot of the texts. It is interesting that the separation
between the pre-communist texts (marked with green and blue points) and communist texts (marked with red and yellow points)
seems even more prominent than the separation between each of the four volumes - indicating that the main variable contributing
to style differences is indeed the period when the text was written.

A better, more detailed understanding of how exactly Marcus' style evolves can be gained from the
plots showing the variation in style from year to year. Overall, after the change of regime in 1990, we can see a slight drop in all of the stylistic features,
especially in word length and lexical richness. Most of the stylistic metrics (especially lexical richness and readability) show a more pronounced change of style right after the year 1990 - the year of the fall of the communist regime in Romania - suggesting that the change of regime did have an effect of its own on the style of the author.
An even more interesting phenomenon can be observed by looking at the variation of the p-values for splitting the article set into two arbitrary periods (before and
after each year). Except from average sentence length, for which the most drastic change seems to happen around the year 1980, for all of the other stylistic metrics
splitting the articles set into texts written under communism and under democracy is statistically significant (with p-values well under 0.05 for the year 1990,
and additionally having at least local minima around this year).

We can conclude from these results that, independently on possible variation at other levels (topic, etc),
there is inarguable variation between the texts at a stylistic level.

A similar but less obvious effect is noticeable in the content analysis. While the vector space plot of the texts again shows
a reasonably clear distinction between the texts in the two periods, the cluster analysis is less conclusive.

\begin{table}[htb]
\centering
\begin{tabular}{ll}
Feature set & Precision\\
\hline
function words (style) & 75.80\%\\
stylistic metrics (style) & 73.12\%\\
content words (semantic content) & 74.46\%\\
topic modelling (semantic content) & 71.24\%\\
\end{tabular}\caption{\label{results1}Results of classifying texts into periods}

\end{table}

\begin{table}[htb]
\centering
\begin{tabular}{ll}
Feature set & Precision\\
\hline
function words & 61.29\%\\
content words & 84.13\%\\
\end{tabular}\caption{\label{results2}Results of classifying texts into volumes}

\end{table}

The results of the classification experiments very clearly show that essays can be easily separated into the two classes corresponding to communism and democracy,
both when using function words, stylistic metrics, and semantic features (content words, topics) as a feature set. A slightly better accuracy is obtained
for stylistic classification.
The stylistic features, in comparison to the content words, seem to be more successful
at predicting the period rather than the specific volume, indicating that style may indeed be the aspect of the texts that varies
the most between the two periods and political regimes.
Analyses of the author's change in style,
as well as statistical significance tests of the change in various stylistic metrics across the two periods, further confirm that
the change of political regime does have its own effect on the author's style.

We can conclude, given all of these results, that during the two periods, corresponding
to two different political regimes, Marcus produced works which are clearly distinct both stylistically and from the
point of view of the topics discussed. It is noteworthy that the classifier manages not only to classify the texts into
the two periods in which they were written, but is also successful at predicting the exact volume to which each essay pertains.
This suggests there may be other (volume-specific) factors contributing to the distinctiveness of the essays in each class, which
it may be interesting to further explore.

\section{Conclusions and Future Directions}
\label{sec-5}

We have proposed in this paper an analysis of the stylistic variation of the works of Solomon Marcus
between two distinct historical periods, and have presented results that show the separation
between the two periods is indeed clear at the level of the author's style.
The clustering approach based on the preference of the author regarding the functional words
shows that the functional words were unconsciously used by author in a different way in communism than in democracy period,
and we can use them to discriminate between the communist and post-communist texts of the given author.
We further confirm in several classification experiments that the texts can be automatically separated into
the two periods, and are distinguishable both at the level of style and topic.
We show the contribution of the change in political regime to the stylistic variation by looking at how the author's
style gradually changes over time, and by performing statistical significance tests on various stylistic metrics, comparing the two periods.

In future work, it would be very interesting to examine more closely how this variation is manifested.
From the stylistic point of view: what are the exact changes that occur with the passing of time and
the change of historical and cultural context?
From the point of view of topic modelling, what are the specific changes in the topic of the text
corresponding to the two periods?
Do these changes match our intuitions about writing in the communist as compared to post-communist era?

Finally, continuing this analysis on other authors with works in both communism and post-communism, or, more generally,
across various political regimes, would be interesting to confirm the phenomenon is not unique to this author. Possibly extending this to
other authors in different cultural contexts could lead to an interesting analysis on the external factors
that influence style.

\section*{Acknowledgement}
\label{sec-6}
Liviu P. Dinu is supported by UEFISCDI, project number 53BG/2016.

\bibliographystyle{splncs03}