summaries: This work uses GANs to recover clean faces from occluded counterparts. The effectiveness of the proposed method is verified qualitatively and quantitatively on CelebA-HQ. The proposed framework can be generalized to several face-related tasks, such as unconstrained face recognition. Although the novelty of the method is not really impressive, the proposed method seems to be useful for face-related applications and the experimental results are convincing to me. The paper proposes a complex generative framework for image completion (particularly human face completion). It aims at solving the following challenges: 1) complete the human face at both low and high resolution; 2) control the attribute of the synthetic content; 3) without the need of complex post-processing. To achieve so, this paper proposes a progressively attentive GAN to complete face image at high resolution with multiple controllable attributes in a single forward pass without post-processing. Particularly it introduces a frequency-oriented attentive module (FAM) to attend on finer details.  
abstractText: Face completion is a challenging task with the difficulty level increasing significantly with respect to high resolution, the complexity of “holes" and the controllable attributes of filled-in fragments. Our system addresses the challenges by learning a fully end-to-end framework that trains generative adversarial networks (GANs) progressively from low resolution to high resolution with conditional vectors encoding controllable attributes. We design a novel coarse-to-fine attentive module network architecture. Our model is encouraged to attend on finer details while the network is growing to a higher resolution, thus being capable of showing progressive attention to different frequency components in a coarse-to-fine way. We term the module Frequency-oriented Attentive Module (FAM). Our system can complete faces with large structural and appearance variations using a single feed-forward pass of computation with mean inference time of 0.54 seconds for images at 1024× 1024 resolution. A pilot human study shows our approach outperforms state-of-the-art face completion methods. The code will be released upon publication.
references.authors: Connelly Barnes Eli Shechtman Adam Finkelstein Dan B Goldman
references.title: Patchmatch: A randomized correspondence algorithm for structural image editing
references.venue: ACM Trans. Graph.,
references.authors: Marcelo Bertalmio Guillermo Sapiro Vincent Caselles Coloma Ballester
references.title: Image inpainting
references.venue: In Proceedings of the 27th annual conference on Computer graphics and interactive techniques,
references.authors: Marcelo Bertalmio Luminita Vese Guillermo Sapiro Stanley Osher
references.title: Simultaneous structure and texture image inpainting
references.venue: IEEE Transactions on image processing,
references.authors: Adrian Bulat Georgios Tzimiropoulos
references.title: How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks)
references.venue: In International Conference on Computer Vision,
references.authors: Wengling Chen James Hays
references.title: Sketchygan: Towards diverse and realistic sketch to image synthesis
references.venue: arXiv preprint arXiv:1801.02753,
references.authors: Xi Chen Yan Duan Rein Houthooft John Schulman Ilya Sutskever Pieter Abbeel
references.title: Infogan: Interpretable representation learning by information maximizing generative adversarial nets
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Yunjey Choi Minje Choi Munyoung Kim Jung-Woo Ha Sunghun Kim Jaegul Choo
references.title: Stargan: Unified generative adversarial networks for multi-domain image-to-image translation
references.venue: arXiv preprint arXiv:1711.09020,
references.authors: Antonio Criminisi Patrick Perez Kentaro Toyama
references.title: Object removal by exemplar-based inpainting
references.venue: In Computer Vision and Pattern Recognition,
references.authors: Soheil Darabi Eli Shechtman Connelly Barnes Dan B Goldman Pradeep Sen
references.title: Image melding: Combining inconsistent images using patch-based synthesis
references.venue: ACM Trans. Graph.,
references.authors: Emily Denton Sam Gross Rob Fergus
references.title: Semi-supervised learning with context-conditional generative adversarial networks
references.venue: arXiv preprint arXiv:1611.06430,
references.authors: Emily L Denton Soumith Chintala Rob Fergus
references.title: Deep generative image models using a laplacian pyramid of adversarial networks
references.venue: In Advances in neural information processing systems,
references.authors: Alexei A Efros Thomas K Leung
references.title: Texture synthesis by non-parametric sampling
references.venue: In Computer Vision,
references.authors: Pablo Garrido Levi Valgaerts Ole Rehmsen Thorsten Thormahlen Patrick Perez Christian Theobalt
references.title: Automatic face reenactment
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville Yoshua Bengio
references.title: Generative adversarial nets
references.venue: In Advances in neural information processing systems,
references.authors: Karol Gregor Ivo Danihelka Alex Graves Danilo Jimenez Rezende Daan Wierstra
references.title: Draw: A recurrent neural network for image generation
references.venue: arXiv preprint arXiv:1502.04623,
references.authors: James Hays Alexei A Efros
references.title: Scene completion using millions of photographs
references.venue: In ACM Transactions on Graphics (TOG),
references.authors: Jia-Bin Huang Sing Bing Kang Narendra Ahuja Johannes Kopf
references.title: Image completion using planar structure guidance
references.venue: ACM Transactions on Graphics (TOG),
references.authors: Satoshi Iizuka Edgar Simo-Serra Hiroshi Ishikawa
references.title: Globally and locally consistent image completion
references.venue: ACM Transactions on Graphics (TOG),
references.authors: Sergey Ioffe Christian Szegedy
references.title: Batch normalization: Accelerating deep network training by reducing internal covariate shift
references.venue: In International Conference on Machine Learning,
references.authors: Phillip Isola Jun-Yan Zhu Tinghui Zhou Alexei A Efros
references.title: Image-to-image translation with conditional adversarial networks
references.venue: arXiv preprint arXiv:1611.07004,
references.authors: Justin Johnson Alexandre Alahi Li Fei-Fei
references.title: Perceptual losses for real-time style transfer and super-resolution
references.venue: In European Conference on Computer Vision,
references.authors: Takuhiro Kaneko Kaoru Hiramatsu Kunio Kashino
references.title: Generative attribute controller with conditional filtered generative adversarial networks
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Tero Karras Timo Aila Samuli Laine Jaakko Lehtinen
references.title: Progressive growing of gans for improved quality, stability, and variation
references.venue: arXiv preprint arXiv:1710.10196,
references.authors: Diederik Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: arXiv preprint arXiv:1412.6980,
references.authors: Nikos Komodakis
references.title: Image completion using global optimization
references.venue: In Computer Vision and Pattern Recognition,
references.authors: Vivek Kwatra Arno Schödl Irfan Essa Greg Turk Aaron Bobick
references.title: Graphcut textures: image and video synthesis using graph cuts
references.venue: In ACM Transactions on Graphics (ToG),
references.authors: Anat Levin Assaf Zomet Yair Weiss
references.title: Learning how to inpaint from global image statistics
references.venue: In null,
references.authors: Kai Li Feng Xu Jue Wang Qionghai Dai Yebin Liu
references.title: A data-driven approach for facial expression synthesis in video
references.venue: In Computer Vision and Pattern Recognition (CVPR),
references.authors: Yijun Li Sifei Liu Jimei Yang Ming-Hsuan Yang
references.title: Generative face completion
references.venue: arXiv preprint arXiv:1704.05838,
references.authors: Guilin Liu Fitsum A Reda Kevin J Shih Ting-Chun Wang Andrew Tao Bryan Catanzaro
references.title: Image inpainting for irregular holes using partial convolutions
references.venue: arXiv preprint arXiv:1804.07723,
references.authors: Andrew L Maas Awni Y Hannun Andrew Y Ng
references.title: Rectifier nonlinearities improve neural network acoustic models
references.venue: In Proc. ICML,
references.authors: Mehdi Mirza Simon Osindero
references.title: Conditional generative adversarial nets
references.venue: arXiv preprint arXiv:1411.1784,
references.authors: Alejandro Newell Kaiyu Yang Jia Deng
references.title: Stacked hourglass networks for human pose estimation
references.venue: In European Conference on Computer Vision,
references.authors: Deepak Pathak Philipp Krahenbuhl Jeff Donahue Trevor Darrell Alexei A Efros
references.title: Context encoders: Feature learning by inpainting
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Patrick Pérez Michel Gangnet Andrew Blake
references.title: Poisson image editing
references.venue: In ACM Transactions on graphics (TOG),
references.authors: Olaf Ronneberger Philipp Fischer Thomas Brox
references.title: U-net: Convolutional networks for biomedical image segmentation
references.venue: In International Conference on Medical Image Computing and ComputerAssisted Intervention,
references.authors: Olga Russakovsky Jia Deng Hao Su Jonathan Krause Sanjeev Satheesh Sean Ma Zhiheng Huang Andrej Karpathy Aditya Khosla Michael Bernstein
references.title: Imagenet large scale visual recognition challenge
references.venue: International Journal of Computer Vision,
references.authors: Patsorn Sangkloy Jingwan Lu Chen Fang Fisher Yu James Hays
references.title: Scribbler: Controlling deep image synthesis with sketch and color
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Ashish Shrivastava Tomas Pfister Oncel Tuzel Josh Susskind Wenda Wang Russ Webb
references.title: Learning from simulated and unsupervised images through adversarial training
references.venue: arXiv preprint arXiv:1612.07828,
references.authors: Karen Simonyan Andrew Zisserman
references.title: Very deep convolutional networks for large-scale image recognition
references.venue: arXiv preprint arXiv:1409.1556,
references.authors: Justus Thies Michael Zollhöfer Marc Stamminger Christian Theobalt Matthias Nießner
references.title: Face2face: Real-time face capture and reenactment of rgb videos
references.venue: In Computer Vision and Pattern Recognition (CVPR),
references.authors: Dmitry Ulyanov Andrea Vedaldi Victor Lempitsky
references.title: Instance normalization: The missing ingredient for fast stylization
references.venue: arXiv preprint arXiv:1607.08022,
references.authors: Ting-Chun Wang Ming-Yu Liu Jun-Yan Zhu Andrew Tao Jan Kautz Bryan Catanzaro
references.title: Highresolution image synthesis and semantic manipulation with conditional gans
references.venue: arXiv preprint arXiv:1711.11585,
references.authors: Yonatan Wexler Eli Shechtman Michal Irani
references.title: Space-time completion of video
references.venue: IEEE Transactions on pattern analysis and machine intelligence,
references.authors: Marta Wilczkowiak Gabriel J Brostow Ben Tordoff Roberto Cipolla
references.title: Hole filling through photomontage
references.venue: In BMVC,
references.authors: Wenqi Xian Patsorn Sangkloy Jingwan Lu Chen Fang Fisher Yu James Hays
references.title: Texturegan: Controlling deep image synthesis with texture patches
references.venue: arXiv preprint arXiv:1706.02823,
references.authors: Chao Yang Xin Lu Zhe Lin Eli Shechtman Oliver Wang Hao Li
references.title: High-resolution image inpainting using multi-scale neural patch synthesis
references.venue: arXiv preprint arXiv:1611.09969,
references.authors: Raymond A Yeh Chen Chen Teck Yian Lim Alexander G Schwing Mark Hasegawa-Johnson Minh N Do
references.title: Semantic image inpainting with deep generative models
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Jiahui Yu Zhe Lin Jimei Yang Xiaohui Shen Xin Lu Thomas S Huang
references.title: Generative image inpainting with contextual attention
references.venue: arXiv preprint arXiv:1801.07892,
references.authors: Jun-Yan Zhu Taesung Park Phillip Isola Alexei A Efros
references.title: Unpaired image-to-image translation using cycle-consistent adversarial networks
references.venue: arXiv preprint arXiv:1703.10593,
references.authors: Iizuka
references.title: 2017). However, the encoding process is a lossy compression, which makes it difficult to reconstruct the original contextual regions. Additionally, since much contextual information is lost during the encoding process, it is also difficult for the encoder to reconstruct content that perfectly match the context. UNet adds skip connections between the mirrored layers of the encoder and decoder, consolidating information
sections.heading: 1 INTRODUCTION
sections.text: Image completion is a technique to replace target regions, either missing or unwanted, of images with synthetic content so that the completed images look natural, realistic and appealing. Two types of methods have been used: data similarity driven methods and data distribution based generative methods. In the first paradigm, texture synthesis or patch matching are usually used (Efros & Leung, 1999; Kwatra et al., 2003; Criminisi et al., 2003; Wilczkowiak et al., 2005; Komodakis, 2006; Barnes et al., 2009; Darabi et al., 2012; Huang et al., 2014; Wexler et al., 2007). The second paradigm learns the underlying distribution governing the data generation with respect to the context and is able to synthesize novel content. Much progress (Iizuka et al., 2017; Yeh et al., 2017; Li et al., 2017; Yang et al., 2016; Denton et al., 2016; Pathak et al., 2016; Yu et al., 2018; Liu et al., 2018) has been made since the generative adversarial network (GAN) was proposed (Goodfellow et al., 2014).
  We adopt the data distribution based generative method and focus on human face completion in this paper. Three important issues are addressed. First, previous methods are only able to complete faces at low resolutions (e.g. 176 × 216 (Iizuka et al., 2017) and 256 × 256 (Yu et al., 2018)). Second, most approaches cannot control the attributes of the synthesized content. Previous methods focus on generating random realistic content. However, users may want to complete the missing parts with certain properties (e.g. expressions). Third, most existing approaches (Iizuka et al., 2017; Yeh et al., 2017; Li et al., 2017) require post processing (e.g. Poisson Blending (Pérez et al., 2003)) or complex inference process (e.g. thousands of optimization iterations (Yeh et al., 2017) or repeatedly feeding an incomplete image to CNNs at multiple scales (Yang et al., 2016)).
  To overcome the above limitations, we propose a novel progressively attentive GAN to complete face images at high resolution with multiple controllable attributes in a single forward pass without any post processing. We utilize facial landmarks as backbone guidance of face structures and propose a straightforward method of integrating them in our system. The training methodology of growing GANs progressively (Karras et al., 2017) is used to generate high-resolution images end-to-end. To avoid distorting the learned coarse structures when the network is growing to a higher resolution, we design a novel Frequency-oriented Attentive Module (FAM) to encourage the model to attend on finer details (i.e. higher-frequency structures, see Figure 1). A conditional version of our network is designed so that the appearance properties (e.g. male or female), and facial expressions of the synthesized faces can be controlled. Moreover, we design a set of loss functions inducing the network
  to blend the synthesized content with the contexts in a realistic way. Our method was compared with state-of-the-art approaches on a high-resolution face dataset CelebA-HQ (Karras et al., 2017). Both the evaluations and a pilot user study showed that our approach completed face images significantly more naturally than existing methods.
  The main contributions of this paper are: (i) We propose a progressively attentive GAN architecture, which incorporates a novel frequency-oriented attention mechanism, to complete face images with random masks in much higher resolution than existing methods. (ii) A conditional version of our model is designed to control multiple attributes of the synthesized content. (iii) Our framework is able to complete images in a single forward pass, without any post-processing, and thus fast.
sections.heading: 2 RELATED WORK
sections.text: There is a large body of image completion literature. Early non-learning based algorithms (Efros & Leung, 1999; Bertalmio et al., 2000; 2003) complete missing content by propagating information from known neighborhoods, based on low level cues or global statistics (Levin et al., 2003). Texture synthesis and patch matching based approaches (Efros & Leung, 1999; Kwatra et al., 2003; Criminisi et al., 2003; Wilczkowiak et al., 2005; Komodakis, 2006; Barnes et al., 2009; Darabi et al., 2012; Huang et al., 2014; Wexler et al., 2007) find similar structures from the context of the input image or from an external database (Hays & Efros, 2007) and then paste them to fill in the holes.
  Recent learning based methods have shown the capability of CNNs to complete large missing content. Based on existing GANs, the Context Encoder (CE) (Pathak et al., 2016) encodes the contexts of masked images to latent representations, and then decodes them to natural content images, which are pasted into the original contexts for completion. However, the synthesized content of CE is often blurry and has inconsistent boundaries. Given a trained generative model, Yeh et al. (Yeh et al., 2017) propose a framework to find the most plausible latent representations of contexts to complete masked images. The Generative Face Completion model (GFC) (Li et al., 2017) and the Global and Local Consistent model (GL) (Iizuka et al., 2017) use both global and local discriminators, combined with post processing, to complete images more coherently. Built on GL, Yu et al. (Yu et al., 2018) design a contextual attention layer (CTX) to help the model borrow contextual information from distant locations. Liu et al. (Liu et al., 2018) incorporates partial convolutions to handle irregular masks. Unfortunately, these approaches can only complete face images in relatively low resolutions (e.g. 176 x 216 (Iizuka et al., 2017) and 256× 256 (Yu et al., 2018)). Yang et al. (Yang et al., 2016) combine a global content network and a texture network, and the networks are trained at multiple scales repeatedly to complete high-resolution images (512× 512). Like the patch matching based approaches, Yang et al. assume that the missing content always shares some similar textures with the context, which is improbable for the face completion task.
  Many methods try to generate high quality images and stabilize the training process. The Laplacian GAN (Denton et al., 2015) uses a a cascade of CNNs to synthesize images from coarse to fine by generating high frequency information at different layers. The Deep Recurrent Attentive Writer (DRAW) architecture (Gregor et al., 2015), which is a spatial attention mechanism, generates images iteratively by learning to read and write parts of images at each time-step with a sequence of variational auto-encoders (VAEs). Unfortunately, these techniques are unable to synthesize high-resolution images (e.g. 64× 64 (Gregor et al., 2015; Denton et al., 2015)). Karras et al. (Karras et al., 2017) put forward a progressive training methodology (Progressive GAN) to grow GANs from low to high resolution, and are able to generate realistic 1024× 1024 images. However, since all the parameters remain trainable at the growing stage, learned coarse structures can be altered and distorted, and thus the training process of Progressive GANs is usually unstable. Note that these generative models cannot be applied to the image completion task directly because they aim at generating natural content which are not necessarily consistent with the image contexts.
sections.heading: 3 APPROACH
sections.heading: 3.1 PROBLEM FORMULATION
sections.text: Denoted by Λ an image lattice (e.g., 1024 × 1024 pixels). Let IΛ be an RGB image defined on the lattice Λ. Denote by Λt and Λc the target region to complete and the remaining context region respectively which form a partition of the lattice.Without loss of generality, we assume Λt is a single connected component region.IΛt is masked out with the same gray pixel value. Let MΛ be a binary mask image with all pixels in MΛt being 1 and all pixels in MΛc being 0. For notational simplicity, we will omit the subscripts Λ, Λt and Λc when the text context is clear.
  The objective of image completion is to generate a synthesized image Isyn that looks natural, realistic and appealing for an observed image Iobs with the target region IobsΛt masked out in the ground-truth full image Igt. Furthermore, it is desirable to control the completion according to a set of attributes which are assumed to be independent from each other (e.g., to respect the underlying intrinsic ambiguities due to the loss of information in the target region). Let A = (a1, · · · , aN ) be a N -dim vector with ai ∈ {0, 1} encoding if a corresponding attribute appears (ai = 1) or not (ai = 0) (e.g. the “Male” attribute in Figure 5). We define the generator as,
  Isyn = G(Iobs,M,A; θG), subject to I syn Λc ≈ IobsΛc (1)
  where θG collects all parameters of the generator, and ≈ represents the two context regions, IsynΛc and IobsΛc , need to kept very similar (both to be elaborated later).
sections.heading: 3.2 THE PROPOSED METHOD
sections.text: The proposed method is built on progressive GANs with well-executed combination between the network architecture, appropriate receipt of loss functions and a novel FAM.
  Denote by Gr and Dr the generator and discriminator at resolution r respectively, where r ∈ {1, · · · , R} is the index of resolution (e.g., r = 1 represents 4 × 4 and r = R = 9 represents 1024× 1024). Accordingly, we have the observed corrupted image, its corresponding binary mask and its ground-truth uncorrupted image (in training only), Iobsr , Mr and I gt r at each resolution.
  The generator, Gr takes as input the observed data XGr and the attribute vector A, then outputs a completed image Isynr . It consists of two components,
  Isynr = Gr(X G r , A; θGr ) = G compl r (G enc r (X G r ; θGencr ), A; θGcomplr ), (2)
  where Gencr (·) encodes the input XGr to a latent low dimensional vector. The latent vector is concatenated with the attribute vector. The concatenated vector plays the role of the noise random variable z in the original GAN. Then, Gcomplr (·) transforms the concatenated vector to a sample Isynr (i.e., the completed image). Gencr and G compl r are mirrored to each other in the form of U-shape networks (Ronneberger et al., 2015; Newell et al., 2016). θGr = (θGencr , θGcomplr ).
  The discriminator, Dr classifies its input XDr and has two output branches, the fake versus real classification branch and the attribute prediction branch. It consists of three components: a shared feature backbone and two head classifiers. We have,
  Dr(X D r ; θDr ) = {Dclsr (Fr(XDr ; θFr ); θDclsr ), D attr r (Fr(XDr ; θFr ); θDattrr )} (3)
  Where the feature backbone, Fr(XDr ; θFr ) computes the feature map. On top of the feature map, the first head classifier, Dcls(·) computes binary classification between real and fake, and the second one, Dattr(·) predicts an attribute vector. All the parameters of the discriminator are collected by θDr = (θFr , θDclsr , θDattrr ). We note that the discriminator is only needed in training. We will omit the notations for parameters in equations when no confusion is caused.
  Progressive Growing of Generators and Discriminators. Following the methodology proposed in (Karras et al., 2017), we start with training G1 and D1. Then, at the growing stage r (r > 1), Gr = (Gr−1, G fade−in r ) is first created on top of the previous trained stage Gr−1 with newly added layers Gfade−inr , and both of them are trainable in a fade-in process for smooth transition. The discriminators grow in the same way. The alternation of Gr−1 in training stage Gr may lead to the instability of the growing process and the loss of learned structures of the underlying image distribution, which motivates the proposed FAM in growing generators (Section 3.2.1).
  Inputs for Generators and Discriminators, XGr and X D r . We have,
  XGr = (Î obs r ,Mr, Lr) and X D r = (Ir, Lr) (4)
  where Lr represents the facial landmarks. Recent works (Isola et al., 2016; Wang et al., 2017; Zhu et al., 2017; Sangkloy et al., 2017; Xian et al., 2017; Chen & Hays, 2018) have shown the capability of GANs to translate sketches or edges to photo-realistic images. Given a corrupted image, it is better if the model is able to “draw” a sketch of face first, which provides a backbone guidance for image
  completion. We utilize the following methods to compute landmarks (Figure 2).
  • In training, we extract landmarks from the uncorrupted image at 256 × 256 resolution using an off-the-shelf pre-trained Face Alignment Network (FAN) (Bulat & Tzimiropoulos, 2017) which achieved very good results for faces in the wild.
  • In testing, to predict landmarks from corrupted images, we first train a single stage face completion network at 256× 256 resolution, denoted by GL, using reconstruction loss (Section 3.2.2) only. For a testing image, we use GL to generate a blurry completed image from which the landmarks are extracted with FAN.
  L is up-sampled or down-sampled to Lr to match the size of networks at different resolutions. Ir in XDr represents either the uncorrupted image or the image synthesized by Gr. For Î obs r in X G r , we have Îobs1 = I obs 1 , and Î obs r is the output of FAM (r > 1) as described in following section.
sections.heading: 3.2.1 FAM: THE PROPOSED FREQUENCY-ORIENTED ATTENTIVE MODULE
sections.text: Figure 3 illustrates the proposed FAM. To obtain a smooth transition from low to high resolutions, we design a FAM architecture (i.e. the red components in Figure 3), which is a frequency-oriented attention mechanism integrated along with the resolution change, so that the model learns filters to encourage Gfade−inr (blue components in Figure 3) to read and write information that are important at level r, but have not been handled well at level r − 1. By doing those, • Our model attends on higher frequency signals as resolution increases (see Figure 1), thus improv-
  ing the completion performance. • Our model preserves what have been learned in previous progressive stages, thus furthering the
  stability of progressive GANs.
  Existing approaches (Gregor et al., 2015; Yu et al., 2018) use spatial attention mechanisms to encourage networks to attend to selected parts of images (e.g. rectangular regions), while FAM is an attention model in the frequency domain. But different from a regular band-pass filter, the filters generated by FAM is predicted based on semantics of images which are enforced by the objective function (Eqn. 10), and thus is also sensitive to locations inferred on-the-fly in a coarse-to-fine manner. For instance, the model pays more attention to eye regions where the rich details aggregate, especially at a high resolution.
  Recall that at the growing stage r (r > 1) in training, we haveGr = (Gr−1, Gfade−inr ). In the vanilla progressive GANs, we will compute the completed image Isynr = Gr(I obs r ,Mr, Lr, A) (simplified from Eqn. 2 for clarity). The previously trained Gr−1 can be changed while optimizing the stage r, which may lead to unexpected updating. Our FAM prevents Gr−1 from changing to wrong directions. To that end, we first introduce a read module which utilizes a read filter F̂read to extract the most valuable information in both Iobsr and I obs r−1,
  Îobsr , Î obs r−1 = read(I obs r , I obs r−1, F̂read), (5)
  which is implemented by,
  Îobsr = F̂read (1−Mr) Iobsr , (6) Îobsr−1 = Downsample((1− F̂read) (1−Mr) Ĩobsr−1), (7)
  where denotes element-wise multiplication. Ĩobsr−1 is up-sampled from Iobsr−1 to match the resolution of Iobsr . Ĩ obs r−1 represents the blurred (i.e. low-frequency) version of I obs r since high-frequency information is lost when Iobsr−1 is down-sampled from I obs r . The read filter F̂read is defined by,
  F̂read = β · Fread + γ, β : {
  2α, 2− 2α, γ : { 0, α ≤ 0.5 2α− 1, 0.5 < α ≤ 1.0 (8)
  where Fread is computed by Fread = ToFilter(G fixed r−1 (I obs r−1,Mr−1, Lr−1)) using a trained generator Gfixedr−1 with fixed weights and a small trainable network ToFilter. α is a weight increasing linearly from 0 to 1 proportional to the number of seen images during growing. F̂read starts as an all-zero filter, is adjusted by a trainable ToFilter at the growing stages and eventually increased to all ones.
  As illustrated in Figure 3, given the outputs from the read module, Îobsr and Î obs r−1, we can generate the corresponding completed images, Isynr and the up-sampled I syn r−1 respectively, and the write filter Fwrite. Then, to generate the final completed image at stage r, we also introduce a write module,
  Îsynr =write(I syn r , I syn r−1, F̂write) (9)
  =(Isynr · α+ I syn r−1 · (1− α)) (1−Mr) + (F̂write Isynr + (1− F̂write) I syn r−1) Mr,
  where F̂write = β ·Fwrite +γ and Fwrite is predicted from the last feature maps. At a low resolution, both the read and write module exploit more on the low-frequency information, but gradually move to exploit higher-frequency information when resolution increases under the guidance of minimizing the objective function (Eqn. 10). Fread and Fwrite can be discarded when the growing process is done. A testing image only needs to go through one generator that is independent of FAM (blue flow in Figure 3). This is more efficient than the Laplacian GAN (Denton et al., 2015) that requires feeding a testing sample to a cascade of generators, and uses multiple discriminators in training.
sections.heading: 3.2.2 LOSS FUNCTIONS
sections.text: Beside extending the original adversarial loss function, we design three new loss functions to enforce sharp image completion.
  Adversarial Loss Given an uncorrupted image Igt, its attribute vector A, a mask M , landmarks L, and the corresponding corrupted image Iobs we define the loss by, ladv(Igt,M,L, Iobs, A|G,D) = log (1−Dcls(Isyn, L) + logDcls(Igt, L), where Isyn = G(Iobs,M,L,A)).
  Attribute Loss Similar to the InfoGAN models (Chen et al., 2016; Choi et al., 2017), for the attribute prediction head classifier in the discriminator, we define the attribute loss based on cross-entropy between the predicted attribute vector, Âreal = Dattr(Ireal, L) and Âobs = Dattr(Iobs, L) and the corresponding targets, A for both a real uncorrupted image and a synthesized image. We have, lattr(I
  gt, A,M, Iobs|G,D) = CrossEntropy(A, Âgt) + CrossEntropy(A, Âobs). Reconstruction Loss Since our method generates the entire completed face rather than only the target region, we define a wighted reconstruction loss lrec to preserve both the target region and the context region, which is defined as, lrec(Igt,M,L, Iobs, A|G) = ‖κ M Idiff‖1 + ‖(1 − κ) (1−M) Idiff‖1, where Idiff = Igt − Isyn and κ is the trade-off parameter. Feature Loss In additional to the reconstruction loss in terms of pixel values, we also encourage the synthesized image to have a similar feature representation (Johnson et al., 2016) based on a pre-trained deep neural network φ. Let φj be the activations of the jth layer of φ, the feature loss is defined by, lfeat(Igt,M,L, Iobs, A|φ,G) = ‖φj(Igt) − φj(Isyn))‖22. In our experiments, φj is the relu2_2 layer of a 16-layer VGG network (Simonyan & Zisserman, 2014) pre-trained on the ImageNet dataset (Russakovsky et al., 2015).
  Boundary Loss To make the generator learn to blend the synthesized target region with the original context region seamlessly, we further define a close-up reconstruction loss along the boundary of the mask. Similar to (Yeh et al., 2017), we first create a weighted kernel w based on the mask image M . w is computed by blurring the mask boundary in M with a mean filter so that the pixels closer to the mask boundary are assigned larger weights. The kernel size of the mean filters is seven in our experiments. We have, lbdy(Igt,M,L, Iobs, A|G) = ‖w (Igt − Isyn)‖1. Our model is trained end-to-end by integrating the expected loss of the different loss functions defined above under the minimax game setting. We have,
  min G max D Ladv(G,D) + λ1Lattr(G,D) + λ2Lrec(G) + λ3Lfeat(G,φ) + λ4Lbdy(G). (10)
  Where λi’s are trade-off parameters between different loss terms.
  Training without Multiple Controllable Attributes. To that end, since it is a special case of the proposed formulation stated above, we can simply remove the components involving attributes such as the attribute loss in a straightforward way. The resulting system still enjoys end-to-end learning.
sections.heading: 4 EXPERIMENTS
sections.text: Datasets and Experiment Settings We used the CelebA-HQ (Karras et al., 2017) dataset for evaluation. It contains 30,000 aligned face images at 1024× 1024 resolution. The dataset is split randomly while ensuring there is no identity overlap between test/training sets: 3,009 images for testing, and 26,991 for training. There were two types of masks: center and random. The center mask was a square region in the middle of the image with a side length of half the size of the image. The random masks, generated in a similar way to previous methods (Iizuka et al., 2017; Yu et al., 2018), were rectangular regions with random width-to-height ratios, sizes and locations covering about 5% to 25% of the original images. Hyper-parameters used for training are listed in the supplemental materials.
  Quality Comparison with the Context Encoder Our method was compared with the Context Encoder (CE) (Pathak et al., 2016) on high-resolution face completion. Since the original networks
  of CE were designed for 128× 128 images, we used a naive approach to fit it to different resolutions. One, two, and three convolutional layers were added to the encoder, decoder and discriminator for 256 × 256, 512 × 512 and 1024 × 1024 networks respectively. The result (Figure 4) shows that, when the resolution increased, our method learned details incrementally and synthesized sharper faces, while CE generated poorer images with more distortions.
  Quantitative Comparison with State-of-the-art Methods As noted in literatures (Yeh et al., 2017; Yu et al., 2018), reconstruction metrics such as mean L1, L2 errors and peak signal-to-noise ratio (RSNR) that are commonly used are not good quantitative evaluation metrics for inpainting methods since image completion aims at completing missing regions with plausible content rather than reconstructing it. As a reference, we show the comparison between our method and state-of-theart models at their reported resolutions respectively: 128 × 128 for GL with center masks (using implementation of (Yu et al., 2018)) and 256× 256 for CTX with random masks (Table 1). Semantic Completion We first trained a high-resolution (1024 × 1024) model with center masks (examples shown in Figure 5) to test whether our model is capable of learning high-level semantics and structures of faces and synthesize large missing regions. The second model was trained with random masks, but was able to handle arbitrary (e.g. irregular hand-drawn) shapes of masks. The result (Figure 5) shows that our model was able to capture the anatomical structures of faces and generate content that is consistent with the holistic semantics.
  Attribute Controller Unlike previous image completion techniques (Iizuka et al., 2017; Yeh et al., 2017; Li et al., 2017; Yang et al., 2016; Pathak et al., 2016; Yu et al., 2018) that generate only random plausible content, our network completes faces with structurally meaningful content whose appearances and expressions are controllable. Existing approaches (Mirza & Osindero, 2014; Choi et al., 2017; Kaneko et al., 2017) can only control facial expressions roughly (e.g. smiling or not smiling). In contrast, our model is able to control subtle expressions. In the experiment, the face appearance was conditioned on a “Male” attribute and we used landmarks from source actors to control the synthesized expressions (Figure 5). This 512× 512 model was trained from scratch. The example demonstrates the potential application of our method in face reenactment (Li et al., 2012; Garrido et al., 2014; Thies et al., 2016).
  Computation Time Our model, once trained, is able to complete a face image with a single forward pass, resulting in much higher efficiency. We tested our model with a Titan Xp GPU by processing 3000 1024 × 1024 images with 512 × 512 holes. The mean completion time is 0.54 second per image. Unlike our model, existing CNN-based high-resolution in-painting approaches often need much longer time to process an image. For instance, it took about 1 min for the model of Yang et al. (Yang et al., 2016) to complete a 512× 512 image with a Titan X GPU. User Study We compared our method with CTX (Yu et al., 2018), which is the state-of-the-art CNN-based face completion approach capable of completing face images at 256× 256 resolution, with a pilot user study at 256 × 256 resolution with random masks. 27 subjects (15 male and 12 female participants, with ages from 22 to 32) were volunteered to participate.
  There were four sessions of pairwise A/B tests. Each time, a user was shown two images and asked choose the more realistic one. In the first session, two images completed from the same image by different methods were chosen. In session two to four, a real image and a corresponding synthesized image were shown. In the first session, time was unlimited. In session two to four, images were on display for 250ms, 1000ms, 4000ms respectively. The result (Figure 6) shows that there were significantly more images generated by our method being favored by the viewers. Overall, our approach generated sharper images with more details and fewer distortions.
  Limitations Though our method has low inference time, the training time is long due to the progressive growing of networks. In our experiment, it took about three weeks to train a 1024× 1024 model on a Titan Xp GPU. Additionally, by carefully zooming in our results, we find that our high-resolution model fails to learn low-level skin textures, such as furrows and sweat holes. Moreover, the model
  could generate distorted content while removing large parts (e.g. hats) or synthesize some plausible but unnatural faces (Figure 7). Furthermore, For facial expression transfer, our method requires that the head poses of the source and target faces are similar. These issues are left for future work.
sections.heading: 5 CONCLUSION
sections.text: We propose a progressive GAN with frequency-oriented attentive modules (FAM) for high-resolution and fast face completion, which learns face structures from coarse to fine guided by the FAM. By consolidating information across all scales, our model not only outperforms state-of-the-art methods by generating sharper images in low resolution, but is also able to synthesize faces in higher resolutions than existing techniques. A conditional version of our model allows users to control the properties of generated images explicitly with attribute vectors and landmarks. Our system is designed in an end-to-end manner, in that it learns to generate completed faces directly and more efficiently.
sections.heading: A APPENDIX
sections.text: A.1 ABLATION STUDY
  The encoder-decoder structure has been widely used in image completion networks (Pathak et al., 2016; Iizuka et al., 2017; Li et al., 2017). However, the encoding process is a lossy compression, which makes it difficult to reconstruct the original contextual regions. Additionally, since much contextual information is lost during the encoding process, it is also difficult for the encoder to reconstruct content that perfectly match the context. UNet adds skip connections between the mirrored layers of the encoder and decoder, consolidating information from all previous layers, rather than depending on only the latent code. Therefore, UNet can be used to generate a completed image conditioned on the corrupted input. Unfortunately, if UNet is applied to image completion networks directly, it will be much easier for the generator to reconstruct the context than the content, resulting in inconsistent colors and textures along boundaries, which often causes the vanishing gradient problem
  where both the generator and discriminator stop learning when the synthesized content is still blurry (Figure 8).
  We try to solve this problem from two aspects. First, we adopt the method of training GANs progressively. Our model starts at a low resolution, so that it is difficult for the discriminator to capture the inconsistency between content and context regions. When the network grows, since the lower-resolution is already trained, the inconsistency between context and content regions is reduced. Second, we design the boundary loss and weighted reconstruction loss, which encourage the network focus more on synthesizing the boundary and content regions respectively. The feature loss also helps stabilizing the training by encouraging the synthesized images to have similar high-level features to real samples. These two improvements have improved the performance significantly.
  The image quality is further enhanced by incorporating FAM. First, the reader and writer act as band-pass filters that minimize the influence of high-frequency noise on low-level network parameters, and thus avoiding distorting the already-learned global structures. Second, FAM is predicted from the facial semantics so that it encourages the model to focus on learning features in regions with richer details. For instance, the eyes synthesized by models trained with FAM are much sharper, more vivid and realistic.
  This ablation study was run at 256× 256 to provide an intuitive illustration of the impact of different components of our method. Since the training of high-resolution models was very time consuming, a more thorough ablation study is left for future work.
  A.2 HIGH-RESOLUTION COMPLETION RESULTS
  More high resolution face completion results with various mask types are demonstrated in Figure 9 and Figure 10. Additionally, Figure 11 and Figure 12 represent two ways to control expressions
  and appearances. In the first example (Figure 11), we use a two-dimensional attribute vector [“Smiling”, “Male”] without landmarks as inputs. In the second one, an attribute [“Male”] is used with landmarks extracted from source actors (Figure 12). The results show that both methods can control the expressions and appearances explicitly. But more subtle expressions can be controlled with landmarks. Moreover, we present more examples of attention filters during growing process in Figure 13.
  A.3 TRAINING DETAILS
  The progressive training process is illustrated in Figure 14. At a resolution lower than 1024×1024, the input face images, masks, landmarks and real images are all down-sampled with average pooling to fit the given scale. One of the major challenges of generating high resolution images is the limitation of Graphics Processing Unit (GPU) memory. Most completion networks use Batch Normalization (Ioffe & Szegedy, 2015) to avoid covariate shift. However, with the limited GPU memory, only a small number of batch sizes are supported at high resolution, resulting in low quality of generated images. We use the Instance Normalization (Ulyanov et al., 2016), similar to Zhu et al. (Zhu et al., 2017), and update D with a history of completed images instead of the latest generated one (Shrivastava et al., 2016) to stabilize training.
  At the growing stage, new layers are added for both D and G and these layers are faded in with current networks smoothly. After the fade-in process, the network is trained on more images for stabilization. We used 300K, and 150K training images for resolution [8 × 8 to 256 × 256] and [512×512, 1024×1024] respectively at growing stage, and 600K, 430K images for 4×4 and [8×8 to 1024× 1024] at stabilizing stage respectively.
  In the experiments, the reconstruction trade-off parameter was set to κ = 0.7 to focus more on the target region. To balance the effects of different objective functions, we used λattr = 2, λrec = 500, λfeat = 8, and λbdy = 5000. The Adam solver (Kingma & Ba, 2014) was employed with a learning rate of 0.0001.
  A.4 NETWORK STRUCTURE
  The generator G in our model is implemented by a U-shape network architecture consisting of the first component Genc transforming the observed image and its mask to a latent vector and the second componentGcompl transforming the concatenated vector (latent and attribute) to the completed image. There are residual connections between layers in Genc and the counterpart in Gcompl similar in the spirit to the U-Net (Ronneberger et al., 2015) and the Hourglass network (Newell et al., 2016) to consolidate information across multiple scales. Figure 15 illustrates the two structures of a layer in the generator for training without and with attributes respectively, which are adapted from the U-Net and Hourglass network.
  Every convolutional layer (Conv) is followed by an Instance Normalization (InsNorm) and a LeakyReLU layer, except that the Conv before the latent vector (i.e. the second Conv layer in Table 2) is not followed by an InsNorm. Additionally, the there are no InsNorms or LeakyReLUs after the last Convs of both Dcls and Dattr. All Convs used in the residual block of the skip connections of our conditional model have a kernel size of three and a stride of one.
  Since we use Instance Normalization rather than Batch Normalization, the batch size is not an important hyper-parameter. Technically, for faster computation, we use as large a batch size as possible so long as it does not exceed the GPU memory limit.
  Tables 2 and 4 demonstrate the architecture of the components of the generator G while Tables 5 shows the components of the discriminator D. In Table 5, depending on the operation of the skip connection (Skip), the number of filters is either doubled (for a concatenation operation) or remains the same (for an addition operation).
  r x r cubes in the figure represent convolutional layers that handle resolution r. For the conditional
  version, attribute labels Aobs are concatenated to the latent vectors. The discriminator D splits into two branches in the final layers: Dcls that classifies if an input image is real, and Dattr that predicts attribute vectors. Note that XG and XD are both a set of inputs as defined in the paper. We use images in this Figure as a simplified illustration.
title: PROGRESSIVELY ATTENTIVE GANS