summaries: This paper proposes TokenGAN for unconditional image synthesis. The token-based generator takes the learned content tokens and style tokens from the latent space as input, and output a series of style-modulated content tokens, which are then concatenated and reshaped to get the output image. Cross-attention is used to assign the styles to different content tokens. The model adopts the style-based generator idea from StyleGAN and introduces the recent token-based structures and transformers to the traditional style-based generator. Experiments are conducted on FFHQ and LSUN CHURCH datasets and the quantitative results are slightly better than StyleGAN2. In this paper, a transformer-based modification of the generator in the ubiquitous StyleGAN model is proposed (named _TokenGAN_). More specifically, the entire generator is replaced by a convolution-free architecture where style-content modulation is now modeled using cross-attention layers instead of AdaIN. Similar to the original StyleGAN, this should allow for decoupled control of style and content at different granularities. The paper shows that the proposed approach achieves FID scores comparable to or better than StyleGAN2 for unconditional image synthesis on FFHQ and LSUN-Churches. This paper proposes to adopt the StyleGAN architecture (i.e. individual styles that affect image generation) to a transformer based model for image generation. The model generates individual image patches which are modulated by style tokens through an attention mechanism. The model is evaluated on the FFHQ and LSUN Church datasets and performs similarly well or better than StyleGAN2 based on FID, precision, and recall. This paper investigates the application of transformer for image generation by replacing the convolutional layer with MLP(including attention). The motivation provided is the limitation of the style-based generator (e.g.,  StyleGAN), which leverages the single style over multiple image regions, allowing a less entangled representation learning in W and visible artifacts in generated images.  To combat this shortcoming, authors introduce transformer-liked architecture, and successfully generate high-quality images (1024*1024) without touching any convolutional layers in generator.
abstractText: We present a new perspective of achieving image synthesis by viewing this task as a visual token generation problem. Different from existing paradigms that directly synthesize a full image from a single input (e.g., a latent code), the new formulation enables a flexible local manipulation for different image regions, which makes it possible to learn content-aware and fine-grained style control for image synthesis. Specifically, it takes as input a sequence of latent tokens to predict the visual tokens for synthesizing an image. Under this perspective, we propose a token-based generator (i.e.,TokenGAN). Particularly, the TokenGAN inputs two semantically different visual tokens, i.e., the learned constant content tokens and the style tokens from the latent space. Given a sequence of style tokens, the TokenGAN is able to control the image synthesis by assigning the styles to the content tokens by attention mechanism with a Transformer. We conduct extensive experiments and show that the proposed TokenGAN has achieved state-of-the-art results on several widelyused image synthesis benchmarks, including FFHQ and LSUN CHURCH with different resolutions. In particular, the generator is able to synthesize high-fidelity images with 1024× 1024 size, dispensing with convolutions entirely.
authors.name: Yanhong Zeng
authors.name: Huan Yang
authors.name: Hongyang Chao
authors.name: Jianbo Wang
authors.name: Jianlong Fu
references.authors: Rameen Abdal Yipeng Qin Peter Wonka
references.title: Image2stylegan: How to embed images into the stylegan latent space
references.venue: In ICCV,
references.authors: Rameen Abdal Yipeng Qin Peter Wonka
references.title: Image2stylegan++: How to edit the embedded images
references.venue: In CVPR,
references.authors: Diederik P. Kingma andJimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: In ICLR,
references.authors: Andrew Brock Jeff Donahue Karen Simonyan
references.title: Large scale gan training for high fidelity natural image synthesis
references.venue: In ICLR,
references.authors: Tom B Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell
references.title: Language models are few-shot learners
references.authors: Nicolas Carion Francisco Massa Gabriel Synnaeve Nicolas Usunier Alexander Kirillov Sergey Zagoruyko
references.title: End-to-end object detection with transformers
references.venue: In ECCV,
references.authors: Hanting Chen Yunhe Wang Tianyu Guo Chang Xu Yiping Deng Zhenhua Liu Siwei Ma Chunjing Xu Chao Xu Wen Gao
references.title: Pre-trained image processing transformer
references.venue: In CVPR,
references.authors: Mark Chen Alec Radford Rewon Child Jeffrey Wu Heewoo Jun David Luan Ilya Sutskever
references.title: Generative pretraining from pixels
references.venue: In ICML,
references.authors: Edo Collins Raja Bala Bob Price Sabine Susstrunk
references.title: Editing in style: Uncovering the local semantics of gans
references.venue: In CVPR,
references.authors: Emily Denton Soumith Chintala Arthur Szlam Robert Fergus
references.title: Deep generative image models using a laplacian pyramid of adversarial networks
references.venue: In NeurIPS,
references.authors: Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova
references.title: Bert: Pre-training of deep bidirectional transformers for language understanding
references.venue: In NAACL-HLT,
references.authors: Chris Donahue Zachary C Lipton Akshay Balsubramani Julian McAuley
references.title: Semantically decomposing the latent spaces of generative adversarial networks
references.venue: In ICLR,
references.authors: Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai Thomas Unterthiner Mostafa Dehghani Matthias Minderer Georg Heigold Sylvain Gelly
references.title: An image is worth 16x16 words: Transformers for image recognition at scale
references.venue: In ICLR,
references.authors: Patrick Esser Robin Rombach Bjorn Ommer
references.title: Taming transformers for high-resolution image synthesis
references.venue: In CVPR,
references.authors: Rinon Gal Dana Cohen Amit Bermano Daniel Cohen-Or
references.title: Swagan: A style-based wavelet-driven generative model
references.venue: arXiv preprint arXiv:2102.06108,
references.authors: Ian J Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron C Courville Yoshua Bengio
references.title: Generative adversarial nets
references.venue: NeurIPS,
references.authors: Jinjin Gu Yujun Shen Bolei Zhou
references.title: Image processing using multi-code gan prior
references.venue: In CVPR,
references.authors: Martin Heusel Hubert Ramsauer Thomas Unterthiner Bernhard Nessler Sepp Hochreiter
references.title: Gans trained by a two time-scale update rule converge to a local nash equilibrium
references.venue: In NeurIPS,
references.authors: Zhiyuan Hu Jia Jia Bei Liu Yaohua Bu Jianlong Fu
references.title: Aesthetic-aware image style transfer
references.venue: In ACM MM,
references.authors: Xun Huang Serge Belongie
references.title: Arbitrary style transfer in real-time with adaptive instance normalization
references.venue: In ICCV,
references.authors: Drew A Hudson C. Lawrence Zitnick
references.title: Generative adversarial transformers
references.authors: Ali Jahanian Lucy Chai Phillip Isola
references.title: On the "steerability" of generative adversarial networks
references.venue: In ICLR,
references.authors: Yifan Jiang Shiyu Chang Zhangyang Wang
references.title: Transgan: Two pure transformers can make one strong gan, and that can scale up
references.venue: NeurIPS,
references.authors: Tero Karras Timo Aila Samuli Laine Jaakko Lehtinen
references.title: Progressive growing of gans for improved quality, stability, and variation
references.venue: In ICLR,
references.authors: Tero Karras Miika Aittala Janne Hellsten Samuli Laine Jaakko Lehtinen Timo Aila
references.title: Training generative adversarial networks with limited data
references.venue: NeurIPS,
references.authors: Tero Karras Samuli Laine Timo Aila
references.title: A style-based generator architecture for generative adversarial networks
references.venue: In CVPR,
references.authors: Tero Karras Samuli Laine Miika Aittala Janne Hellsten Jaakko Lehtinen Timo Aila
references.title: Analyzing and improving the image quality of stylegan
references.venue: In CVPR,
references.authors: Tuomas Kynkäänniemi Tero Karras Samuli Laine Jaakko Lehtinen Timo Aila
references.title: Improved precision and recall metric for assessing generative models
references.venue: NeurIPS,
references.authors: Ze Liu Yutong Lin Yue Cao Han Hu Yixuan Wei Zheng Zhang Stephen Lin Baining Guo
references.title: Swin transformer: Hierarchical vision transformer using shifted windows
references.venue: In ICCV,
references.authors: Sachit Menon Alexandru Damian Shijia Hu Nikhil Ravi Cynthia Rudin
references.title: Pulse: Selfsupervised photo upsampling via latent space exploration of generative models
references.venue: In CVPR,
references.authors: Lars Mescheder Andreas Geiger Sebastian Nowozin
references.title: Which training methods for gans do actually converge
references.venue: In ICML,
references.authors: Takeru Miyato Toshiki Kataoka Masanori Koyama Yuichi Yoshida
references.title: Spectral normalization for generative adversarial networks
references.venue: In ICLR,
references.authors: Aäron van den Oord Nal Kalchbrenner Oriol Vinyals Lasse Espeholt Alex Graves Koray Kavukcuoglu
references.title: Conditional image generation with pixelcnn decoders
references.venue: In NeurIPS,
references.authors: Taesung Park Ming-Yu Liu Ting-Chun Wang Jun-Yan Zhu
references.title: Semantic image synthesis with spatially-adaptive normalization
references.venue: In CVPR,
references.authors: Stanislav Pidhorskyi Donald A Adjeroh Gianfranco Doretto
references.title: Adversarial latent autoencoders
references.venue: In CVPR,
references.authors: Elad Richardson Yuval Alaluf Or Patashnik Yotam Nitzan Yaniv Azar Stav Shapiro Daniel Cohen-Or
references.title: Encoding in style: a stylegan encoder for image-to-image translation
references.venue: In CVPR,
references.authors: Mehdi SM Sajjadi Olivier Bachem Mario Lucic Olivier Bousquet Sylvain Gelly
references.title: Assessing generative models via precision and recall
references.venue: In NeurIPS,
references.authors: Yujun Shen Jinjin Gu Xiaoou Tang Bolei Zhou
references.title: Interpreting the latent space of gans for semantic face editing
references.venue: In CVPR,
references.authors: Dmitry Ulyanov Andrea Vedaldi Victor Lempitsky
references.title: Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis
references.venue: In CVPR,
references.authors: Aaron Van Oord Nal Kalchbrenner Koray Kavukcuoglu
references.title: Pixel recurrent neural networks
references.venue: In ICML,
references.authors: Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Łukasz Kaiser Illia Polosukhin
references.title: Attention is all you need
references.venue: In NeurIPS,
references.authors: Yi Wang Ying-Cong Chen Xiangyu Zhang Jian Sun Jiaya Jia
references.title: Attentive normalization for conditional image generation
references.venue: In CVPR,
references.authors: Hongwei Xue Bei Liu Huan Yang Jianlong Fu Houqiang Li Jiebo Luo
references.title: Learning fine-grained motion embedding for landscape animation
references.venue: In ACM MM,
references.authors: Fuzhi Yang Huan Yang Jianlong Fu Hongtao Lu Baining Guo
references.title: Learning texture transformer network for image super-resolution
references.venue: In CVPR,
references.authors: Fisher Yu Ari Seff Yinda Zhang Shuran Song Thomas Funkhouser Jianxiong Xiao
references.title: Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop
references.venue: arXiv preprint arXiv:1506.03365,
references.authors: Yanhong Zeng Jianlong Fu Hongyang Chao
references.title: Learning joint spatial-temporal transformations for video inpainting
references.venue: In ECCV,
references.authors: Yanhong Zeng Jianlong Fu Hongyang Chao Baining Guo
references.title: Learning pyramid-context encoder network for high-quality image inpainting
references.venue: In CVPR,
references.authors: Han Zhang Ian Goodfellow Dimitris Metaxas Augustus Odena
references.title: Self-attention generative adversarial networks
references.venue: In ICML,
references.authors: Richard Zhang Phillip Isola Alexei A Efros Eli Shechtman Oliver Wang
references.title: The unreasonable effectiveness of deep features as a perceptual metric
references.venue: In CVPR,
references.authors: Heliang Zheng Jianlong Fu Yanhong Zeng Jiebo Luo Zheng-Jun Zha
references.title: Learning semanticaware normalization for generative adversarial networks
references.venue: NeurIPS,
references.authors: Jiapeng Zhu Yujun Shen Deli Zhao Bolei Zhou
references.title: In-domain gan inversion for real image editing
references.venue: In ECCV,
references.authors: Peihao Zhu Rameen Abdal Yipeng Qin Peter Wonka. Sean
references.title: Image synthesis with semantic region-adaptive normalization
references.venue: In CVPR,
references.authors: Xizhou Zhu Weijie Su Lewei Lu Bin Li Xiaogang Wang Jifeng Dai
references.title: Deformable detr: Deformable transformers for end-to-end object detection
sections.heading: 1 Introduction
sections.text: Unconditional image synthesis generates images from latent codes by adversarial training [10, 16, 19, 25, 27, 33, 50]. Recent advances have been achieved by a style-based generator architecture in terms of both the visual quality and resolution of generated images [26, 27, 28, 37, 52]. In particular, the style-based generator has been widely used in many other generative tasks, including facial editing [9, 40], style transfer [1, 38], image super-resolution [17, 31], and image inpainting [2, 52].
  The key to the success of the style-based generator lies in the learning of the style control based on the intermediate latent spaceW [27, 28]. Instead of feeding the input latent code z ∈ Z through the input layer only (Figure1-a), the style-based generator maps the input z to an intermediate latent space w ∈ W , which then controls the “style” of the image at each layer via adaptive instance normalization (AdaIN [21]) (Figure 1-b). It has been demonstrated that such a design allows a less entangled representation learning inW , leading to better generative image modeling [12, 23, 27, 40]. Despite the promising results, the style-based generator can suffer from the style control via AdaIN operation [28, 52]. Specifically, the style control is content-independent. It “washes away” the original information of features by normalization and assigns new styles decided by the latent codes regardless of the image/feature content. Besides, the style code w affects the entire image by scaling and biasing complete feature maps with a single value via AdaIN operation [35, 44, 54]. Such an imposed single style over multiple image regions can inevitably result in entangled representation of
  ∗This work was done while Yanhong Zeng was a research intern at Microsoft Research Asia.
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  different image variations (e.g., hairstyle, facial expression) [28, 44, 52]. These limitations for image modeling can even lead to visible artifacts in the synthesized results (e.g., droplet artifacts [27]).
  To get rid of the issues caused by StyleGAN’s style modeling, we introduce a new perspective that views image synthesis as a visual token generation problem. The visual token is a popular representation of an image patch with a predefined size and position [6, 7, 13]; and has shown an impressive superiority in various tasks with the development of Transformer models [6, 13, 43, 46]. Inspired by the appealing property of the token-based representation, we propose to achieve image synthesis by visual token generation. Specifically, it takes as input a sequence of latent tokens to predict the visual tokens of an image. Such a token-based representation enables a flexible local manipulation for different image regions, which makes it possible to learn content-aware and fine-grained style control for image synthesis.
  Under this new paradigm, we design a token-based generator, i.e., TokenGAN, for the visual token generation problem. Specifically, we consider two different types of input tokens in the generator, i.e., the content tokens and the style tokens. The content tokens are learned as the constant input in the generator network and the style tokens are projected from a learned intermediate latent space (Figure1-c). Given a sequence of style tokens, the TokenGAN learns to control the visual token generation by rendering each content token with related style tokens according to their semantics. In particular, since the Transformer has been verified to be effective in sequence modeling in a broad range of tasks [6, 43], we adopt a generator network architecture from a visual Transformer to model the relations between the content tokens and the style tokens. Through such a content-dependent style modeling by the attention mechanism in Transformer, the TokenGAN is able to achieve content-aware and fine-grained style learning for image synthesis.
  We conduct both quantitative comparisons and qualitative analysis on several unconditional image generation benchmarks. The results show that the token-based generator has achieved comparable results to the state-of-the-art in image synthesis. We summarize our contributions as below:
  • We propose a new perspective of achieving image synthesis by visual token generation. Such a token-based representation enables flexible local manipulation for different image regions, leading to a better image modeling. • We propose a token-based generator (i.e., TokenGAN) for the visual token generation. Specifically, the TokenGAN introduces the style tokens and the content tokens. It adopts a Transformer-based network for content-aware style modeling. • We show extensive experiments (quantitative and qualitative comparisons, study on style editing, image inversion and image interpolation) to verify the effectiveness of the tokenbased generator. Specifically, the token-based generator is able to synthesize high-fidelity 1024× 1024 images without any convolutions in the generator.
sections.heading: 2 Related works
sections.heading: 2.1 Style-based generator
sections.text: The distinguishing feature of the style-based generator is its unconventional generator architecture [26, 27, 28, 52]. Typically, the style-based generator consists of a mapping network f and a synthesis
  network g. The mapping network is used to transform the input latent code z to an intermediate latent code w ∈ W for learning a less entangled latent space. The synthesis network follows a progressive growing design that is able to first output low-resolution images that are not affected significantly by high-resolution layers [25, 27]. Such an architecture design can lead to an automatic, unsupervised separation of high-level attributes in different layers. For example, when trained on human faces, the low-resolution layers control coarse styles (e.g., pose, identity) of images and high-resolution layers control fine styles (e.g., micro-structure, color scheme) [25].
  To control the attributes of the image, the style-based generator produces styles from the intermediate latent code w by affine transforms, which then control each layer of the synthesis network via adaptive instance normalization (AdaIN) [21, 35]. The AdaIN operation removes styles from previous layers by normalizing each feature map to zero mean and unit deviation, and it assigns the new styles by scaling and biasing the complete normalized feature maps. However, it has been demonstrated such an AdaIN operation can destroy the magnitude information of features and thus results in the well-known droplet artifacts [28]. Besides, the scale-specific style disentanglement in StyleGAN can be limited due to the complex semantic components in each feature map [28, 52]. In this paper, we explore a new architecture by flattening the image as a sequence of tokens, which enables fine-grained control of the image by assigning token-wise semantic-aware styles based on the attention mechanism.
sections.heading: 2.2 Transformer in vision
sections.text: The Transformer typically takes as input a sequence of vectors, called tokens [43]. The Transformer is built for sequence modeling solely on attention mechanisms over the tokens, dispensing with recurrence and convolutions entirely [5, 11, 43, 30]. Due to its great success in the field of natural language processing, an increasing number of works attempt to extend Transformer for computer vision tasks [6, 7, 8, 13, 18, 36, 55]. For example, iGPT shows promising results in image classification by pre-training a sequence Transformer with the task of auto-regressive next pixel prediction and masked pixel prediction [8]. Dosoyvitskiy et al. propose to split an image into patches and feed these patches into a standard Transformer (ViT) for image classification [13]. They show that a ViT with large-scale training can trump CNNs equipped with inductive bias. DETR is a seminal work that views object detection as a direct set prediction problem, eliminating the need for hand-crafted components and achieving impressive performance by Transformer [6]. To mitigate the issue of high computation complexity associated with long sequences caused by high-resolution images, a branch of works explores lightweight Transformer (e.g.., Deformable DETR) for vision tasks [55].
  Transformer is also attracting increasing attention in low-level vision tasks [14, 24, 34, 36, 42]. Parmar et al. cast image generation as an autoregressive sequence generation problem and propose Image Transformer with a local self-attention mechanism [36]. However, Image Transformer can suffer from its quadratic computation cost and its long inference time due to auto-regressive prediction. Most existing works adopt a hybrid CNN-Transformer architecture, which consists of a CNN head for feature extraction, a Transformer encoder-decoder backbone, and a CNN tail for feature decoding [7, 46, 48]. For example, IPT fully utilizes the Transformer architecture by a large-scale pre-training and achieves promising results in several image restoration tasks [7]. In parallel work, Hudson et al. introduce bipartite structure to maintain computation of linear efficiency for long-range interactions across the image based on a convolution backbone [22]. Our token-based generator inherits the network architecture from Transformer without any convolution layers and is able to yield surprising promising results for high-resolution image synthesis.
sections.heading: 3 Approach
sections.text: In this section, we introduce the details of the proposed token-based generator (TokenGAN). As depicted in Figure 2, the TokenGAN takes as input two kinds of tokens to generate the visual tokens of images by a Transformer. We introduce the input tokens in Section 3.1 and the token-based generator architecture in Section 3.2, following the overall optimization in Section 3.3.
sections.heading: 3.1 Visual tokens
sections.text: It’s intuitive that the visual tokens of an image contain the information of both content (i.e., semantics) and related styles. In our token-based generator, we choose to separate these two kinds of information
  by a sequence of content tokens and a sequence of style tokens, so that the interactions between them can be modeled and controlled for the image synthesis. We introduce the details of each as below.
  Content tokens. As depicted in Figure 2-a, the TokenGAN starts from the sequence of learned constant content tokens c ∈ Rm×(p2×d) through the input layer. Specifically, d is the dimension in terms of channels, p × p is the patch size of the content token, and m is the length of the content token sequence. To maintain the position information of each token, we add position encodings p ∈ Rm×(p2×d) to each content token following a commonly used paradigm [7, 13]. For simplicity, we rewrite the notation for the content tokens as {c1, · · · , cm}, where ci ∈ Rd.
  Style tokens. Given a latent code z from the input latent space Z , the mapping network adopts several MLPs to map the input z to a set of different style tokens {s1, · · · , sn} ∈ S, where si ∈ Rd. As shown in Figure 2-b, the style tokens are paired with a set of learnable semantic embedding as a key-value structure in each style modulation layer of the TokenGAN. With the Transformer modeling, all the content tokens will match with the semantic embedding and then fetch the new style from the style tokens based on the matching results. The fetched new styles are used to control the values of the content tokens, which are finally decoded to images. Such a token-wise style control enables content-aware and fine-grained style learning for image synthesis.
sections.heading: 3.2 Token-based generator
sections.text: As shown in Figure 2, the TokenGAN consists of multiple layers of style blocks and each style block consists of a style normalization layer and a style modulation layer. Specifically, the style normalization layer removes the styles from previous layers by relieving the dependence on the original statistics of input token features. At the same time, the style modulation layer assigns new styles from the current layer to the content tokens. In particular, the new styles are calculated based on the pairwise interactions between the style tokens and the content tokens by a cross-attention mechanism. We introduce more details about the style normalization, content-aware style modeling, and style modulation in our generator as below.
  Style normalization. We apply style normalization on the input content tokens to remove the styles from previous layers, which is denoted as:
  Norm(ci) = (ci − µ(ci)) / σ(ci), i = 1, · · · ,m, (1)
  where each ci ∈ Rd denotes a content token, d denotes the dimension of each content token, and m is the length of the token sequence. Specifically, the style-based generator typically removes the styles from previous layers by Instance Normalization [41], which may destroy the information conveyed by the magnitude of the feature map relative to each other [27, 28, 41]. To avoid the above issue, we adopt LayerNorm [43] to relive the style dependence while maintaining the information of the relative magnitude following previous works [28, 29, 43].
  Content-aware style modeling. After style normalization, the TokenGAN calculates new styles to control the input content tokens. As shown in the style modulation in Figure 2-b, the style tokens are paired with a set of learnable semantic embedding as a key-value structure. Given the content tokens {c1, · · · , cm} from the input and the style tokens from the mapping network {s1, · · · , sn} ∈ S, the token-wise new styles are calculated by attention mechanism:
  S′ = Attention(C,K,S) = softmax( CKT√
  d )S, (2)
  where S′ ∈ Rm×d denotes new styles for all the content tokens, the input content tokens are packed together into a matrix C ∈ Rm×d, similarly with the style tokens as matrix S ∈ Rn×d, and the matrix K ∈ Rn×d for the learnable semantic embedding in where each row vector indicates a learned semantic. We omit the linear projection in the standard attention calculation for simplicity [43].
  Through such a cross-attention mechanism, the new token-wise styles are fetched for style modulation based on the matching results. Particularly, tokens with similar semantics will have similar styles (e.g., the same color for the two eyes). We visualize the attended regions for different key-value pairs (i.e., semantic-style embedding) in Figure 3, and we can find that the learned embedding K is able to attend to meaningful image regions.
  Style modulation. Different from the traditional Transformer that adds the attention results back to the input features as residual features, our attention results are used as new styles by amplifying each channel of the content tokens: C′ = C S′, (3) where each row vector in C ∈ Rm×d indicates a content token, and indicates element-wise multiplication. Such a style modulation affects the operation of subsequent embedding layers (which we implemented by fully-connected layers) and thus control the style of generated images.
  Implementation details. We translate the generated visual tokens to the image by concatenation and reshaping. In practice, we follow StyleGAN2 and adopt a skip-generator architecture in TokenGAN. Such a skip generator generates images in each layer at different resolutions (e.g., from 42 to 82 ) and progressively upsamples and sums the images from the previous layer by skip connections to the next layer [28, 25]. After progressive summing, the generator takes the output in the last layer as the final results.
sections.heading: 3.3 Overall optimization
sections.text: Style mixing. To further encourage the styles to localize, we employ the mixing regularization technique during training [27]. To be specific, a given percentage of images are generated using two random latent codes. We run two latent codes z1, z2 through the mapping network, and have the corresponding {s11, · · · , s1n}, {s21, · · · , s2n}. We randomly choose an inject point to mix the styles, so that a part of the final style tokens from z1 and another part from z2. This regularization technique prevents the network from assuming that a set of style tokens are correlated. A similar strategy is adopted in terms of different layers following the style-based generator [27].
  Optimization objectives. We denote the generated image output by the token-based generator as: G(z) = g(f(z),C), (4)
  where z ∈ Z ∼ N (0, 1) indicates the input latent code, C = {c1, · · · , cm} denotes the sequence of learned constant content tokens through the input layer, and f , g, G indicate the mapping network, the generator network and the token-based generator, respectively. Our token-based generator follows the same optimization objectives used by the style-based generator [27, 28], i.e., an non-saturating logistic adversarial loss with R1 regularization. To be specific, the generator loss is:
  LG = Ez∼N (0,1)[log(1 + exp(−D(G(z))))], (5) where D is the discriminator and the discriminator loss is
  LD = Ex∼Pd [log(1 + exp(−D(x)))] + Ez∼N (0,1)[log(1 + exp(D(G(z))))] +R1(ψ), (6) where x ∼ Pd denotes images from the real data, and R1(ψ) is the regularization term calculated by:
  R1(ψ) = γ
  2 Ex∼Pd
  [ ‖∇Dψ(x)‖2 ] . (7)
  Specifically, the R1 regularization term [32] is computed less frequently than the main loss function following the commonly used lazy regularization strategy, thus greatly diminishing the computational cost and the overall memory usage [27, 28, 52].
sections.heading: 4 Experiments
sections.heading: 4.1 Experiment Setup
sections.text: Dataset To evaluate the token-based generator and make fair comparisons with the SOTA approach (i.e., the style-based generator [28]), we conduct experiments on the most commonly-used public research datasets, i.e., Flickr-Faces-HQ (FFHQ) [27] and Large-scale Scene Understanding (LSUN) [47]. Specifically, FFHQ consists of 70,000 high-quality images of human face, which is able to evaluate the model’s ability to synthesize high-frequency details. We conduct experiments on FFHQ with both 256× 256 and 1024× 1024 image size following previous works [15, 22]. Besides, we choose the church set of LSUN [47] to evaluate the synthesis in terms of complex scenes.
  Evaluation. In this section, we conduct both quantitative and qualitative experiments on the tokenbased generator. Specifically, we report quantitative results by three commonly-used metrics, i.e., Fréchet Inception Distance (FID) [19], Precision and Recall [29, 39]. We use FID as it has been widely used in many generation tasks as an effective perceptual metric [4, 26, 27]. In addition to the image quality assessment, we use the metrics of Precision and Recall to evaluate the distribution learned by GAN. Specifically, the Precision metric intuitively measures the quality of samples from the generated images, while recall measures the proportion of real images that is covered by the learned distribution by GAN [39].
  Training details We use 8 NVIDIA V100 GPUs for training each model. We build upon the public Pytorch implementation of StyleGAN22 [28]. For fair comparisons, we inherit all of the training details and parameter settings from the configuration E setup. All models are trained and report the best results under the same training iterations. For each iteration, we set the size of mini-batch as 32. We use the Adam solver [3] with the same momentum parameters β1 = 0, β2 = 0.99 to train both the generator and discriminator. We apply R1 regularization for every 16 iterations.
  2https://github.com/rosinality/stylegan2-pytorch
sections.heading: 4.2 Unconditional image synthesis
sections.text: Quantitative comparisons. We report quantitative comparisons with StyleGAN2 [28] due to its state-of-the-art performance. For fair comparisons, we report the results with the same training iterations on FFHQ-256 [27], FFHQ-1024 [27] and LSUN CHURCH [47] by three objective metrics, i.e., FID, Precision and Recall in Table 1. The results show that the token-based generator has achieved the state-of-the-art results in terms of both perceived quality and distribution modeling.
  Qualitative results. To demonstrate the image quality of the synthesis results for the token-based generator, we randomly sample noise z ∈ Z ∼ N (0, 1) to generate images for high-resolution images (i.e., with resolution 1024× 1024) of human face and the images of complex church scenes. The uncurated results can be found in Figure 4. It shows that the token-based generator is able to synthesize both high-frequency details of the human face and plausible structures for complex scenes.
  Style editing. We study and report the results of style editing by the TokenGAN in Figure 5. All the results are obtained by editing the style token of interest and then re-synthesizing images using edited style tokens. The result shows several interesting properties of the TokenGAN. First, the TokenGAN has a similar behavior as in StyleGAN2, i.e., controlling coarse/middle/fine styles at different layers (e.g., pose/hairstyle/color scheme in 82/162 − 642/1282 − 2562 layers). Besides, the TokenGAN shows localized behaviors for the style tokens. For example, in the first row, editing the style token for the pose "globally" changes the pose while remaining the styles of other regions (e.g., hairstyle, background). In the second row, editing the style token for hair length, TokenGAN "locally" turns the hair longer while remaining other styles.
  Image inversion. To better apply well-trained GANs to real-world applications, the technique of GAN inversion has attracted an increasing attention [40, 53, 1, 2]. Such an inversion technique enables real image editing by searching the most accurate latent code in the learned latent space to recover the real image [53]. It has been demonstrated that the model with better inversion results tend to learn a better image modeling of the real data [28]. Specifically, we randomly sample several real images from FFHQ-256 [28] and adopt the optimization-based inversion technique used by most works [27, 40, 53]. For fair comparisons, we use the same setting following StyleGAN2 [27]. The inversion results by the style-based generator and the token-based generator are shown in Figure 6. It shows that the token-based generator is able to reconstruct real images better. For example, StyleGAN2 tends to reconstruct the lips and the headscarf in the fifth case with the same red color,
  while the token-based generator is able to reconstruct accurate colors for the lips and the headscarf respectively. This is because TokenGAN renders different regions by using different style tokens via attention mechanism, while StyleGAN2 renders them by a single style vector.
  For a better quantitative comparison for image inversion, we randomly sample 1,000 real images from FFHQ-256 and report the mean absolute error (MAE, range=[0,255]) and the LPIPS distance [51] of the inversion results by StyleGAN2 and TokenGAN. The results in Table 2 show that TokenGAN is able to reconstruct significantly more accurate results with much lower MAE and shorter LPIPS distance, which is in line with the visual results in Figure 6.
  Image interpolation. To further explore the property of the learned latent space by the token-based generator, we perform image interpolation by a linear interpolation in the learned intermediate latent space S. In practice, we randomly sample two noises z1, z2 and run them through the mapping network to have their corresponding sequence of style tokens s1 := {s11, · · · , s1n}, s2 := {s21, · · · , s2n}. After that we perform linear interpolation for each token by s3i = α× s1i + (1− α)× s2i , α ∈ (0, 1) and use the new style tokens s3 := {s31, · · · , s3n} to synthesize a new image. As shown in Figure 7, we show the sampled styles s1, s2 in the first column and the last column, and the interpolated styles between them. The results show that the token-based generator is able to perform perceptually smooth transition between two different styles (e.g., glasses, age, hair length, coloring, etc.)
sections.heading: 4.3 Ablation study
sections.text: To study the effectiveness of each component of the token-based generator, this section presents ablation studies on FFHQ-256 [27] after 0.4 million training iterations in terms of FID [19]. Specifically, we calculate the metric 10 times with different random seed and report the average results.
  The number of style tokens. To study the effectiveness of the style tokens, we conduct ablation studies by using different number of style tokens. As shown in Table 3, with the growing number of the style tokens, there are more style tokens to model the distribution of styles for different semantics and the quality of images are improved. However, when the number of style tokens grows to 64, the learning will be difficult and performance would drop.
  The number of content tokens. We empirically set the number of content tokens m at different layers according to the image size for the best performance and training efficiency. We conduct ablation study from layer 642 to 2562 and denote the number of content tokens in these layers as (m64,m128,m256). The results in Table 4 show that more content tokens could provide more fine-grained control over the whole images, leading to better results in image generation.
  Style normalization layers. We compare different style normalization layers in Table 5. The results show that, PixelNorm [25] and LayerNorm [43] significantly outperforms the style normalizatioin by instance normalization in the token-based generator [41]. We choose LayerNorm as the style normalization layer in the token-based generator.
sections.heading: 5 Conclusion
sections.text: In this paper, we propose a novel TokenGAN showing that the token-based representation of image features and styles could enable content-aware and fine-grained style learning for image synthesis. Specifically, we use Transformer to model the interaction between content tokens and style tokens, which facilitates the perceived quality of generated images and shows promising properties in terms of style editing. In the future, we will study to extend the generative transformers to extensive applications, e.g., style transfer [20], animation generation [45], image inpainting [49], etc.
  Acknowledgments and Disclosure of Funding
  Funding in direct support of this work: NSF of China under Grant 61672548 and U1611461, GPUs provided by Microsoft Research Asia. Additional revenues related to this work: internship at Microsoft Research Asia.
title: Improving Visual Quality of Image Synthesis by A Token-based Generator with Transformers