summaries: The paper bases its methodology on well known developments in image analysis/synthesis about similarity of pixel values in adjacent locations. Many techniques have been used for modelling this similarity, including predictive models, cliques and graphs. The paper uses a simple autoregressive model for generating pixel values based on the values of previously processed pixels, estimating the differences between these neighboring pixel values.  The paper proposes an approach for image generation that relies on an autoregressive model for the image pixels. These models are popularly used in image coding and compression settings, and have been used in generative models like PixelCNN. In contrast to this prior work, the proposed model is based on the selection of a previously available pixel and the modeling of the differences between the old pixel and the new one. The copy and adjustment models, i.e., eqs (3) and (5-6), are straightforward. Applications to image-to-image translation are also presented. In this paper the authors present a new way to use autoregressive modeling to generate images pixel by pixel where each pixel is generated by modeling the difference between the current pixel value and  the preexistent ones. In order to achieve that, the authors propose a copy and adjustment mechanism that select an existing pixel, and then adjust its sub-pixel (channel values) to generate the new pixel. The proposed model is demonstrated with a suite of experiments in classic image generation benchmark. The authors also demonstrate the use of their technique in Image to Image translation.
abstractText: In natural images, transitions between adjacent pixels tend to be smooth and gradual, a fact that has long been exploited in image compression models based on predictive coding. In contrast, existing neural autoregressive image generation models predict the absolute pixel intensities at each position, which is a more challenging problem. In this paper, we propose to predict pixels relatively, by predicting new pixels relative to previously generated pixels (or pixels from the conditioning context, when available). We show that this form of prediction fare favorably to its absolute counterpart when used independently, but their coordination under an unified probabilistic model yields optimal performance, as the model learns to predict sharp transitions using the absolute predictor, while generating smooth transitions using the relative predictor. Experiments on multiple benchmarks for unconditional image generation, image colorization, and super-resolution indicate that our presented mechanism leads to improvements in terms of likelihood compared to the absolute prediction counterparts.
references.authors: Sungjin Ahn Heeyoul Choi Tanel Pärnamaa Yoshua Bengio
references.title: A neural knowledge language model
references.authors: Haider al Mahmood Zainab Al-Rubaye
references.title: Lossless image compression based on predictive coding and bit plane slicing
references.venue: International Journal of Computer Applications, 93:1–6,
references.authors: Andrew Brock Jeff Donahue Karen Simonyan
references.title: Large scale GAN training for high fidelity natural image synthesis
references.venue: CoRR, abs/1809.11096,
references.authors: XI Chen Nikhil Mishra Mostafa Rohaninejad Pieter Abbeel
references.title: PixelSNAIL: An improved autoregressive generative model
references.venue: In Proceedings of the 35th International Conference on Machine Learning,
references.authors: Rewon Child Scott Gray Alec Radford Ilya Sutskever
references.title: Generating long sequences with sparse transformers
references.venue: URL https://openai.com/blog/sparse-transformers,
references.authors: Patryk Chrabaszcz Ilya Loshchilov Frank Hutter
references.title: A downsampled variant of imagenet as an alternative to the CIFAR
references.venue: datasets. CoRR,
references.authors: Ryan Dahl Mohammad Norouzi Jonathon Shlens
references.title: Pixel recursive super resolution
references.venue: CoRR, abs/1702.00783,
references.authors: J. Deng W. Dong R. Socher L.-J. Li K. Li L. Fei-Fei
references.title: ImageNet: A Large-Scale Hierarchical Image Database
references.venue: In CVPR09,
references.authors: Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville Yoshua Bengio
references.title: Generative adversarial nets
references.venue: Advances in Neural Information Processing Systems
references.authors: Karol Gregor Ivo Danihelka Alex Graves Daan Wierstra
references.title: DRAW: A recurrent neural network for image generation
references.venue: CoRR, abs/1502.04623,
references.authors: Jiatao Gu Zhengdong Lu Hang Li Victor O.K. Li
references.title: Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1631–1640
references.venue: Association for Computational Linguistics,
references.authors: Sergio Guadarrama Ryan Dahl David Bieber Mohammad Norouzi Jonathon Shlens Kevin Murphy
references.title: Pixcolor: Pixel recursive colorization
references.venue: CoRR, abs/1705.07208,
references.authors: C.W. Harrison
references.title: Experiments with linear prediction in television
references.venue: Bell System Technical Journal,
references.authors: Barry G. Haskell Arun N. Netravali
references.title: Digital Pictures: Representation, Compression, and Standards
references.venue: Perseus Publishing,
references.authors: Satoshi Iizuka Edgar Simo-Serra Hiroshi Ishikawa
references.title: Let there be color!: Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification
references.venue: ACM Trans. Graph.,
references.authors: Phillip Isola Jun-Yan Zhu Tinghui Zhou Alexei A. Efros
references.title: Image-to-image translation with conditional adversarial networks
references.venue: CoRR, abs/1611.07004,
references.authors: Tero Karras Timo Aila Samuli Laine Jaakko Lehtinen
references.title: Progressive growing of gans for improved quality, stability, and variation
references.venue: CoRR, abs/1710.10196,
references.authors: Diederik P. Kingma Prafulla Dhariwal
references.title: Glow: Generative flow with invertible 1x1 convolutions, 2018
references.authors: Alex Krizhevsky
references.title: Learning multiple layers of features from tiny images
references.venue: Technical report,
references.authors: Christian Ledig Lucas Theis Ferenc Huszar Jose Caballero Andrew P. Aitken Alykhan Tejani Johannes Totz Zehan Wang Wenzhe Shi
references.title: Photo-realistic single image super-resolution using a generative adversarial network
references.venue: CoRR, abs/1609.04802,
references.authors: Wang Ling Phil Blunsom Edward Grefenstette Karl Moritz Hermann Tomáš Kočiský Fumin Wang Andrew Senior
references.title: Latent predictor networks for code generation
references.venue: In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),
references.authors: Wang Ling Dani Yogatama Chris Dyer Phil Blunsom
references.title: Program induction by rationale generation: Learning to solve and explain algebraic word problems
references.venue: CoRR, abs/1705.04146,
references.authors: Kamyar Nazeri Eric Ng
references.title: Image colorization with generative adversarial networks
references.venue: CoRR, abs/1803.05400,
references.authors: Niki Parmar Ashish Vaswani Jakob Uszkoreit Lukasz Kaiser Noam Shazeer Alexander Ku Dustin Tran
references.title: Image transformer
references.venue: In ICML, volume 80 of JMLR Workshop and Conference Proceedings,
references.authors: Alec Radford Luke Metz Soumith Chintala
references.title: Unsupervised representation learning with deep convolutional generative adversarial networks
references.venue: CoRR, abs/1511.06434,
references.authors: Rajesh P.N. Rao Dana H. Ballard
references.title: Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects
references.venue: Nature Neuroscience,
references.authors: Ali Razavi Aaron van den Oord Ben Poole Oriol Vinyals
references.title: Preventing posterior collapse with delta-VAEs
references.venue: In International Conference on Learning Representations,
references.authors: Morgane Rivière Olivier Teytaud Jérémy Rapin Yann LeCun Camille Couprie
references.title: Inspirational adversarial image generation
references.venue: CoRR, abs/1906.11661,
references.authors: Olaf Ronneberger Philipp Fischer Thomas Brox
references.title: U-net: Convolutional networks for biomedical image segmentation
references.venue: CoRR, abs/1505.04597,
references.authors: Tim Salimans Andrej Karpathy Xi Chen Diederik P. Kingma
references.title: Pixelcnn++: A pixelcnn implementation with discretized logistic mixture likelihood and other modifications
references.venue: In ICLR,
references.authors: Yang Song Chenlin Meng Stefano Ermon
references.title: Mintnet: Building invertible neural networks with masked convolutions
references.venue: CoRR, abs/1907.07945,
references.authors: Ilya Sutskever Oriol Vinyals Quoc V. Le
references.title: Sequence to sequence learning with neural networks
references.venue: CoRR, abs/1409.3215,
references.authors: Seishi Takamura
references.title: Efficient Lossless Still Image Coding Based on the Autoregressive Model
references.venue: PhD thesis, University of Tokyo,
references.authors: Aaron van den Oord Nal Kalchbrenner Lasse Espeholt koray kavukcuoglu Oriol Vinyals Alex Graves
references.title: Conditional image generation with pixelcnn decoders
references.venue: Advances in Neural Information Processing Systems
references.authors: Aäron van den Oord Oriol Vinyals Koray Kavukcuoglu
references.title: Neural discrete representation learning
references.authors: Hanchao Wang Jun Huan
references.title: AGAN: towards automated design of generative adversarial networks
references.venue: CoRR, abs/1906.11080,
references.authors: Richard Zhang Phillip Isola Alexei A. Efros
references.title: Colorful image colorization
references.venue: CoRR, abs/1603.08511,
sections.heading: 1 INTRODUCTION
sections.text: It is has long been appreciated in the field of image compression (Harrison, 1952; Haskell & Netravali, 1997) that adjacent pixels in natural images tend to share similar colors and intensities. This is evidenced in Figure 1, where we plot the marginal distribution of absolute sub-pixel (i.e., color channel) intensities and the difference between the sub-pixel intensities to the corresponding sub-pixel immediately to its left. While the absolute pixel values tend to be distributed non-trivially across the whole range of possible sub-pixel values [0, 255] with peaks around the values of 100 and 255, the relative sub-pixel values closely resembles a Laplacian distribution with the mean at 0 as most pixels do not differ greatly from their neighbouring pixels (Takamura, 1996). This suggests that image generation process could mostly be done by predicting the relative distance between existing and new pixels, and absolute pixel prediction is only needed to generate more abrupt transitions, such as boundaries between two different objects. In predictive coding compression models, this fact is leveraged to achieve increased compression rates by learning to predict pixels as a delta between pixels rather than predicting their absolute values (al Mahmood & Al-Rubaye, 2014).
  Autoregressive models are one of the main forces driving research in image generation van den Oord et al. (2016a); Salimans et al. (2017); Parmar et al. (2018); Chen et al. (2018); Child et al. (2019). In contrast to other popular image models based on adversarial methods (Goodfellow et al., 2014; Radford et al., 2015) or that use latent variables (van den Oord et al., 2017), autoregressive models have a tractable likelihood function that decomposes image generation into a sequence of conditionally dependent pixel predictions. This is a desirable property that provides the grounds for applying the same principles of predictive coding in the image generation process.
  In this paper, we propose an image generation model where new pixels are generated by modeling the differences between new pixels and pre-existing ones. In contrast with existing approaches that tackle the challenging problem of learning to generate each pixel in terms of their absolute values (van den Oord et al., 2016b), we model gradually shifting pixel intensities in relative terms. From the modeling perspective, we propose a copy and adjustment mechanism, where new pixels are predicted by selecting an existing pixel, and adjusting its sub-pixel values to generate a new pixel. We show that new generation methodology generalizes better for multiple datasets in unconditional image generation and image translation tasks, namely colorization (Cheng et al., 2016; Zhang et al., 2016;
  Guadarrama et al., 2017; Iizuka et al., 2016; Guadarrama et al., 2017) and super-resolution (Parmar et al., 2018). Additionally, we show that our mechanism can be adapted to copy and adjust pixels from the input image in image-to-image translation tasks. Finally, we show that the different generation mechanisms can be unified into a single loss function with tractable likelihood computation.
  The paper is organized as follows: Firstly, we describe our copy and adjustment model (§2). Then, we describe its application in image-to-image translation tasks (§3). Afterwards, we describe a mixture model that combines different forms of pixel prediction (§4). Then, we provide experimental results that support our claims (§5). Finally, we describe the positioning of our work with respect the previous research (§6) and proceed to the conclusions (§7).
sections.heading: 2 RELATIVE PIXEL PREDICTION
sections.text: PixelCNN (van den Oord et al., 2016a) models the generation of an image x = (x1,x2, . . . ,xn2) with width and height n as a probability density function p(x) that is factorized into an autoregressive prediction of pixels as follows1:
  p(x) = n2∏ i=1 p(xi | xk<i), (1)
  where xi, which consists of C subpixel channels (xi,1, xi,2, . . . , xi,C), is conditioned on k previously generated pixels. Each pixel prediction p(xi | xk<i) is further factorized by the prediction of its C subpixel channels,
  p(xi | xk<i) = C∏
  c=1
  p(xi,c | xi,m<c,xk<i). (2)
  Similarly, each new sub-pixel is conditioned on the previously k generated pixels and them generated sub-pixels. The probability density function of each sub-pixel xi,c is modelled by bucketing the range of continuous values in the range [0, 1] into 256 evenly distributed classes (van den Oord et al., 2016a), which is modeled as a categorical distribution over Va = [0, 255]. More formally, given a hidden state hi,c−1, obtained by composing the history of generated sub-pixels, the following pixel is generated as follows:
  p(xi,c | xi,m<c,xk<i) = softmaxVa(Wshi,c−1). (3)
  Our work changes the probability density function in Equation 2, which computes the subpixel values directly, with one that computes these values relative to previously generated pixels. More formally, rather than predicting xi,c, we compute the distance ∆ xi,c xr,c = xi,c − xr,c, where xr denotes the previously generated pixel at position r. Under this formulation, we rewrite Equation 2 as follows:
  1Sequences of pixels representing either an image of part of an image are written as x, a vector representing a multi-channel pixel is written x, and scalars are written x. The ith scalar element of vector x is xi, and the jth vector element of a sequence x is xj .
  p(xi | xk<i) = p(xr,∆xi,∗xr,c | xk<i) = p(xr | xk<i) C∏
  c=1
  p(∆xi,cxr,c | xr,xi,m<c,xk<i). (4)
  Illustrated in Figure 2, our relative pixel prediction can be interpreted as selecting a pixel to copy under the probability mass function p(xr | xk<i) and then predicting the adjustment needed to add to that pixel to obtain xi modelled by p(∆ xi,c xr,c | xr,xi,m<c,xk<i). We name the former the copy mechanism and the latter the adjustment mechanism.
sections.heading: 2.1 COPY MECHANISM
sections.text: The concept of learning to reference a pre-existing pixel under a given distribution p(xr | xki) is closely related to the copy mechanism frequently used in natural language generation models (Ling et al., 2016; Gu et al., 2016; Merity et al., 2016), where observed words, either from the input or previously generated output, can be copied to generate new outputs using a pointer network (Vinyals et al., 2015). Pointer networks define a probability distribution over a discrete space of units, in our case previously generated pixels, and compute p(xr | xk<i) as follows:
  ur′ = v T tanh(Washi,0 + W a rhr′+1,0 + W a xxr′) (5)
  p(xr | xk<i) = softmaxUi(u), (6)
  where ur′ (for each r′ in Ui) computes the affinity between current state hi,0 before any sub-pixels at position i are generated and state of a previously generated pixel hr′+1,0 after all pixels in candidate source position r′ have been generated with sub-pixel values xr. Parameters Was , W a r , W a x and v are used to model this function.
  The probability of choosing a given pixel xr is computed as a softmax over the affinities of all pixels that can be selected Ui. While applications of pointer networks in natural language processing can be applied to whole sentences, which contain perhaps 40–200 words, the large number of pixels in images (e.g. 4096 in 64×64 images) makes this practice prohibitive. Following (Parmar et al., 2018), we restrict this choice to a predefined set of neighbouring pixels.
  Finally, near the edges of images, some positions in Ui will lie outside of the image. To address this case, we always can copy the pixel with value xnull = [0, 0, 0] and learn a special state hnull. This is needed for the top and left edges of the image, where no pixels have been previously generated.
sections.heading: 2.2 ADJUSTMENT MECHANISM
sections.text: The adjustment mechanism aims to learn a continuous variable ∆ba which measures the absolute distance between two sub-pixels a and b. In order to keep log-likelihood computations consistent with the buckets defined in previous work (van den Oord et al., 2016a), we compute the distances over the discretized pixel values. Thus, we model the distance function as a categorical distribution over integers in the range Vr = [−255, 255]. This quantifies the number of buckets that need to be adjusted with relation to the pixel to be copied. The computation of the distance distribution is performed as follows:
  a = Wpshi,c−1 + W p rhr,C + W p xxr (7)
  p(∆xi,cxr,c | xr,xi,m<c,xk<i) = softmaxVr (a). (8)
  Thus, the softmax that computes the required adjustment depends on the current hidden state hi,c−1, and the state and the pixel to copy from hr,C and its values xr. It is worth mentioning that this softmax can be replaced by any mechanism that defines a categorical distribution, such as that described in (Salimans et al., 2017).
  While the outcome of the relative predictor consists of a value in the interval [−255, 255], for a given copied pixel, the number of valid adjustment values that situate the intensity within the valid range [0, 255] remains as 256. For instance, if the copied pixel’s value is 100, the range of possible adjustment values is [−100, 155]. While the model can learn to avoid such mistakes from the data, any probability mass assigned to invalid adjustment values is penalized in terms of perplexity. Thereby, we ensure that the model avoids such errors by limiting softmax defined in Equation 7 to the range of valid values.
sections.heading: 2.3 MARGINALIZATION OVER RELATIVE PIXELS
sections.text: For likelihood computation, the choice of the relative pixel xr is unobserved and must be inferred during training. As we define a small range of possible pixels for xr, limited to neighbouring pixels, the marginal likelihood over the latent domain Ui is tractable and can be defined as follows:
  p(xi | xk<i) = ∑
  xr∈Ui
  p(xr,∆ xi,1 xr,c , . . . ,∆ xi,C xr,c | xk<i)
  ∑ xr∈Ui p(xr | xk<i) C∏ c=1 p(∆xi,cxr,c | xr,xi,m<c,xk<i)
  For generation, we first sample the pixel to copy xr from the distribution p(xr | xk<i), then we sample an adjustment from the distributions p(∆xi,cxr,c | xr,xi,m<c,xk<i) for each sub-pixel value c, thereby generating xi.
sections.heading: 3 IMAGE-TO-IMAGE TRANSLATION
sections.text: If the image generation process is conditioned on an input image y, the copy mechanism can also be applied to copy pixels from the input image. This is desirable in tasks, such as image colorization and super-resolution, where the input image is structurally similar to the output image and we hypothesize that learning the adjustment in such tasks generalizes better than learning to generate from scratch.
  PixelCNN can be adjusted for image translation by conditioning on a fully visible input image y. More formally, each pixel p(xi | xk<i,y) is now generated as follows:
  p(xi | xk<i,y) = softmaxVa(Wshi,c−1 + Wmgi)
  where gi is the hidden state of the input image y at pixel i. This differs from the output image hidden states hi,m in that gi has visibility of the whole input image.
  Relative pixel prediction over the input image can be embedded by reformulating Equations 5 and 7 as follows:
  ur′ = v T tanh(Washi,0 + W a mgr′ + W a yyr′) (9)
  p(xr | xk<i,y) = softmaxBi(u) (10)
  a = Wpshi,c−1 + W p rgr + W p yyr (11)
  p(∆xi,cyr,c | xr,xi,m<c,xk<i,y) = softmaxVr (a), (12)
  where we replace the output pixels and their hidden states of the output pixels with those of the input, namely y and g. Secondly, we define Bi as the set of pixels in the input image y that are available to be copied from position i. For simplicity, we also assume that y and x have the same dimensions. For image colorization, where the input image has only one channel, we simply copy the same channel three times. For super-resolution, we ensure this by first increasing to the desired dimensions using a heuristic method (e.g. bilinear interpolation). These do not correspond to optimal architecture choices for these tasks, but still allow the fair evaluation between the absolute and relative predictors.
sections.heading: 4 UNIFIED PREDICTOR
sections.text: The copy and adjust mechanism is self-sufficient for generating any image, and benefits from a denser distribution when neighbouring pixels share similar sub-pixel values. However, when large color perturbations are observed and no correlations between neighbouring pixels can be found, using this mechanism is intuitively disadvantageous, since the range of differences between two sub-pixel values is larger than the range of sub-pixel values (511 vs. 256). Furthermore, in image-to-image tasks it would also be desirable to combine both forms of relative prediction (e.g. copying from the input and output). Our solution to supporting both absolute and relative generation mechanisms is using latent predictors (Ling et al., 2016). We define our set of predictors as P = [Pabs, Pi-rel, Po-rel], where Pabs denotes the absolute pixel predictor computed according to Equation 2 and Pi-rel and Po-rel denote the relative pixel predictors defined in Equation 4 and 11. At each generation step i, we define the predictor probability p(π | xk<i) as follows:
  p(π | xk<i) = softmax[Pabs,Prel-i,Prel-o](Wlhi), (13) where Wl defines the parameters used by the gating system that switches between the absolute and relative predictors.
  It is also beneficial to prune marginally less frequent adjustment values in the relative predictor by removing them from the softmax in Equations 7, as these generally correspond to sharp transitions between pixels, which are better handled by the absolute predictor. We do this by pruning the domain of the softmax over the interval [−255, 255] to the set of the k most frequently observed adjustment values in the training data, where k is a hyperparameter tuned on the validation set. This means that for pruned adjustment values, the relative predictor will predict the pixel with probability 0, which is acceptable in the unified predictor, as the absolute predictor would present a valid probabilistic interpretation for such pixels.
sections.heading: 5 EXPERIMENTS
sections.heading: 5.1 SETUP
sections.text: Our experiments were performed on existing benchmarks for image generation, namely, CIFAR10 (Krizhevsky, 2009) and downsampled ImageNet (Deng et al., 2009; Chrabaszcz et al., 2017).
  For unconditional image generation, hidden states h are obtained using gated convolutional layers (van den Oord et al., 2016a). More concretely, after discrediting the input image x, we apply a convolutional layer with kernel size 7 and hidden size H1. Then, we apply an additional D layers with kernel size 5 and hidden size H1, with the exception of the final layer which has the hidden size
  of H2. Furthermore, we add a skip connection (Ronneberger et al., 2015), between each intermediate layer to the final layer. This final layer corresponds to the hidden states h. For CIFAR-10, we set D = 15, H1 = 64 and H2 = 512, and train with a batch size of 32. For ImageNet, we set D = 20, H1 = 256 and H2 = 2048 and train with a batch size of 128.
  For image-to-image translation, we apply the operations to both x and y to obtain h and g, respectively. However, as y is fully observed, no mask is applied in the convolutions to obtain g. For CIFAR-10, we set D = 15, H1 = 128 and H2 = 512, and train with a batch size of 32. For ImageNet, we set D = 20, H1 = 256 and H2 = 2048 and train with a batch size of 64. For colorization, we remove the colorization of the original image. For super-resolution, we follow the approach applied in (Dahl et al., 2017), and reduce resolution of the image to 8×8. Evaluation is performed in terms of negative log-likelihood measured in bits/dim2.
  As for the prediction mechanisms, we compare the baseline that relies solely on absolute predictors against the best setups found on CIFAR-10 from our ablation study (Appendix A), which is carried out on the validation set. For unconditional image generation, we use a relative predictor that can copy from the pixel to the left and top of the generated pixel. We report results, when only relative prediction is used and when both absolute and relative predictors are used. Here, the pruning threshold is set to 20%, which is also chosen based on the results of the ablation test. For image-to-image translation, the same setup is used when only relative predictors that copy from the output is used. When we consider the input image as another source that can be copied from, we define a 3× 3 grid around the position of the output pixel, and any corresponding pixel in the input image that falls within that grid can be selected as a target to be copied from.
sections.heading: 5.2 QUANTITATIVE RESULTS
sections.text: Table 1 reports the results on the test set of CIFAR-10 and validation sets of ImageNet. Our baseline implementation of PixelCNN (row 0) achieves a negative log-likelihood of 3.02 bits/dim. From rows 1 and 2, we observe that both relative predictors yield improvements over the baseline. Overall, combining each relative form of prediction with the absolute predictor (rows 3 and 4) tends to work well for all tasks except for colorization, where combining the output relative predictor and absolute predictors tend to yield lower results. Finally, combining the input and output relative predictors yields the optimal overall results (row 5). We do not illustrate results when all predictors are combined, as the presence of the input predictor and the absolute predictor causes the latent predictor network (Ling et al., 2016) to ignore the absolute predictor, and thereby, results are identical to that in row 5.
  2We acknowledge that extensive work has been conducted in the tasks of super-resolution (Ledig et al., 2016) and colorization (Nazeri & Ng, 2018), which are evaluated on different metrics, we refrain from directly comparing our model to the aforementioned work as our main motivation is to measure the effectiveness of the relative pixel prediction mechanism with respect to copying pixels from the input.
sections.heading: 5.3 LATENT VARIABLE ANALYSIS
sections.text: We observe that the majority of the images are generated using the relative prediction mechanism from the left pixel, with top pixels only used when the left pixel is unavailable (e.g. left edge of the image) or vertically oriented objects (e.g. legs of the cat). However, there are some instances where images are predominantly predicted by copying from the top pixel. In simpler tasks, such as colorization, the model seems to rarely use the absolute predictors applying it only to generate the transitions between objects. In more advanced tasks, such as super-resolution and unconditional image generation, relative predictors are often applied to generate continuous textures (e.g. sky and glass) whereas more complex objects such as houses and trees are generated by combining both absolute and relative predictors.
sections.heading: 5.4 IMPACT ON STATE-OF-THE-ART MODELS
sections.text: While we show that relative pixel prediction can improve the quality of the predictions of the model presented in (van den Oord et al., 2016a), state-of-the-art models can achieve significantly better results by incorporating multiple modeling innovations (Salimans et al., 2017; Parmar et al., 2018; Chen et al., 2018). Thus, to investigate if the work proposed can be used to further augment state-ofthe-art models. We build upon the model described in (Razavi et al., 2019), which defines a 30-layer neural network with transformer layers. We replace the discretized logistic mixture of components with 32 components with a relative pixel prediction that copies from the left and top of the desired pixel with, whose output is pruned with a 20% threshold. Finally, we also use outcome masking. Results are illustrated in Table 2, where we can see that our model fares favourably with state-of-the-
  art models in this task. Improvements are more marginal (from 2.83 to 2.82) but supplements the upward trend from experiments using the vanilla PixelCNN.
sections.heading: 6 RELATED WORK
sections.text: Copy mechanisms have been widely used in sequence-to-sequence models (Sutskever et al., 2014) in order to copy tokens from the input sequence (Ling et al., 2016; Gu et al., 2016), previously generated tokens (Merity et al., 2016), or from structured data, such as databases (Yang et al., 2016; Ahn et al., 2016). The basic copy mechanism recalls pointer networks (Vinyals et al., 2015) but extends it by allowing the copied element to be modified. A similar approach can be found in (Ling et al., 2017), where copied tokens can be modified with multiple operations before generation. Image compression techniques encode deltas between pixels rather than absolute intensities (Harrison, 1952), and even natural visual systems code deltas rather than absolute magnitudes Rao & Ballard (1999).
  A major force driving progress on auto-regressive image generation models is the advancements of self-attention mechanisms (Parmar et al., 2018; Child et al., 2019), which allows better modeling of long range dependencies within images. However, improving generalization by incorporating better prediction mechinism has also been explored in the past. The usage of discretized logistic mixture likelihoods (Salimans et al., 2017), where the independence between consecutive pixel intensities is alleviated here, which leads to be better generalization is an example of such exploits. Our work is another instance of such lines of research, where we incorporate a mechanism that exploits the fact that consecutive pixels tend to share similar values.
  Another major driver of progress in image generation is the trend of adversarial image generation (Goodfellow et al., 2014; Isola et al., 2016; Karras et al., 2017; Brock et al., 2018; Rivière et al., 2019; Wang & Huan, 2019). While our model is not directly applicable to this set of models, as our proposed work operates on explicit probability estimates of pixel distributions rather than implicit ones, the concept of relative pixel prediction is likely to be beneficial to these models.
sections.heading: 7 CONCLUSION
sections.text: We presented an alternative parameterization for autoregressive models for image generation, where pixels are generated relative to pre-existing pixels. This is achieved by learning two probability mass functions over what pixel to copy and how the copied pixel must be adjusted to obtain the new pixel. We show that generating new pixels as deltas from previously generated neighbouring pixels has better generalization properties compared to existing methods that generate pixels in absolute terms. Finally, we propose a mechanism that combines the multiple types of predictions into mixture model, which allows exact inference at training and decoding time. Experimental results show that our proposed models achieve improvements to their absolute counterparts, and can achieve competitive results compared to state-of-the-art models in unconditional image generation.
sections.heading: A ABLATION STUDY
sections.text: To test the effect of different components described in our work, we start by performing an ablation test over different setups on CIFAR-10, and select the setups that are most promising to be applied to the larger datasets. This study is reported on Table 3. Columns U, C and S report the negative loglikelihood scores in bits/dim for unconditional image generation, colorization and super-resolution, respectively. Each numbered row uses a different set of predictors starting from our baseline implementation of PixelCNN (van den Oord et al., 2016a) in row 0.
  Output Relative Prediction Rows 1 and 2 show the results obtained by computing the adjustment score in Equation 7 with respect to the left or top pixels, respectively. As the choice of which pixel to select as the reference for the adjustment is predetermined, the copy mechanism is not required. Surprisingly, this simple change yields improvements in multiple datasets compared to absolute predictors. On average, adjusting from the left pixel yields more improvement compared to adjusting the top pixel, yet differences seem to be minor. Combining both predictors tends to improve the scores of the model, as observed in row 3, with the exception of the colorization task. In row 4, we also observe that adding all pixels within an 2×2 grid apart from the generated pixel (top, left and diagonals) does not yield additional improvements. The same occurs if we extend this grid to two
  positions, as shown in row 5. We believe this is because neighbouring pixels tend to share similar colors, so more additional pixels are unlikely to add any additional information than the left and top pixels.
  Rows 6, 7 and 8 illustrate the results obtained by combining the absolute predictor from row 0 with each of the relative predictors in rows 3, 4, and 5. Such setups cause the latent predictor networks to assign most probability mass to the absolute predictor. This suggests that the model is over-fitting to the absolute predictor, which has a faster convergence rate due to its smaller softmax (256 vs. 511).
  Thresholding Relative Prediction To effectively combine the absolute and relative predictors we pruned the softmax of the adjustment mechanism to the k most observed values in the training set (as discussed in §4). Rows 9, 10 and 11 report the results obtained when we set the pruning threshold to 20%, keeping only the 20% most frequent adjustment values in the training set. In CIFAR-10, we obtain results that are close to the best performing setups. In Figure 4, we study the results obtained using different threshold values in CIFAR-10. The red line represents the relative improvement obtained for unconditional image generation in the validation set using the model setup corresponding to row 9, and we can observe the highest improvement at 20%. We also report the percentage of pixels generated using the relative prediction mechanism when generating 100 samples with the blue line. We observe that while at lower thresholds the relative prediction mechanism is used approximately for the generation of 70% of images, when this value approaches 100%, this relative prediction is rarely used, explaining the lack of improvements without pruning.
  Parameter Efficiency Figure 5 illustrates the log-likelihood scores for unconditioned image generation for different hyper-parameters for the hidden layer sizes, namely H1 and H2. We can observe that both absolute (red line) and relative (blue line) loss functions start under-performing as we increase the size of the model. Yet, the relative improvement of our loss function is maintained. However, the relative predictor requires more parameters to start performing effectively, if we drop the number of parameters to half, the absolute predictor exhibits a superior performance. We believe that this is because the relative predictor requires more parameters to learn the harsher transitions between pixels. Finally, if we combine the absolute and pruned relative predictors, the model achieves lower overall log-likelihood scores across different parametrizations.
  Input Relative Prediction For relative pixel prediction from the input, we tested models that copy from the center pixel, then in a 2×2 grid and 3×3 grid centered at the position of the pixel to be predicted. Results are reported in rows 12, 13 and 14. Here, we observe that a large patch of pixels to copy from benefits the model. Interestingly, we did not encounter any problems combining the absolute and relative predictors when copying from the input in these tasks. We believe this is because the transitions between the input and output pixels in these tasks is more straight-forward, whereas the transition between neighbouring pixels can contain harsh transitions (e.g. edges between objects), which take longer to learn. Results that combine the input relative predictor and absolute predictor are shown in rows 15, 16 and 17. Here, we observe that combining predictors yields in marginally better scores on average.
  Predictor Combination Rows 18 and 19 show the improvements obtained by combining both input and output predictors, where we observe that optimal results exclude the usage of an absolute predictor. This suggests that the information that the two forms of relative prediction are complementary for the tasks that are tested. That is, the output relative prediction learns the softer transitions between continuous surfaces, whereas input relative prediction learns the harsher transitions between objects. Figure 6 illustrates the percentage to usages of the absolute predictor, input relative predictor and output relative predictor for the colorization task. We observe that the absolute predictor is applied considerably when used individually with either of the relative predictors (A+Ro and A+Ri), while rarely used with both predictors (A+Ri+Ro).
  Outcome Masking Rows 20, 21 and 22 show the improvements that can be obtained by masking invalid entries in the relative predictors. While improvements are marginal, we can observe that perplexities are consistently better using masking, as we are removing invalid outcomes that are certainly leading to losses in probability mass.
sections.heading: B SAMPLED IMAGES
sections.text: Examples of sampled images are illustrated in Figure 7. Columns “Absolute” “Left Pixel Only” and “Absolute + Relative” refer to the models trained using setups 0, 1 and 9 in Table 3. Here, we can observe that the proposed models can capture the richness of the image distribution from the data. Interestingly, sharp transitions are also captured when all pixels are predicted relatively from the pixel to the left (Column “Left Pixel”).
title: RELATIVE PIXEL PREDICTION FOR AUTOREGRESSIVE IMAGE GENERATION