summaries: The paper proposes a transformer-based image super-resolution method for screen content images which have many thin edges. The transformer is designed to learn the mapping from coordinates to rgb values. It additionally has a scale token representing the magnification factor. Due to targeting a new application, the paper also constructs suitable training and testing datatests.

summaries: This work addresses the SCISR problem by combining transformers to the network architecture. The backbone of the architecture utilizes a CNN to extract feature maps, to enlarge the receptive field, an implicit transformer is used. After that the implicit position encoding is conducted via MLP. Natural images are suitable for SR but there is a lack of SR techniques that are suitable for screen content, which this paper tries to address.

summaries: This work presents a continuous image super-resolution based on an implicit transformer. Authors extend LIIF [6] with some modifications including transformer-like formulation in (2), scale token in (5), and implicit position encoding (7). The proposed implicit transformer learns the transformation weight from query (high-resolution) coordinate, key (low-resolution) coordinate, and scale token. A pixel intensity is then computed by transforming the pixel features (obtained from CNN backbone) with the transformation weights. The pixel intensity is further refined with the implicit position encoding, which is an adaptive weighting scheme with learned weights. Experimental results on screen content images demonstrate superior performance over existing continuous super-resolution methods [5,6].

summaries: This paper addresses the problem of arbitrary screen content image super-resolution, which is quite different from natural image super-resolution.  The author proposed two datasets specifically designed for this scenario, which can greatly motivate research  effort into this field. Besides, the authors proposed Implicit Transformer Super-Resolution Network (ITSRN) to achieve arbitrary scaling for screen content images. The ITSRN is largely inspired by LIIF, the main novelty of ITSRN includes the proposed  implicit transformer and the implicit position encoding, these schemes are empirically evaluated to show its effectiveness.

summaries: This paper presents a novel method called implicit transformer for the task of super-resolution of screen content images. The proposed method is able to integrate the advantage of transformer in capturing relation between pixels and that of implicit neural representation in producing continuous and sharp results. The effectiveness of the proposed method is validated on both newly collected datasets w/o and w/ compression degradation, and datasets originally created for image quality assessment. Ablation study also demonstrates the role of the proposed modules.

abstractText: Nowadays, there is an explosive growth of screen contents due to the wide application of screen sharing, remote cooperation, and online education. To match the limited terminal bandwidth, high-resolution (HR) screen contents may be downsampled and compressed. At the receiver side, the super-resolution (SR) of low-resolution (LR) screen content images (SCIs) is highly demanded by the HR display or by the users to zoom in for detail observation. However, image SR methods mostly designed for natural images do not generalize well for SCIs due to the very different image characteristics as well as the requirement of SCI browsing at arbitrary scales. To this end, we propose a novel Implicit Transformer Super-Resolution Network (ITSRN) for SCISR. For high-quality continuous SR at arbitrary ratios, pixel values at query coordinates are inferred from image features at key coordinates by the proposed implicit transformer and an implicit position encoding scheme is proposed to aggregate similar neighboring pixel values to the query one. We construct benchmark SCI1K and SCI1K-compression datasets with LR and HR SCI pairs. Extensive experiments show that the proposed ITSRN significantly outperforms several competitive continuous and discrete SR methods for both compressed and uncompressed SCIs.

authors:
Meng Wang
Jizheng Xu
Li Zhang
Junru Li
Shiqi Wang

title: Super resolution for compressed screen content video

venue: Data Compression Conference (DCC),

authors:
B. Lim
S. Son
H. Kim
S. Nah
K.M. Lee

title: Enhanced deep residual networks for single image super-resolution

venue: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW),

authors:
J. Kim
J.K. Lee
K.M. Lee

title: Accurate image super-resolution using very deep convolutional networks

venue: In IEEE Conference on Computer Vision Pattern Recognition,

authors:
Yulun Zhang
Kunpeng Li
Kai Li
Lichen Wang
Bineng Zhong
Yun Fu

title: Image superresolution using very deep residual channel attention networks

venue: In Proceedings of the European Conference on Computer Vision (ECCV),

authors:
Xuecai Hu
Haoyuan Mu
Xiangyu Zhang
Zilei Wang
Tieniu Tan
Jian Sun

title: Meta-sr: A magnification-arbitrary network for super-resolution

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

authors:
Yinbo Chen
Sifei Liu
Xiaolong Wang

title: Learning continuous image representation with local implicit image function

venue: arXiv preprint arXiv:2012.09161,

title: Compositional pattern producing networks: A novel abstraction of development

venue: Genetic Programming and Evolvable Machines,

authors:
Lars Mescheder
Michael Oechsle
Michael Niemeyer
Sebastian Nowozin
Andreas Geiger

title: Occupancy networks: Learning 3d reconstruction in function space

authors:
B. Mildenhall
P.P. Srinivasan
M. Tancik
J.T. Barron
R. Ramamoorthi
N. Ren

title: Nerf: Representing scenes as neural radiance fields for view synthesis

venue: In European Conference on Computer Vision,

authors:
Huan Yang
Yuming Fang
Weisi Lin

title: Perceptual quality assessment of screen content images

venue: IEEE Transactions on Image Processing,

authors:
Ke Gu
Guangtao Zhai
Weisi Lin
Xiaokang Yang
Wenjun Zhang

title: Learning a blind quality evaluation engine of screen content

venue: images. Neurocomputing,

authors:
Zhangkai Ni
Lin Ma
Huanqiang Zeng
Jing Chen
Canhui Cai
Kai-Kuang Ma

title: Esim: Edge similarity for screen content image quality assessment

authors:
Shiqi Wang
Xinfeng Zhang
Xianming Liu
Jian Zhang
Siwei Ma
Wen Gao

title: Utility-driven adaptive preprocessing for screen content video compression

venue: IEEE Transactions on Multimedia,

authors:
Wen-Hsiao Peng
Frederick G. Walls
Robert A. Cohen
Jizheng Xu
Jorn Ostermann
Alexander MacInnis
Tao Lin

title: Overview of screen content video coding: Technologies, standards, and beyond

venue: IEEE Journal on Emerging and Selected Topics in Circuits and Systems,

authors:
Chao Dong
Chen Change Loy
Kaiming He
Xiaoou Tang

title: Learning a deep convolutional network for image super-resolution

authors:
W.S. Lai
J.B. Huang
N. Ahuja
M.H. Yang

title: Deep laplacian pyramid networks for fast and accurate super-resolution

venue: In IEEE Conference on Computer Vision & Pattern Recognition,

authors:
Yulun Zhang
Yapeng Tian
Yu Kong
Bineng Zhong
Yun Fu

title: Residual dense network for image super-resolution

venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition,

authors:
S. Anwar
N. Barnes

title: Densely residual laplacian super-resolution

venue: IEEE Transactions on Software Engineering,

authors:
Y. Mei
Y. Fan
Y. Zhou
L. Huang
T.S. Huang
H. Shi

title: Image super-resolution with cross-scale non-local attention and exhaustive self-exemplars mining

authors:
Zhiqin Chen
Hao Zhang

title: Learning implicit fields for generative shape modeling

authors:
Matan Atzmon
Yaron Lipman

title: Sal: Sign agnostic learning of shapes from raw data

venue: In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),

authors:
Kyle Genova
Forrester Cole
Avneesh Sud
Aaron Sarna
Thomas Funkhouser

title: Local deep implicit functions for 3d shape

authors:
Jeong Joon Park
Peter Florence
Julian Straub
Richard Newcombe
Steven Lovegrove

title: Deepsdf: Learning continuous signed distance functions for shape representation

authors:
Daniel Rebain
Wei Jiang
Soroosh Yazdani
Ke Li
Kwang Moo Yi
Andrea Tagliasacchi

title: Derf: Decomposed radiance fields

venue: arXiv preprint arXiv:2011.12490,

authors:
Songyou Peng
Michael Niemeyer
Lars M. Mescheder
Marc Pollefeys
Andreas Geiger

title: Convolutional occupancy networks

venue: In ECCV

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N. Gomez
Lukasz Kaiser
Illia Polosukhin

title: Attention is all you need

venue: In Proceedings of the 31st International Conference on Neural Information Processing Systems,

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina N. Toutanova. Bert

title: Pre-training of deep bidirectional transformers for language understanding

authors:
Jonas Gehring
Michael Auli
David Grangier
Denis Yarats
Yann N Dauphin

title: Convolutional sequence to sequence learning

venue: In International Conference on Machine Learning,

authors:
Vincent Sitzmann
Julien Martel
Alexander Bergman
David Lindell
Gordon Wetzstein

title: Implicit neural representations with periodic activation functions

venue: Advances in Neural Information Processing Systems,

authors:
Matthew Tancik
Pratul P. Srinivasan
Ben Mildenhall
Sara Fridovich-Keil
Nithin Raghavan
Utkarsh Singhal
Ravi Ramamoorthi
Jonathan T. Barron
Ren Ng

title: Fourier features let networks learn high frequency functions in low dimensional domains

venue: In Advances in Neural Information Processing Systems,

authors:
Alexey Dosovitskiy
Lucas Beyer
Alexander Kolesnikov
Dirk Weissenborn
Xiaohua Zhai
Thomas Unterthiner
Mostafa Dehghani
Matthias Minderer
Georg Heigold
Sylvain Gelly
Jakob Uszkoreit
Neil Houlsby

title: An image is worth 16x16 words: Transformers for image recognition at scale

venue: In International Conference on Learning Representations,

authors:
Xiangxiang Chu
Bo Zhang
Zhi Tian
Xiaolin Wei
Huaxia Xia

title: Do we really need explicit position encodings for vision transformers

venue: arXiv preprint arXiv:2102.10882,

authors:
Diederik Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: International Conference on Learning Representations,

heading: 1 Introduction

text:
Nowadays, screen content images are becoming ubiquitous due to the wide application of screen sharing and wireless display. Meanwhile, due to the limited bandwidth, screen content images received by users may be in low-resolution (LR) and users may need to zoom in the content for detail inspection. Therefore, screen content image super-resolution (SCI SR) is to improve the quality of LR SCIs.
However, different from natural scene images, SCIs are dominated by the contents generated or rendered by computers, such as texts and graphics. Such contents highly demanded are characterized by thin and sharp edges, little color variance, and high contrast. In contrast, the natural scene images are relatively smooth, and contain rich colors and textures. Conventional image SR methods designed for nature images are good at modeling the local smoothness of natural images other than the thin and sharp edges in SCIs. Very recently, Wang et al. [1] proposed a SR method for compressed screen content videos, which addressed the compression artifacts of screen content videos by introducing a distortion differential guided reconstruction module. However, their network is still composed of fully convolution layers without designing specific structures for the thin and sharp edges in screen contents. In addition, they utilize previous frames to help reconstruct the current frame, which makes it unsuitable for frame-wise SCI SR.
∗Corresponding author: Huanjing Yue
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
On the other hand, conventional SR methods are designed for discrete (i.e., several fixed) magnification ratios [2, 3, 4], making them hard to fit screens with various sizes. Recently, a few SR methods for continuous magnification have been proposed [5, 6]. Hu et. al. [5] proposed to perform arbitrary-scale SR with a learnable up-sampling weight matrix based on meta-learning. The work LIIF [6] introduced the concept of implicit function [7, 8, 9] to image SR. The implicit function, which attempts to represent images with continuous coordinates and directly maps the coordinates to values, enables continuous magnification.
In this work, we observe that convolution filters could be harmful to sharp and thin edges in SCIs since the weight sharing strategy makes them tend to produce a smooth reconstruction result. Therefore, we propose to render the pixel values by a point-to-point implicit function, which adapts to image content according to image coordinates and pixel features. Fortunately, this also enables us to perform continuous magnification for SCIs. We would like to point out that even with the point-to-point implicit function, reconstructing dense edges are still quite challenging. As shown in Fig. 1, LIIF [7] cannot reconstruct the dense edges well since it directly concatenates the pixel coordinates and features together to predict the pixel values, which is not optimal since the two variables have different physical meanings. As a departure, we reformulate the interpolation process as a transformer and introduce implicit mapping to model the relationship between pixel coordinates, which are used to aggregate the pixel feature. Our main contributions are summarized as follows.
• First, we propose a novel Implicit Transformer Super-Resolution Network (ITSRN) for SCI SR. The LR and HR image coordinates are termed as the “key” and “query”, respectively. Correspondingly, the LR image pixel features are termed as “value”. In this way, we can infer pixel values by an implicit transformer, where implicit means we model the relationship between LR and HR images in terms of coordinates instead of pixel values.
• Second, instead of directly concatenating the coordinates and pixel features to predict the pixel value, we propose to predict the transform weights with query and key coordinates via nonlinear mapping, which are then used to transform the pixel features to pixel values. In
addition, we propose an implicit position encoding to aggregate similar neighboring pixel values to the central pixel.
• Third, we construct a benchmark dataset with various screen contents for SCI SR. Extensive experiments demonstrate that the proposed method outperforms the competitive continuous and discrete SR methods for both compressed and uncompressed screen content images. Fig. 1 presents an example of our SR results, which demonstrates that the proposed method is good at reconstructing thin and sharp edges for various magnification ratios.

heading: 2 Related Work

heading: 2.1 Screen Content Processing

text: The screen content is generally dominated by texts and graphics rendered by computers, making the pixel distribution of screen contents totally different from that of natural scenes. Therefore, many works specifically designed for screen contents are proposed, such as screen content image quality assessment [10, 11, 12, 13], screen content video (image) compression [14, 15]. However, there is still no work exploring screen content image SR. Very recently, Wang et al. [1] proposed screen content video SR, which reconstructed the current frame by taking advantage of the correlations between neighboring frames, making it cannot deal with image SR. In addition, its main motivation is solving the SR problem when the videos are degraded by compression other than designing specific structures for continuously recovering thin and sharp edges in screen contents. In this work, we address this issue by introducing point-to-point implicit transformation.

heading: 2.2 Continuous Image Super-Resolution

text: Image SR refers to the task of recovering HR images from LR observations. Many deep learning based methods have been proposed for super-resolving the LR image with a fixed scale [16, 3, 2, 17, 18, 4, 19, 20]. Since the screen contents are usually required to be displayed on screens with various sizes. Therefore, continuous SR is essential for screen contents. In recent years, several continuous image SR methods [5, 6] are proposed in order to achieve arbitrary resolution SR. MetaSR [5] introduces a meta-upscale module to generate continuous magnification but it has limited performance in dealing with out-of-training-scale upsampling factors. LIIF [6] reformulates the SR process as an implicit neural representation(INR) problem, which achieves promising results for both in-distribution and out-of-distribution upsampling ratios. Inspired by LIIF, we utilize the point-to-point implicit function for SCI SR.

heading: 2.3 Implicit Neural Representation

text: Implicit Neural Representation (INR) usually refers to continuous and differentiable function (e.g., MLP), which can map coordinates to a certain signal. INR was widely used in 3D shape modeling [21, 22, 23, 24], volume rendering (i.e., neural radiance fields(Nerf)) [9, 25], and 3D reconstruction [8, 26]. Very Recently, LIIF [6] was proposed for continuous image representation, in which networks took image coordinates and the deep features around the coordinate as inputs, and then map them to the RGB value of the corresponding position. Inspired by LIIF, we propose an implicit transformer network to achieve continuous magnification while retaining the sharp edges of SCIs well.

heading: 2.4 Positional Encoding (Mapping)

text:
Positional encoding is critical to exploit the position order of the sequence in transformer networks [27, 28, 29]. We have to utilize positional encoding to indicate the order of the sequence since there are no other modules to model the relative or absolute position information in the sequenceto-sequence model. In the literature, sine and cosine functions are used for positional encoding in transformer [27]. Coincidentally, we find that the Fourier feature (combined with a set of sinusoids) based positional mapping is used in the implicit function to improve the convergence speed and generalization ability [30, 9, 31]. Specifically, a Fourier feature mapping is applied on the input coordinates to map them to a higher dimensional hypersphere before going through the coordinatebased MLP. Although the two schemes are proposed based on different motivations, they show significant effects in representing position information, which further boost the final results. Inspired
by them, we propose implicit positional encoding to model the relationship between neighboring pixel values. Here "implicit" means that we do not explicitly encode (map) the coordinates but encode the pixel values of neighboring coordinates.

heading: 3 Approach

text: Figure 2 illustrates the framework of the proposed ITSRN. In the following, we give details for the proposed implicit transformer network and implicit position encoding.

heading: 3.1 Implicit Transformer Network

text:
Let’s first review the process of image interpolation. Suppose we have an LR image IL that needs to be upsampled. The pixel value of query point q(i, j) in the HR image IH is obtained by fusing pixel values of its closest key points k(i′, j′) in IL with a weighting matrix. Denoted by Q the query points in upsampled image, K the key points in the input LR image, and V the feature values on the corresponding key points. Then, the image interpolator can be reformulated as a transformer [27]. Different from the explicit transformer which takes pixel values as Q and K, the interpolation transformer deals with pixels’ coordinates instead of their values. Inspired by the implicit function in NeRF [9], which uses the pixel coordinates to generate RGB values, we reformulate the interpolation process as Implicit Transformer, and propose a novel Implicit Transformer Network for SCI SR.
The super-resolution process as well as image interpolation is reformulated in terms of implicit function as
Iq = Φ(q, k, v), (1)
where Iq is the target RGB value that need to be predicted, q is the query coordinate(s) in IH , k is the key coordinate(s) in IL, and v is the pixel value(s) or feature(s) corresponding to the key coordinate(s). Note that both q and k are coordinates in the continuous image domain. Φ is a mapping function which maps coordinates and features to RGB values. In image interpolation, k is the neighboring key coordinate of q, v are the pixel value of k, Φ is the weighting matrix. In implicit function based SR method LIIF [6], k is the nearest neighbor coordinate in IL for q in IH , and v is the corresponding pixel feature of k. LIIF directly concatenates v and the relative coordinate of q from k, and then utilize the nonlinear mapping Φ realized by a multi-layer perceptron (MLP) to render the pixel value Iq. It achieves promising results due to the strong fitting ability of the MLP. However, we observe that directly concatenating the pixel feature and the relative coordinates is not optimal since they
have different physical meanings. To solve this problem, following the idea of transformer [27], we reorganize Formula (1) as follows.
Iq = Φ(q, k, v) = φ(q, k)v. (2)
Different from the explicit transformer, here q and k are pixel coordinates other than pixel values. In other words, we are not learning the the RGB → RGB mapping, but the coordinate→ RGB mapping. Due to the continuity of coordinates, φ(·) is a continuous projection function, which computes the transform weights φ(q, k) to aggregate the feature v, i.e., Iq is computed with the multiplication of φ and v . To further illustrate it, we decompose φ as:
φ(q, k) = f(δ(q, k)), (3) where the function δ produces a vector that represents the relationship between q and k. Then the mapping function f projects this vector into another vector, which is then multiplied with feature v as indicated in Eq. 2.
There are many ways to model the relation function δ, such as dot product and hadamard product commonly used in transformers [27]. Different from their approaches, in this paper, we utilize subtraction operation, i.e., δ(q, k) = γ(q)− τ(k), (4) where γ and τ can use trainable functions or identity mapping. In this work, we utilize identity mapping since it generates similar results as that of trainable functions.
Inspired by ViT’s [32] [class] token, which is extra global information for classification, we augment the input coordinates with scale information, and name it as [scale] token. It represents the global magnification factor. With the [scale] token, the implicit transformer can take the shape of the query pixel as additional information for reconstructing the target RGB value. Although it is feasible to predict RGB values without [scale] token, it is not optimal since the predicted RGB value should not be completely independent of its shape 2. The output of δ(q, k) is concatenated with the [scale] token, and is then fed to the mapping function f . In this way, φ in Formula (3) is reformulated as
φ(q, k) = f([δ(q, k), s]), (5)
where s = (sh, sw) is a two-dimensional scale token, representing the upsampling scale along the row and column for the query pixel. [δ(q, k), s] means the concatenation of δ(q, k) and s along the channel dimension. Note that, to avoid large coordinates in HR space, we normalize the coordinates as
p(i, j) = [−1 + 2i+ 1 H ,−1 + 2i+ 1 W ], i ∈ [0, H − 1], j ∈ [0,W − 1], (6)
where (i, j) represents the spatial location within the RH×W space.

heading: 3.2 Implicit Position Encoding

text:
Although we have considered the positional relationship between q and k, we still ignore the relative position of different query points. Therefore, inspired by the position encoding in explicit transformer, we propose to introduce implicit position encoding (IPE) to avoid the discontinuity of neighboring predictions. Here, "Implicit Position Encoding" means that we do not explicitly encode the positional relationship among the pixel coordinates in the Q sequence since they are already absolute position coordinates. On the contrary, we encode the pixel value relationships within a local neighborhood. Chu et al. claimed that convolution, which models the relationship between neighboring pixels, could be considered as an implicit position encoding scheme [33]. Therefore, we termed Eq.7 as implicit position encoding. Specifically, we unfold the query within a local window. The final predict value of the query coordinate is conditioned on its neighbor values. The IPE process is denoted as
Îq = ∑
p∈Ω(q)
w(p, q)Iq, (7)
where Îq is the refined pixel value, Ω(q) is a local window centered at q, w represents the neighbors’ (denoted by p) contribution to the target pixel. In traditional image filtering, w(p, q) is generally realized by a Gaussian filter or bilateral filter. In this work, we utilize an MLP to learn adaptive weighting parameters for each q.
2It is surprised to find that our formulation in terms of transformer for the [scale] token is similar to the cell decoding strategy in LIIF.

heading: 4 Architecture Details of ITSRN

text:
As shown in Figure 2, the proposed ITSRN has three parts, i.e., a CNN backbone to extract compact feature representations for each pixel, an implicit transformer for mapping coordinates to target values, and an implicit position encoding that further enhances the target values. In the following, we give details for the three modules.
Backbone. Given an LR image IL ∈ R3×h×w, we utilize a CNN to extract its feature map V ∈ Rc×h×w. In our experiments, c = 64. Normally, any CNN without downsampling / upsampling can be adopted as the feature extraction backbone. To be consistent with LIIF [6], we utilize RDN [18] (excluding its up-sampling layers) as the backbone. The extracted features V will be used in the following implicit transformer.
Implicit Transformer. First, inspired by LIIF [6] and MetaSR [5], to enlarge each feature’s local receptive field, we apply feature unfolding, namely concatenating the features for the pixels in a local region (3× 3 in this work) to get v′ ∈ Rc×9×1×1. After that, v′ replaces v for the following processes. All the key coordinates corresponding to the feature vectors construct a coordinate matrix K ∈ R2×h×w with the same spatial shape as V . The channel dimension is 2, which includes the row and column coordinates. Similarly, the query coordinates build a matrix Q ∈ R2×H×W , where H×W is larger than h×w. Therefore, we first utilize the nearest neighbor interpolation to upsample the coordinates K to the size of Q. Hereafter, we utilize an MLP as the nonlinear mapping function f(·) (mentioned in Eq. 3) to learn the relationship between the coordinate pairs in upsampled K and Q. As demonstrated in Eq. 5, f(·) maps the 4-dimensional (coordinates and scale token) input to a 9c-dimensional output(9c× 3 for RGB channels). Finally, the 9c-dimensional output is multiplied by the corresponding pixel feature v, generating the coarse result for Iq .
Implicit Position Encoding. After obtaining the coarse value Iq, we further utilize IPE to refine it. In this way, the central pixel can be more continuous with its neighbors. As mentioned in Eq. 7, we set Ω to 3 × 3 and w for each point is learned via an MLP. The MLP includes two layers, i.e., {Linear → Gelu[27]→ Linear}, and the numbers of neurons for the two layers are 256 and 1, respectively.

heading: 5 Experiments

heading: 5.1 Datasets

text:
To the best of our knowledge, there is still no SCI SR dataset for public usage. Therefore, we first build a dataset named SCI1K. It contains 1000 screenshots with various screen contents, including but not limited to web pages, game scenes, cartoons, slides, documents, etc. Among them, 800 images with 1280×720 are used for training and validation. The other 200 images with resolution ranging
from 1280×720 to 2560×1440 are used for testing. To be consistent with previous works [4, 5, 6, 18], we utilize bicubic downsampling to synthesize the LR images. In addition, to simulate the degradation introduced in transmission and storage, we build another dataset, named SCI1K-compression, by utilizing JPEG compression to further degrade the LR images. The quality factor of JPEG is randomly selected from 75, 85, and 95.
To evaluate the generalization of the trained model, besides our test set, we also test on two other screen content datasets constructed for image quality assessment, i.e., SCID (including 40 images with resolution 1280×720 ) [12] and SIQAD (including 20 images with resolution around 600×800)[13].

heading: 5.2 Training Details

text:
In the training phase, to simulate continuous magnification, the downsampling scale is sampled in a uniform distribution U(1, 4). We then randomly crop 48 × 48 patches from the LR images and augment them via flipping and rotation.
Following [18], we utilize the `1 distance between the reconstructed image and the ground truth as the loss function. The Adam [34] optimizer is used with beta1=0.9 and beta2=0.999. All the parameters are initialized with He initialization and the whole network is trained end-to-end. Following [6], the learning rate starts with 1e− 4 for all modules and decays in half every 200 epochs. We parallelly run our ITSRN-RDN on two GeForce GTX 1080Ti GPU with mini-batch size 16 and it cost 2 days to reach convergence (about 500 epochs). During the test, we directly feed the whole LR image into our network (as long as the graphic memory is sufficient) to generate the SR result.

heading: 5.3 Comparison with State-of-the-arts

text:
To demonstrate the effectiveness of the proposed SR strategy, we compare our method with state-ofthe-art continuous SR methods, i.e., MetaSR [5] and LIIF [6]. We also compare with the discrete SR methods, i.e., RDN [18] and RCAN [4], which are state-of-the-art natural image SR methods. Since RDN and RCAN rely on specific up-sampling modules, they have to train different models for different upsampling scales and cannot be tested for the scales not in the training set. For a fair comparison, we retrain all the compared methods using our training set with the recommended parameters and codes released by the authors. Note that, during test, we downsample the ground truth with different ratios to generate the LR inputs for different magnification ratios. For larger magnification ratios, the details are fewer in its corresponding LR input.
Table 1 and Table 2 present the quantitative comparison results on different datasets. Table 1 lists the average SR results for 200 screen content images in our SCI1K and SCI1K-compression test sets. All the methods are retrained on the corresponding training sets of SCI1K and SCI1K-compression. It can be observed that our method consistently outperforms all the compared methods. Compared with RDN, which is our backbone, our method achieves nearly 1 dB gain at ×4 SR on SCI1K test set. Compared with the competitive RCAN, our method still achieves nearly 0.4 dB gain at ×3 upsampling for the uncompressed dataset. Note that at ×4 SR, the result of our method is similar as that of RCAN. The main reason is that our backbone RDN generates much worse results than RCAN at ×4 SR. If we change our backbone to more powerful structures, our results could also be further improved. For the scales that are not used in the training process (denoted by out-of-trainingscales), RDN and RCAN are not applicable. Meanwhile, our method outperforms the continuous SR methods (i.e., MetaSR and LIIF), which demonstrates that the proposed implicit transformer scheme is superior in modeling both coordinates and features. Table 2 presents the SR results on two SCI quality assessment datasets. Since the images in the two datasets are not compressed, we directly utilize the models trained on SCI1K training set to test. It can be observed that our method still outperforms the compared methods.
Besides visual results in Fig. 1, Fig. 3 presents an example of the qualitative comparison results on SCI1K test set3. It can be observed that our method recovers more realistic edges of characters than the compared methods. Fig. 4 presents the visual results for ×10 SR. It can be observed that our method reconstructs the thin edges better than the compared methods at large magnification ratios.
3More visual comparison results are presented in the supplementary file.

heading: 5.4 Ablation Study Results

text:
In this section, we perform ablation study to demonstrate the effectiveness of the proposed modules. Table 3 lists quantitative comparison results on SCI1K test set. It can be observed that the scale token and implicit position encoding totally contributes 0.39 dB to the final SR result at ×4 SR. Even for the out-of-training-scales, the two modules also contribute 0.06 dB gain. Note that, the gain is lower for larger magnification factor is because that the thin edges in screen contents are hardly to be distinguished after ×6 downsampling. Therefore, it is difficult to bring gains with scale token and position encoding.
We also conduct ablation study on the w in IPE by changing it to different realizations. Table 4 lists the SR results on SCI1K-compression test set. From the table, we can observe that our proposed learnable scheme is superior to the fixed scheme when the input LR images suffer from JPEG compression. This is because that, with JPEG compression, the neighboring information can help alleviate the compression artifacts and the fixed distance based weighting strategy is not suitable in this case. Note that, for very large magnification ratios, the learned w is slightly inferior to the fixed w by 0.02 dB since the heavy distortions in the LR input may confuse the learning process. Besides, we conduct ablation study on the training scales. As shown in Table 5, the SR results at ×6 and ×8 upsampling with training scales ×2−×8 are better than that with training scale ×2−×4. The main reason is that the scales of ×6 and ×8 are in the range of the second training scales. In addition, the second strategy is also beneficial for ×10 upsampling. This indicates us that using wide distributed training scales is better than using narrow distributed training scales.

heading: 5.5 Discussion

text:
The most related work to our proposed ITSRN is LIIF [6], which utilizes implicit function for continuous SR. However, it directly concatenates the image coordinate and its CNN feature, and then map them to the RGB value via an MLP. Different from it, we reformulate the SR process as an implicit transformer. There are two MLPs in our scheme. The first MLP is utilized to model the relationship between pixel coordinates. Then the "relationship” is multiplied by the pixel features to generate the pixel value, which can also be termed as an attention strategy. The second MLP is utilized for implicit position encoding, which is useful to keep the continuous of the reconstructed
pixels. The experiment results also demonstrate that the proposed two strategies make our method consistently outperform LIIF on all the four test sets.

heading: 6 Conclusion

text:
In this paper, we propose a novel arbitrary-scale SR method for screen content images. With the proposed implicit transformer and implicit position encoding modules, the proposed method achieves the best results on four datasets at various magnification ratios. Due to the continuous magnification ability, our method enables users to display the received screen contents on screens with various sizes. In addition, we construct the first SCI SR dataset, which will facilitate more research on this topic.
Note that, although our method outperforms state-of-the-arts on screen content images, it may not work the best for natural images at a fixed magnification ratio. The main reason is that our method is designed for SCIs with high contrast and dense edges, which are suitable to be modeled by point-topoint mapping. Besides, we simulate the degradation process with bicubic and JPEG compression, which may be different from the actual degradation in transmission and acquisition. Thus, there will be limitation in practical applications. In the future, we would like to develop blind distortion based SCI SR to make our model adapt to real scenarios better.

heading: Acknowledgments

text:
The authors would like to thank the anonymous reviewers for their valuable comments. This research was supported in part by the National Natural Science Foundation of China under Grant 62072331, Grant 62171317, and Grant 61771339.
Broader Impact
This work is an exploratory work on screen content images super-resolution with arbitrary-scale magnification. The constructed dataset can facilitate research on this topic. In addition, the proposed method can be combined with image (video) compression technology to enable screen contents transmission with limited bandwidth. As for societal influence, this work will improve the quality of pictures displayed on the screen of any resolution. However, We would like to point out that SR is actually predicting (hallucinating) new pixels, which may make the image deviate from the ground truth. Therefore, image SR has a weak link with deep fakes. It is worth noting that the positive social impact of this technology far exceeds the potential problems. We call on people to use this technology and its derivative applications without harming the personal interests of the public.

title: Implicit Transformer Network for Screen Content Image Continuous Super-Resolution


summaries: The paper proposes to learn interventional distributions using a family of generative models known as sum-product networks (SPN). SPNs are models based on gate functions that allow for tractable inference. Specifically, the authors propose interventional SPNs (iSPN) which are conditional SPNs (they model conditional distributions) that, licensed by a causal graph and samples from the interventional distribution, are shown to be expressive and causally adequate.

summaries: The paper proposes a new causal inference framework based on so-called ‘interventional Sum-Product Networks’ to target the problem of a lack of tractability in causal inference. It is based on standard SPNs and employs deep learning techniques to capture complex nonparametric functions describing multivariate conditional probability distributions. It is evaluated on several small synthetic data sets.

summaries: This paper proposes a modification of sum product networks termed iSPN to estimate interventional distributions from data. The proposed model performs a manipulation to the structure of the SPN in order to provide estimates from the mutilated distribution. The authors motivate the use of this model in terms of computational complexity and the ability to represent estimates of arbitrary causal quantities on the graph. Empirical results show the proposed model having impressive running time and comparable error results to modern software packages for causal effect estimation.

summaries: This submission proposes interventional sum-product networks, a tractable method for estimating causal effects from interventional data. The authors prove that iSPNs are universal function approximators. They show that iSPNs produce accurate estimates of interventional queries from interventional data on collection of synthetic data generating processes.

summaries: This paper discusses a variant of sum-product networks (SPNs) called "interventional sum-product networks" which is a variant of SPNs that take as input a causal graph and use this to model interventional distributions. The claim is that this is the first paper that applies tractable probabilistic models to causality. Theoretical aspects are discussed and the method is evaluated and compared to existing methods.

abstractText: While probabilistic models are an important tool for studying causality, doing so suffers from the intractability of inference. As a step towards tractable causal models, we consider the problem of learning interventional distributions using sumproduct networks (SPNs) that are over-parameterized by gate functions, e.g., neural networks. Providing an arbitrarily intervened causal graph as input, effectively subsuming Pearl’s do-operator, the gate function predicts the parameters of the SPN. The resulting interventional SPNs are motivated and illustrated by a structural causal model themed around personal health. Our empirical evaluation against competing methods from both generative and causal modelling demonstrates that interventional SPNs indeed are both expressive and causally adequate.

authors:
Jayadev Acharya
Arnab Bhattacharyya
Constantinos Daskalakis
Saravanan Kandasamy

title: Learning and testing causal models with interventions

venue: NeurIPS,

authors:
Elias Bareinboim
Judea Pearl

title: Causal inference and the data-fusion problem

authors:
Sander Beckers
Joseph Y Halpern

title: Abstracting causal models

venue: In AAAI,

authors:
Ioana Bica
Ahmed Alaa
Mihaela Van Der Schaar

title: Time series deconfounder: Estimating treatment effects over time in the presence of hidden confounders

venue: In ICML,

title: Mixture density networks

authors:
Philippe Brouillard
Sébastien Lachapelle
Alexandre Lacoste
Simon Lacoste-Julien
Alexandre Drouin

title: Differentiable causal discovery from interventional data

authors:
John F Carriger
Mace G Barron

title: A bayesian network approach to refining ecological risk assessments: Mercury and the florida panther (puma concolor coryi)

venue: Ecological Modelling,

authors:
Daniel C Castro
Jeremy Tan
Bernhard Kainz
Ender Konukoglu
Ben Glocker

title: Morpho-mnist: quantitative assessment and diagnostics for representation learning

venue: Journal of Machine Learning Research,

authors:
Huigang Chen
Totte Harinen
Jeong-Yoon Lee
Mike Yung
Zhenyu Zhao

title: Causalml: Python package for causal machine learning

venue: arXiv preprint arXiv:2002.11631,

authors:
Arthur Choi
Adnan Darwiche

title: On the relative expressiveness of bayesian and neural networks

venue: In International Conference on Probabilistic Graphical Models,

title: The computational complexity of probabilistic inference using bayesian belief networks

venue: Artificial intelligence,

title: A differential approach to inference in bayesian networks

venue: Journal of the ACM (JACM),

authors:
Ishita Dasgupta
Jane Wang
Silvia Chiappa
Jovana Mitrovic
Pedro Ortega
David Raposo
Edward Hughes
Peter Battaglia
Matthew Botvinick
Zeb Kurth-Nelson

title: Causal reasoning from meta-reinforcement learning

authors:
Norman E Fenton
Martin Neil
Magda Osman
Scott McLachlan

title: Covid-19 infection and death rates: the need to incorporate causal explanations for the data and avoid bias in testing

venue: Journal of Risk Research,

title: Forecasting the patterns of covid-19 and causal impacts of lockdown in top five affected countries using bayesian structural time series models

venue: Chaos, Solitons & Fractals,

authors:
Mathieu Germain
Karol Gregor
Iain Murray
Hugo Larochelle

title: Made: Masked autoencoder for distribution estimation

title: Investigating causal relations by econometric models and cross-spectral methods

venue: Econometrica: journal of the Econometric Society,

authors:
York Hagmayer
Steven A Sloman
David A Lagnado
Michael R Waldmann

title: Causal reasoning through intervention

venue: Causal learning: Psychology, philosophy, and computation,

authors:
Joseph F Hair Jr.
Marko Sarstedt

title: Data, measurement, and causal inferences in machine learning: opportunities and challenges for marketing

venue: Journal of Marketing Theory and Practice,

authors:
David Heckerman
Dan Geiger
David M Chickering

title: Learning bayesian networks: The combination of knowledge and statistical data

venue: Machine learning,

authors:
William Hoiles
Mihaela Schaar

title: Bounded off-policy evaluation with missing data for course recommendation and curriculum design

authors:
Agastya Kalra
Abdullah Rashwan
Wei-Shou Hsu
Pascal Poupart
Prashant Doshi
Georgios Trimponias

title: Online structure learning for feed-forward and recurrent sum-product networks. NeurIPS, 2018

authors:
Nan Rosemary Ke
Olexa Bilaniuk
Anirudh Goyal
Stefan Bauer
Hugo Larochelle
Bernhard Schölkopf
Michael C Mozer
Chris Pal
Yoshua Bengio

title: Learning neural causal models from unknown interventions

authors:
Doga Kisa
Guy Van den Broeck
Arthur Choi
Adnan Darwiche

title: Probabilistic sentential decision diagrams

venue: In KR,

authors:
Daniel Koch
Robert S Eisinger
Alexander Gebharter

title: A causal bayesian network model of disease progression mechanisms in chronic myeloid leukemia

venue: Journal of theoretical biology,

authors:
Kevin B Korb
Ann E Nicholson

title: Bayesian artificial intelligence

venue: CRC press,

authors:
Steffen L Lauritzen
David J Spiegelhalter

title: Local computations with probabilities on graphical structures and their application to expert systems

venue: Journal of the Royal Statistical Society: Series B (Methodological),

authors:
Amélie Levray
Vaishak Belle

title: Learning credal sum-product networks

venue: AKBC,

authors:
Stephen L Morgan
Christopher Winship

title: Counterfactuals and causal inference

title: Learning bayesian networks

venue: Pearson Prentice Hall Upper Saddle River, NJ,

authors:
Mike Oaksford
Nick Chater

title: Causal models and conditional reasoning

venue: Oxford library of psychology. The Oxford handbook of causal reasoning,

authors:
Michael Oberst
David Sontag

title: Counterfactual off-policy evaluation with gumbel-max structural causal models

authors:
Ioannis Papantonis
Vaishak Belle

title: Interventions and counterfactuals in tractable probabilistic models: Limitations of contemporary transformations

venue: arXiv preprint arXiv:2001.10905,

authors:
Iago París
Raquel Sánchez-Cauce
Francisco Javier Díez

title: Sum-product networks: A survey

venue: arXiv preprint arXiv:2004.01167,

title: From bayesian networks to causal networks. In Mathematical models for handling partial knowledge in artificial intelligence

title: The seven tools of causal inference, with reflections on machine learning

venue: Communications of the ACM,

authors:
Judea Pearl
Madelyn Glymour
Nicholas P Jewell

title: Causal inference in statistics: A primer

authors:
Robert Peharz
Antonio Vergari
Karl Stelzner
Alejandro Molina
Xiaoting Shao
Martin Trapp
Kristian Kersting
Zoubin Ghahramani

title: Random sum-product networks: A simple and effective approach to probabilistic deep learning

venue: In UAI,

authors:
Jonas Peters
Dominik Janzing
Bernhard Schölkopf

title: Elements of causal inference

authors:
Hoifung Poon
Pedro Domingos

title: Sum-product networks: A new deep architecture

venue: IEEE International Conference on Computer Vision Workshops (ICCV Workshops),

title: On the hardness of approximate reasoning

venue: Artificial Intelligence,

title: Causality for machine learning

venue: arXiv preprint arXiv:1911.10500,

authors:
Karthikeyan Shanmugam
Murat Kocaoglu
Alexandros G Dimakis
Sriram Vishwanath

title: Learning causal graphs with small interventions

authors:
Xiaoting Shao
Alejandro Molina
Antonio Vergari
Karl Stelzner
Robert Peharz
Thomas Liebig
Kristian Kersting

title: Conditional sum-product networks: Imposing structure on deep probabilistic architectures

authors:
Amit Sharma
Emre Kiciman

title: Dowhy: An end-to-end library for causal inference

venue: arXiv preprint arXiv:2011.04216,

title: The interpretation of interaction in contingency tables

authors:
G Van den Broeck
N Di Mauro
A Vergari

title: Tractable probabilistic models: Representations, algorithms, learning, and applications

venue: Tutorial at UAI,

authors:
Kevin Xia
Kai-Zhan Lee
Yoshua Bengio
Elias Bareinboim

title: The causal-neural connection: Expressiveness

venue: learnability, and inference

authors:
Kun Zhang
Bernhard Schölkopf
Peter Spirtes
Clark Glymour

title: Learning causality and causalityrelated learning: some recent progress

venue: National science review,

authors:
Han Zhao
Mazen Melibari
Pascal Poupart

title: On the relationship between sum-product networks and bayesian networks

heading: 1 Introduction

text:
Identifying causal relationships between variables in observational data is one of the fundamental and well-studied problem in machine learning. There have been several great strides in causality [Granger, 1969, Pearl, 2009, Bareinboim and Pearl, 2016] over the years specifically characterized by efforts that focused on reasoning about interventions [Hagmayer et al., 2007, Dasgupta et al., 2019] and counterfactuals [Morgan and Winship, 2015, Oberst and Sontag, 2019].
The notion of causality has long been explored in the realm of probabilistic models [Oaksford and Chater, 2017, Beckers and Halpern, 2019] with a special focus on graphical models, called causal Bayesian networks (CBNs) [Heckerman et al., 1995, Neapolitan, 2004, Pearl, 1995, Acharya et al., 2018]. CBNs have widely been applied to infer causal relationships in high-impact diverse applications such as disease progression [Koch et al., 2017], ecological risk assessment [Carriger and Barron, 2020] and more recently Covid-19 [Fenton et al., 2020, Feroze, 2020] to name a few. Although successful, classical CBN models are difficult to scale and also suffer from the problem of intractable inference. Recently, tractable probabilistic models such as probabilistic sentential decision diagrams [Kisa et al., 2014] and sum-product networks [Poon and Domingos, 2011] have emerged, which guarantee that conditional marginals can be computed in time linear in the size of the model. While weaving in the notion of interpretability, the computational view on probabilistic models allows one to exploit ideas from deep learning and can thus be very useful in modelling complex problems.
Recently, there has been an effort to take advantage of this tractability to reason for causality. Zhao et al. [2015] showed how to compile back and forth between sum-product networks (SPNs) and Bayesian networks (BNs). Although this opened up a whole range of possibilities for tractable
35th Conference on Neural Information Processing Systems (NeurIPS 2021), virtual.
causal models, Papantonis and Belle [2020] argued that such conversion leads to degenerated BNs thereby rendering it ineffective for causal reasoning. For the considered compilation of SPNs to BNs, this is indeed the case since a bipartite graph between the hidden and observed variables loses the relationships between the actual variables. Thus, either a new compilation method for transforming between tractable and causal model or alternatively a method to condition the probabilistic models directly on the do-operator to obtain interventional distributions, P (y| do(x)), is being required. Here, we consider the latter strategy and extend the idea of conditionally parameterizing SPNs [Shao et al., 2019] by conditioning on the do-operator while predicting the complete set of observed variables therefore capturing the effect of intervention(s). The resulting interventional sum-product networks (iSPNs) take advantage of both the expressivity, due to the neural network, and the tractability, due to the SPN in order to capture the interventional distributions faithfully. This shows that the dream of tractable causal models is not insurmountable, since iSPNs are causal. Pearl [2019] defined a three-level causal hierarchy that separates association (purely statistical level 1) from intervention (level 2) and counterfactuals (level 3), and argued that the latter two levels involve causal inference. iSPNs are a modelling scheme for arbitrary interventional distributions. They thus belong to level 2 and, in turn, are causal. So, while SPNs are not universal function approximators and the use of gate functions turns them into universal approximators, we go one step ahead and make the first effort towards introducing causality to SPNs without the need for compilation to Bayesian networks, as the functional approximator subsumes the do-operator. Fig. 1 shows an example of the effectiveness of iSPNs to capture interventional distributions on the ASIA data set. Our extensive experiments against strong baselines demonstrate that our method is able to capture ideal interventional distributions.
To summarize, we make the following contributions:
1. We introduce iSPNs, the first method that applies the idea of tractable probabilistic models to causality without the need for compilation and that accordingly generate interventional distributions w.r.t. the provided causal structure.
2. We formulate the inductive bias necessary for turning conditional SPN into iSPN while taking advantage of the neural network modelling capacities within the gating nodes for for modelling interventional distributions while capturing all influences within the given intervention (i.e., consequences propagated through the structural hierarchy).
3. We show that, by construction, iSPNs can identify any interventional distribution permitted by the underlying structural causal model due to the inducted bias on the interface modalities in junction with the universal function approximation of the underlying gated SPN.
We proceed as follows. We start by reviewing the basic concepts required and related work, namely the tractable model class of sum-product networks and key concepts from causality. Then we motivate using a newly curated causal model themed around personal health. Subsequently, we introduce iSPNs formally and prove that by construction they are capable of approximating any do-query (given corresponding data). Before concluding, we present our experimental evaluation in which we
challenge iSPN in its density estimation and causal inference capabilities against various baselines. We make our code publically available at: https://github.com/zecevic-matej/iSPN.

heading: 2 Background and Related Work

text:
Let us briefly review the background on tractable probabilistic models and causal models used in subsequent sections for developing our new model class based on CSPNs that allow for identifying causal quantities i.e., interventional distributions.
Notation. We denote indices by lower-case letters, functions by the general form g(·), scalars or random variables interchangeably by upper-case letters, vectors, matrices and tensors with different boldface font v,V,V respectively, and probabilities of a set of random variables X as p(X).
Sum-Product Networks (SPNs). Introduced by Poon and Domingos [2011], generalizing the notion of network polynomials based on indicator variables λX=x(x) ∈ [0, 1] for (finite-state) RVs X from [Darwiche, 2003], Sum-Product Networks (SPNs) represent a special type of probabilistic model that allows for a variety of exact and efficient inference routines. Generally, SPNs are considered as directed acyclic graphs (DAG) consisting of product, sum and leaf (or distribution) nodes whose structure and parameterization can be efficiently learned from data to allow for efficient modelling of joint probability distributions p(X). Formally a SPN S = (G,w) consists of non-negative parameters w and a DAG G = (V,E) with indicator variable λ leaf nodes and exclusively internal sum and product nodes given by,
S(λ) = ∑
C∈ch(S)
wS,CC(λ) P(λ) = ∏
C∈ch(S)
C(λ), (1)
where the SPN output S is computed at the root node (S(λ) = S(x)) and the probability density for x is p(x) = S(x)∑
x′∈X S(x′) . They are members of the family of probabilistic circuits [Van den
Broeck et al., 2019]. A special class, to be precise, that satisfies properties known as completeness and decomposability. Let N denote a node in SPN S, then
sc(N) =
{ {X} if N is IV (λX=x)⋃
C∈ch(N) sc(C) else (2)
is called the scope of N and
∀S ∈ S : (∀C1,C2 ∈ ch(S) : sc(C1) = sc(C2)) (3) ∀P ∈ S : (∀C1,C2 ∈ ch(S) : C1 ̸= C2 =⇒ sc(C1) ∩ sc(C2) = ∅) (4)
are the completeness and decomposability properties respectively. Since their introduction, SPNs have been heavily studied such as by [Trapp et al., 2019] that present a way to learn SPNs in a Bayesian realm whereas [Kalra et al., 2018] learn SPNs in an online setting. Several different types of SPNs have also been studied such as Random SPN [Peharz et al., 2020], Credal SPNs [Levray and Belle, 2020] and Sum-Product-Quotient Networks [Sharir and Shashua, 2018]) to name a few. For more details readers are referred to the survey of París, Sánchez-Cauce, and Díez [2020]. On another note, Gated or Conditional SPNs (CSPNs) are deep tractable models for estimating multivariate, conditional probability distributions p(Y|X) over mixed variables Y [Shao et al., 2019]. They introduce functional gate nodes gi(X) that act as a functional parameterization of the SPN’s information flow and leaf distributions given the provided evidence X.
Causal Models. A Structural Causal Model (SCM) as defined by Peters et al. [2017] is specified as C := (S, PN) where PN is a product distribution over noise variables and S is defined to be a set of d structural equations
Xi := fi(pa(Xi), Ni), where i = 1, . . . , d (5)
with pa(Xi) representing the parents of Xi in graph G(C). An intervention on a SCM C as defined in (5) occurs when (multiple) structural equations are being replaced through new non-parametric functions f̂(p̂a(Xi), N̂i) thus effectively creating an alternate SCM Ĉ. Interventions are referred to as imperfect if p̂a(Xi) = pa(Xi) and as atomic if f̂ = a for a ∈ R. An important property
of interventions often referred to as "modularity" or "autonomy"1 states that interventions are fundamentally of local nature, formally
pC(Xi | pa(Xi)) = pĈ(Xi | pa(Xi)) , (6)
where the intervention of Ĉ occured on variable Xk opposed to Xi. This suggests that mechanisms remain invariant to changes in other mechanisms which implies that only information about the effective changes induced by the intervention need to be compensated for. An important consequence of autonomy is the truncated factorization
p(V ) = ∏
i/∈S p(Xi | pa(Xi)) (7)
derived by Pearl [2009], which suggests that an intervention S introduces an independence of an intervened node Xi to its causal parents. Another important assumption in causality is that causal mechanisms do not change through intervention suggesting a notion of invariance to the cause-effect relations of variables which further implies an invariance to the origin of the mechanism i.e., whether it occurs naturally or through means of intervention [Pearl et al., 2016].
A SCM C is capable of emitting various mathematical objects such as graph structure, statistical and causal quantities placing it at the heart of causal inference, rendering it applicable to machine learning applications in marketing [Hair Jr and Sarstedt, 2021]), healthcare [Bica et al., 2020]) and education [Hoiles and Schaar, 2016]. A SCM induces a causal graphG, an observational/associational distribution pC, can be intervened upon using the do-operator and thus generate interventional distributions pC;do(...) and given some observations v can also be queried for interventions within a system with fixed noise terms amounting to counterfactual distributions pC|V=v;do(...). To query for samples of a given SCM, the structural equations are being simulated sequentially following the underlying causal structure starting from independent, exogenous variables and then moving along the causal hierarchy of endogenous variables (i.e., following the causal descendants).
The work closest to our work is by Brouillard et al. [2020] although it solves the different problem of causal discovery. Causality for machine learning has recently gained a lot of traction [Schölkopf, 2019] with the study of both interventions [Shanmugam et al., 2015] and counterfactuals [Kusner et al., 2017] gaining speed. For more details the readers are referred to [Zhang et al., 2018].

heading: 3 Interventional SPNs

text: Now we are ready to develop interventional SPNs (iSPNs). To this end, we re-introduce the importance of adaptability of models to interventional queries and present a newly curated synthetic data set to both motivate and validate the formalism of iSPNs that, through over-parametric extension of SPNs, allows them to adhere to causal quantities.

heading: 3.1 Adaptation to Causal Change

text:
Peters et al. [2017] motivated the necessity of causality via the adequate generalizability of predictive models. Specifically, consider a simple regression problem f(a) = b with data vectors a,b ∈ Rk that are strongly positively correlated in some given region, e.g. (1 < a <∞, 1 < b <∞). Now a query is posed outside the data support, e.g. (0, f(0)). As argued by Peters et al., the underlying data generating processes can be an ambiguous causal process i.e., the data at hand can be explained by two different causal structures being either A→ B or a common confounder with A← C → B. Assuming the wrong causal structure or ignoring it altogether could be fatal, therefore, any form of generalization out of data support requires assumptions to be made about the underlying causal structure. We adopt this point of view and further argue that ignoring causal change(s) in a system, i.e., the change of structural equation(s) underlying the system, can lead to a significant performance decrease and safety hazards2. Therefore, it is important to account for distributional changes present in the data due to experimental (thus interventional) settings.
1See Section 6.6 in [Peters et al., 2017]. 2This extended notion of performance degeneration through ignorance to the underlying causality is priori-
tized in this paper.
Consequently, we consider the learning problem, where the given data samples have been generated by different interventions do(Uj = uj) in a common SCM C while the induced mutilated causal graphs G(C, do(Uj = uj)) are assumed to be known, such that the trained model is capable of at least inferring all involved causal distributions p(V (C) | do(Uj = uj)) with V being the variables.

heading: 3.2 Data Generating Process

text:
To both validate and demonstrate the expressivity of interventional SPNs in modelling arbitrary interventional distributions, we curate a new causal data set based on the SCM C presented in Fig. 2(a), which we subsequently refer to as Causal Health data set. The SCM encompasses four different structural equations of the form Vi = f(pa(Vi), Ni), where pa(Vi) are the parents of variable Vi and Ni are the respective noise terms that form a factor distribution PN1:N i.e., the Ni are jointly independent. Now, the SCM C describes the causal relations of an individual’s health and mobility attributes with respect to their age and nutrition.
Note that the Causal Health data set does not impose assumptions over the type of random variables or functional domains of the structural equations3, which additionally constraints a learned model to adapt flexibly. While we generally do not restrict our method to any particular type of intervention, the following mainly considers perfect interventions as introduced in Sec. 2.
Perfect interventions fully remove the causal mechanism of the parents of a given node, which is consistent with the idea behind randomized controlled trials (RCTs) where the given intervention randomizes the given specific node, often referred to as gold standard in causality related literature. We consider the special case of uniform randomization, i.e., uniform across the domain of the given variable4. An intervention performed on one node immediately changes the population and, thus, has a major effect on the generating processes of subsequent causal mechanisms in the respective causal sequence of events.
To provide the reader with a concrete example of interventions within the causal health data set, consider the following: In concern of a virus infection the individuals of the Causal Health study should be vaccinated. The vaccine is expected to have side-effect(s), however, it has been poorly designed and has reached the population with the capability of completely changing the individual’s
3Assumptions on the func. form of structural eq. are crucial for identification (Tab. 7.1 [Peters et al., 2017]) 4Note that for binary variables this amounts to Bernoulli B( 1
2 ).
health state. The observed changes do not show any form of pattern and are therefore assumed to be random. A young fit person could thus become sick, while an old person might feel better health wise. This sudden change will have an effect on the individual’s mobility and also be independent of their age and nutrition. Such a scenario is mathematically being captured through do(H = U(H)) where U(·) is the uniform distribution over a given domain.

heading: 3.3 Introducing Interventional SPNs

text:
After motivating both the importance and the occurrences of interventions within relevant systems, we now start introducing interventional SPNs (iSPNs).
Definition of iSPN. As motivated in Sec. 1, the usage of the compilation method from [Zhao et al., 2015] for causal inference within SPN is arguably of degenerate5 nature given the properties of the compilation method [Papantonis and Belle, 2020]. While the results of Papantonis and Belle [2020] are arguably negative, there exists yet no proof of non-existence of such a compilation method and as the authors point out in their argument for future lines of research in this direction, a model class extension poses a viable candidate for overcoming the problems of using SPN for causal inference.
While agreeing on the latter aspect, we do not go the “compilation road” but extend the idea of conditional parameterization for SPN [Shao et al., 2019] by conditioning on a modified form of the do-operator introduced by Pearl [2009] while predicting the complete set of observed variables.
Mathematically, we estimate p(Vi | do(Uj = uj)) by learning a non-parametric function approximator f(G;θ) (e.g. neural network), which takes as input the (mutilated) causal graph G ∈ {0, 1}N×N encoded as an adjacency matrix, to predict the parameters ψ of a SPN g(D;ψ) that estimates the density of the given data matrix {Vk}Kk = D ∈ RK×N . With this, iSPNs are defined as follows: Definition 1 (Interventional Sum-Product Network). An interventional sum-product network (iSPN) is the joint model m(G,D) = g(D;ψ = f(G;θ)), where g(·) is a SPN, f(·) a non-parametric function approximator and ψ = f(G) are shared parameters.
They are called interventional because we consider it to be a causal model given its capability of answering queries from the second level of the causal hierarchy [Pearl, 2019], namely, that of interventions6. The shared parameters ψ allow for information flow during learning between the conditions and the estimated densities. Setting the conditions such that they contain information about the interventions, in the form of the mutilated graphs G, effectively renders f to subsume a sort of do-calculus in the spirit of truncated factorization shown in Eq.(7) i.e., the gate model acts as an estimand selector. Generally, we note that our formulation allows for different function and density estimators f, g. We choose f to be a neural network for two reasons (1) their empirically established capability to act as causal sub-modules (e.g. Ke et al. [2019] use a cohort of neural nets to mimic a set of structural equations, thus, a SCM) and (2) their model capacity being a universal function approximator, while we choose g to be a SPN for its tractability properties for inference.
We have argued the importance of adaptability to interventional changes within the causal system and intend now to prove that iSPN are capable of approximating these different causal quantities.
Proposition 1 (Expressivity). Assuming autonomy and invariance, an iSPN m(G,D) is able to identify any interventional distribution pG(Vi = vi | do(Uj = uj)), permitted by a SCM C through interventions, with knowledge of the mutilated causal graph Ĝ and data D generated from the intervened SCMs by modelling the conditional distribution pĜ(Vi = vi | Uj = uj).
Proof. It follows directly from the definition of the do-calculus [Pearl, 2009] that pG(Vi = vi | do(Uj = uj)) = pĜ(Vi = vi | Uj = uj) where Ĝ is the mutilated causal graph according to the intervention do(Uj = uj) i.e., observations in the intervened system are akin to observations made when intervening on the system. Given the mutilated causal graph Ĝ (as adjacency matrix), the only remaining aspect to show is that the density estimating SPN can approximate a joint probability p(X) using D. This naturally follows from [Poon and Domingos, 2011].
5A bipartite graph in which the actual variables of interest are not connected is called degenerate. 6The first level of the causal hierarchy —association— is considered to be purely statistical.
The expressivity of iSPN stems from both the capacities of gate function and the knowledge of intervention as well as availability of respective data. As an important remark, causal inference is often interested in estimating interventional distributions, i.e, causal quantities from purely observational models. Therefore, an alternative formulation to Prop. 1 would be to replace the knowledge of the intervened causal structure G with knowledge on a valid adjustment set. In the following, we only consider the direct setting where actual interventional data from the system is assumed to be captured7 thereby freeing the investigation of iSPN from the independent research around hidden confounding.
Universal Function Approximation (UFA). The gating nodes of CSPNs extend SPNs in a way that allows them to also induce functions which are universal approximators. For instance, using threshold gates xi ≤ c ∈ R, one can realize testing arithmetic circuits [Choi and Darwiche, 2018] which have been proven to be universal approximators - rendering iSPN to be UFA by construction.
Learning of iSPN. An interventional sum-product network is being learned using a set of mixeddistribution samples generated from simulating the Causal Health SCM for different interventions, where the observational case is considered to be equivalent to an intervention on the empty set. The parameters θ,ψ of the iSPN describe the weights of the gate nodes and the distributions at the leaf nodes. The full model m is differentiable if the provided gate function f and each of the leaf models of g are differentiable. Therefore, to train an iSPN, as depicted in Fig. 2(b), it is sufficient to optimize the conditional log-likelihood end-to-end using gradient based optimization techniques. We assess the performance of our learned model through inspection of the adaptation of the model to the different interventions manifesting in the resulting marginals p(Vi | do(Uj)). As can be observed in Fig. 2(c) or alternatively in Fig. 4 (top row), the learned iSPN successfully adapts to both the interventions as well as its consequences. Considering for instance the intervention p(Vi | do(F = B( 12 ))) which removes the edge A→ F and thus renders Age (A) and Food Habits (F ) independent. Given the drastic population change in F and the fact that the Health (H) of an individual is causally dependent on both A and F a significant change in H is being expected. Indeed, both H and Mobility (M ), being a causal child of H , broaden distribution wise and also these subsequent changes are captured correctly.
Causal Estimation. The algebraic-graphical do-calculus [Pearl, 2009] is complete in that it can find estimands for any identifiable query in a finite amount of transformation. An SPN, and thereby also iSPN, is capable of providing estimates to said demands when given observational data (L1 on the PCH). Consider for instance the well-known non-Markovian "Napkin" graph G = {W → Z → X → Y,W ∗↔ X,W ∗↔ Y } where ∗↔ denotes a confounded relation. The causal effect is identified as p(y| do(x)) = ( ∑ w p(x|w, z)p(w))−1( ∑ w p(y, x|w, z)p(w)) using the do-calculus where each of the r.h.s. components can be modelled by (i)SPN respectively. However, iSPN are also capable of directly expressing the causal quantity p(y| do(x)) as proven in Prop.1, similar to other neural-causal models like CausalGAN [Kocaoglu et al., 2019] or MLP-based NCM [Xia et al., 2021].
Tractability. Inference in BNs and Markov networks is at least an NP-problem and existing exact algorithms have worst-case exponential complexity [Cooper, 1990, Roth, 1996], while SPNs can do inference in time proportional to the number of links in the graph. I.e., SPNs in general are able to compute any marginalization and conditioning query in time linear of the model’s representation size r, that is O(r). We deploy simple neural network architectures for which the runtime for the forward-pass is that of matrix-multiplication i.e., the time complexity scales cubically in the size of the input n, that is O(n3). The gradient descent procedure that involves forward and backward passes, assuming m gradient descent iterations, scales to O(mn3) which is the overall complexity we achieve during training phase. For SPN we deploy a random SPN structure which circumvents structure learning as such. Therefore, overall, training will generally be in the O(mn3r) regime while any causal query (that is, both L1 observational and L2 interventional in our case) will be answered within O(n3r). However, assuming that the weights of the random SPN were already initialized by the neural parameter-provider in a previous step, any causal query becomes answerable in O(r). Since asking for inferences/queries q within a single distribution (q → d ∈ Li) are more common than changes between distributions (d, d̂ ⊂ Li), this linear complexity by our tractable model is being leveraged fully for performing causal inference.
Discussion. To reconsider and answer the general question of why the modelling of a conditional distribution via an over-parameterized architecture is a sensible idea consider the following. One can
7For details on the remarked alternative formulation consider the Supplement.
represent a conditional distribution p(Y | X) by applying Bayes Rule to a joint distribution density model (e.g. a regular SPN) p(Y | X) = p(Y,X)p(X) . However, this assumes non-empty support i.e., p(X) > 0. Furthermore, the joint distribution p(Y,X) optimizes all possibly derivable distributions, diminishing single distr. expressivity. Therefore, our considered formulation of a gate model allows for effectively subsuming the do-operator i.e., the gate model orchestrates the do queries such that the density estimator can easily switch between different interventional distributions. While not introducing specific limitations, general CSPN limitation regarding OOD generalization are inherited.

heading: 4 Experimental Results

text:
The assumptions made in causality usually require control over the data generating process which is almost never readily available in the real world. This amounts to scarcity of the available public data sets and also their implications for transfers to the real world and even when available, they are usually artificially generated as some causal extension of a known, pre-existing data set (e.g. MorphoMNIST data set introduced by Castro et al. [2019]). While it is difficult to consider realworld-esque experimental settings for causal models, we do not restrict our investigations of iSPN to rather specific problem settings like certain noise or decision variable instances (which is common in causal inference). The evaluation is performed on data sets with varying number of variables and in both continuous and discrete domains. For the introduced causal health data set, we even consider arbitrary underlying noise distributions. We have made our code repository publicly available8.
Data Sets. We evaluate iSPNs on four data sets. Three benchmarks: ASIA (A) [Lauritzen and Spiegelhalter, 1988] with 8 variables, Earthquake (E) with 5 variables and Cancer (C) [Korb and Nicholson, 2010] with 5 variables. One newly curated synthetic causal health (H) data set with 4 variables. More information about the data sets is presented in the Supplement.
Baselines. For generative capacities, we compare our method against Mixture Density Networks (MDN) [Bishop, 1994] and Masked Autoencoder for Density Estimation (MADE) [Germain et al., 2015]. Both methods are expressive, parametric neural network based approaches for density estimation. Generally, the causality for machine learning literature suggests a strong favor for neural based function approximators for modelling causal mechanisms [Ke et al., 2019]. For causal capacities, we compare our method against the renown causal baselines from CausalML [Chen et al., 2020] and DoWhy [Sharma and Kiciman, 2020] for the modelling of average treatment effects (ATE) [Pearl, 2009, Peters et al., 2017] considered to be a gold standard task within causal inference.
Protocol and Parameters. To account for reproducibility and stability of the presented results, we used learned models for five different random seeds per configuration. For means of visual clarity, the competing baselines MDN and MADE only present the best performing seed while iSPN is being presented with a mean plot and the standard deviation. The CBN, which performs exact inference according to the do-calculus (while best performing) has to be considered as gold standard and is therefore not part of the visualization. Furthermore, as it cannot compare in terms of feasibility. The considered interventions were of uniform nature. Each block of four columns represents a variable being randomized uniformly. We deployed a RAT-SPN [Peharz et al., 2020] selecting the leaf node distributions to be Gaussian distributions. For further experimental details consider the Supplement.
8https://github.com/zecevic-matej/iSPN
Our empirical analysis investigates iSPN for the following questions: Q1: How is the estimation quality for interv. distributions? Q2: How important is the model’s capacity? Q3: How is the runtime performance? Q4/5: How is the performance relative to state-of-the-art generative and causal models? Q6: How does the model adapt to different types of interventions?
(Q1. Precise Estimation of do-influenced variables) The density functions learned by iSPN (see Fig. 4) fit with a high degree of precision as visualized by the difference in the peak of the modes of the learned distribution and that of the ground truth. While our visual analysis is arguably the superior method of evaluation as it conveys information about how the distributions of interest compare (which is feasible due to marginal inference and the locality of interventions within a SCM), we have also considered numerical evaluation criteria like the Jenson-Shannon-Divergence on which (as visually confirmed) iSPN outperforms the baselines (see Tab.1). For more detailled observation and interpretation consider the Supplement.
(Q2. Capacity Ablation Study) We test the robustness of iSPN as the size of the associated SPN g(D;ψ) is varied. We obtain 5 different iSPNs for each of the 4 data sets by using 5 different numbers of sum node weights, 600, 1200, 1800, 2400, 3200, effectively changing the capacity of the parameter-sharing neural network f(G;θ). We observe iSPNs to be robust to varying hyperparameters that control the size of the SPN g(·) and effectively the complexity of the associated function approximator f(·). More details and figures are in the Supplement. (Q3. Comparison of running times) SPNs are tractable by design as long the networks size is polynomial in the input size. The superiority in running time also becomes apparent during training on the same (Causal Health) data set where we observe the mean run times over 50 passes of the whole data set to be significantly faster than competing methods (see Fig.3).
(Q4. Comparison to Generative models) We compare the performances in terms of precision of fit of the learned distributions as well as the flexibility of the models. iSPN outperforms the baselines across all 4 data sets both in precision of the fit of the learned density functions and flexibility of adaptation to the different interventions as seen in figure 4. In our experiments, MDN had the worst performance with estimated densities being consistently and significantly different from the ground truth, as the model settled for an average distribution across all interventions. MADE is able to estimate to high precisions but showed to be generally inconsistent across the experimental settings.
(Q5. Comparison to Causal models) We compare the numerical results as seen in figure 5 and observe that iSPN matches the performance of the causal baselines. The simple regressor employed by CausalML fails in the confounding case as it estimates wrongly the conditional, both DoWhy and iSPN can handle even the more difficult Simpson’s paradox [Simpson, 1951] scenario. An analytical derivation for the ATE is exampled in the Supplement.
(Q6. Different Types of Intervention) By construction, iSPNs are capable to handle arbitrary interventions and our empirical results corroborate this impression. Fig.6 shows the training of multiple models on different interventions alongside two example interventions and their appearance in the marginal distributions. Depending on the training setup, e.g. how many different interventional distributions need be learned, any single intervention might become more easily optimizable since the model can exploit similarities between distributions. For a more detailled elaboration and also visualizations of other interventional distributions (including atomic and non-continuous interventions), we direct the reader to our extensive supplementary material.

heading: 5 Conclusions

text:
We presented a way to connect causality with tractable probabilistic models by using sum-product networks parameterized by universal function approximators in the form of neural networks. We show that our proposed method can adapt to the underlying causal changes in a given domain and generate near perfect interventional distributions irrespective of the data distribution and the intervention type thereby exhibiting flexibility. Our empirical evaluation shows that our method is able to precisely estimate the conditioned variables and outperform generative baselines.
Finding a different compilation method for SPNs such as by making use of tree CBNs is important for learning pure causal probabilistic models. Testing our method on larger real world causal data sets is an interesting direction. Finally, using rich expert domain knowledge in addition to observational data is essential for causality and extending our method to incorporate such knowledge is essential.
Acknowledgments and Disclosure of Funding
The authors thank the anonymous reviewers for their valuable feedback. This work was supported by the ICT-48 Network of AI Research Excellence Center “TAILOR" (EU Horizon 2020, GA No 952215) and by the Federal Ministry of Education and Research (BMBF; project “PlexPlain”, FKZ 01IS19081). It benefited from the Hessian research priority programme LOEWE within the project WhiteBox, the HMWK cluster project “The Third Wave of AI.” and the Collaboration Lab “AI in Construction” (AICO) of the TU Darmstadt and HOCHTIEF. Furthermore, the authors gratefully acknowledge the support of 1R01HD101246 from NICHD and W911NF2010224 from US Army Research Office (ARO).

title: Interventional Sum-Product Networks: Causal Inference with Tractable Probabilistic Models