WHAT DOES IT TAKE TO GENERATE NATURAL TEXTURES?
Natural image generation is currently one of the most actively explored fields in Deep Learning. Many approaches, e.g. for state-of-the-art artistic style transfer or natural texture synthesis, rely on the statistics of hierarchical representations in supervisedly trained deep neural networks. It is, however, unclear what aspects of this feature representation are crucial for natural image generation: is it the depth, the pooling or the training of the features on natural images? We here address this question for the task of natural texture synthesis and show that none of the above aspects are indispensable. Instead, we demonstrate that natural textures of high perceptual quality can be generated from networks with only a single layer, no pooling and random filters.
1 INTRODUCTION
During the last two years several different approaches towards natural image generation have been suggested, among them generative adversarial networks (Goodfellow et al., 2014; Chen et al., 2016), probabilistic generative models like the conditional PixelCNN (van den Oord et al., 2016b;a) or maximum entropy models that rely on the representations of deep neural networks (e.g. Gatys et al., 2015b; Johnson et al., 2016; Ulyanov et al., 2016). The latter approach has been particularly groundbreaking for artistic style transfer and natural texture generation (e.g. Gatys et al., 2015a;b) and has the potential to uncover the regularities that supervisedly trained deep neural networks infer from natural images.
For the sake of clarity and concreteness, this paper will focus on natural texture synthesis. Parametric texture models aim to uniquely describe each texture by a set of statistical measurements that are taken over the spatial extent of the image. Each image with the same spatial summary statistics should be perceived as the same texture. Consequently, synthesizing a texture corresponds to finding a new image that reproduces the summary statistics inferred from the reference texture. Starting from Nth-order joint histograms of the pixels by Julesz (1962), many different statistical measures have been proposed (see e.g. Heeger & Bergen, 1995; Portilla & Simoncelli, 2000). The quality of the synthesized textures is usually determined by human inspection; the synthesis is successful if a human observer cannot tell the reference texture from the synthesized ones.
The current state of the art in parametric texture modeling (Gatys et al., 2015a) employs the hierarchical image representation in a deep 19-layer convolutional network (Simonyan & Zisserman (2014); in the following referred to as VGG network) that was trained on object recognition in natural images(Russakovsky et al. (2015)). In this model textures are described by the raw correlations between feature activations in response to the texture image from a collection of network layers (see section 5 for details). Since its initial reception several papers explored which additional elements or constraints can further increase the perceptual quality of the generated textures (Berger & Memisevic, 2016; Liu et al., 2016; Aittala et al., 2016). In this work we go the opposite way and ask which elements of the original texture synthesis algorithm (Gatys et al., 2015a) are absolutely indispensable.
In particular two aspects have been deemed critical for natural texture synthesis: the hierarchical multi-layer representation of the textures, and the supervised training of the feature spaces. Here we show that neither aspect is imperative for texture modeling and that in fact a single convolutional layer with random features can synthesize textures that often rival the perceptual quality of Gatys et al. (2015a). This is in contrast to earlier reports (Gatys et al., 2015a) that suggested that networks with random weights fail to generate perceptually interesting images. We suggest that this discrepancy originates from a more elaborate tuning of the optimization procedure (see section 4).
Our main contributions are:
• We present a strong minimal baseline for parametric texture synthesis that solely relies on a single-layer network and random, data-independent filters. • We show that textures synthesized from the baseline are of high quality and often rival
state-of-the-art approaches, suggesting that the depth and the pre-training of multi-layer image representations are not as indispensable for natural image generation as has previously been thought. • We test and compare a wide range of single-layer architectures with different filter-sizes and
different types of filters (random, hand-crafted and unsupervisedly learnt filters) against the state-of-the-art texture model by Gatys et al. (2015a). • We utilize a quantitative texture quality measure based on the synthesis loss in the VGG-
based model (Gatys et al., 2015a) to replace the common-place evaluation of texture models through qualitative human inspection. • We discuss a formal generalization of maximum entropy models to account for the natural
variability of textures with limited spatial extent.
2 CONVOLUTIONAL NEURAL NETWORK
If not mentioned otherwise, all our models employ single-layer CNNs with standard rectified linear units (ReLUs) and convolutions with stride one, no bias and padding (f − 1)/2 where f is the filter-size. This choice ensures that the spatial dimension of the output feature maps is the same as the input. All networks except the last one employ filters of size 11× 11× 3 (filter width × filter height × no. of input channels), but the number of feature maps as well as the selection of the filters differ:
• Fourier-363: Each color channel (R, G, B) is filtered separately by each element Bi ∈ R11×11 of the 2D Fourier basis (11×11 = 121 feature maps/channel), yielding 3·121 = 363 feature maps in total. More concretely, each filter can be described as the tensor product Bi ⊗ ek where the elements of the unit-norm ek ∈ R3 are all zero except one. • Fourier-3267: All color channels (R, G, B) are filtered simultaneously by each element Bi of the 2D Fourier basis but with different weighting terms wR, wG, wB ∈ [1, 0,−1], yielding 3 · 3 · 3 · 121 = 3267 feature maps in total. More concretely, each filter can be described by the tensor product Bi ⊗ [wR, wG, wB ]. • Kmeans-363: We randomly sample and whiten 1e7 patches of size 11 × 11 from the
Imagenet dataset (Russakovsky et al., 2015), partition the patches into 363 clusters using k-means (Rubinstein et al., 2009), and use the cluster means as convolutional filters. • Kmeans-3267: Same as Kmeans-363 but with 3267 clusters. • Kmeans-NonWhite-363/3267: Same as Kmeans-363/3267 but without whitening of the
patches. • Kmeans-Sample-363/3267: Same as Kmeans-363/3267, but patches are only sampled
from the target texture. • PCA-363: We randomly sample 1e7 patches of size 11 × 11 from the Imagenet dataset
(Russakovsky et al., 2015), vectorize each patch, perform PCA and use the set of principal axes as convolutional filters. • Random-363: Filters are drawn from a uniform distribution according to (Glorot & Bengio,
2010), 363 feature maps in total. • Random-3267: Same as Random-363 but with 3267 feature maps.
Original Fourier-363, 11x11 K-means-363, 11x11 K-samples-363, 11x11 K-NW-363, 11x11 Random-363, 11x11 PCA-363, 11x11
Gatys et al. Fourier-3267, 11x11 K-means-3267, 11x11 K-samples-3267, 11x11 K-NW-3267, 11x11 Random-3267, 11x11 Random multi-scale
The networks were implemented in Lasagne (Dieleman et al., 2015; Theano Development Team, 2016). We remove the DC component of the inputs by subtracting the mean intensity in each color channel (estimated over the Imagenet dataset (Russakovsky et al., 2015)).
3 TEXTURE MODEL
The texture model closely follows (Gatys et al., 2015a). In essence, to characterise a given vectorised texture x ∈ RM , we first pass x through the convolutional layer and compute the output activations. The output can be understood as a non-linear filter bank, and thus its activations form a set of filtered images (so-called feature maps). For N distinct feature maps, the rectified output activations can be
described by a matrix F ∈ RN×M . To capture the stationary structure of the textures, we compute the covariances (or, more precisely, the Gramian matrix) G ∈ RN×N between the feature activations F by averaging the outer product of the point-wise feature vectors,
Gij = 1
M M∑ m=1 FimFjm. (1)
We will denote G(x) as the Gram matrix of the feature activations for the input x. To determine the relative distance between two textures x and y we compute the euclidean distance of the normalized Gram matrices,
d(x,y) = 1√∑
m,n Gmn(x)2 √∑ m,n Gmn(y)2 N∑ i,j=1 (Gij(x)−Gij(y))2 . (2)
To compare with the distance in the raw pixel values, we compute
dp(x,y) = 1√∑ m x2m √∑ m y2m N∑ i=1 (xi − yi)2 . (3)
4 TEXTURE SYNTHESIS
To generate a new texture we start from a uniform noise image (in the range [0, 1]) and iteratively optimize it to match the Gram matrix of the reference texture. More precisely, let G(x) be the Gram matrix of the reference texture. The goal is to find a synthesised image x̃ such that the squared distance between G(x) and the Gram matrix G(x̃) of the synthesized image is minimized, i.e.
x̃ = argmin y∈RM E(y), (4)
E(y) = 1∑N
i,j=1Gij(x) 2 N∑ i,j=1 ( Gij(x)−Gij(y) )2 . (5)
The gradient ∂E(y)/∂y of the reconstruction error with respect to the image can readily be computed using standard backpropagation, which we then use in conjunction with the L-BFGS-B algorithm (Jones et al., 2001–) to solve (4). We leave all parameters of the optimization algorithm at their default value except for the maximum number of iterations (2000), and add a box constraints with range [0, 1]. In addition, we scale the loss and the gradients by a factor of 107 in order to avoid early stopping of the optimization algorithm.
5 TEXTURE EVALUATION
Evaluating the quality of the synthesized textures is traditionally performed by human inspection. Optimal texture synthesis should generate samples that humans perceive as being the same texture as the reference. The high quality of the synthesized textures by (Gatys et al., 2015a) suggests that the summary statistics from multiple layers of VGG can approximate the perceptual metric of humans. Even though the VGG texture representation is not perfect, this allows us to utilize these statistics as a more objective quantification of texture quality.
For all details of the VGG-based texture model see (Gatys et al., 2015a). Here we use the standard 19-layer VGG network (Simonyan & Zisserman, 2014) with pretrained weights and average- instead of max-pooling1. We compute a Gram matrix on the output of each convolutional layer that follows a pooling layer. Let G`(.) be the Gram matrix on the activations of the `-th layer and
E`(y) = 1∑N
i,j=1G ` ij(x) 2 N∑ i,j=1 ( G`ij(x)−G`ij(y) )2 . (6)
the corresponding relative reconstruction cost. The total reconstruction cost is then defined as the average distance between the reference Gram matrices and the synthesized ones, i.e.
E(y) = 1
5 5∑ `=1 E`(y). (7)
This cost is reported on top of each synthesised texture in Figures 4. To visually evaluate samples from our single- and multi-scale model against the VGG-based model (Gatys et al., 2015a), we additionally synthesize textures from VGG by minimizing (7) using L-BFGS-B as in section 4.
6 RESULTS
In Fig. 1 we show textures synthesised from two random single- and multi-scale models, as well as eight other non-random single-layer models for three different source images (top left). For
1https://github.com/Lasagne/Recipes/blob/master/modelzoo/vgg19.py as accessed on 12.05.2016.
comparison, we also plot samples generated from the VGG model by Gatys et al. (Gatys et al., 2015a) (bottom left). There are roughly two groups of models: those with a small number of feature maps (363, top row), and those with a large number of feature maps (3267, bottom row). Only the multi-scale model employs 1024 feature maps. Within each group, we can differentiate models for which the filters are unsupervisedly trained on natural images (e.g. sparse coding filters from k-means), principally devised filter banks (e.g. 2D Fourier basis) and completely random filters (see sec. 2 for all details). All single-layer networks, except for multi-scale, feature 11× 11 filters. Remarkably, despite the small spatial size of the filters, all models capture much of the small- and mid-scale structure of the textures, in particular if the number of feature maps is large. Notably, the scale of these structures extends far beyond the receptive fields of the single units (see e.g. the pebble texture). We further observe that a larger number of feature maps generally increases the perceptual quality of the generated textures. Surprisingly, however, completely random filters perform on par or better then filters that have been trained on the statistics of natural images. This is particularly true for the multi-scale model that clearly outperforms the single-scale models on all textures. The captured structures in the multi-scale model are generally much larger and often reach the full size of the texture (see e.g. the wall).
While the above results show that for natural texture synthesis one neither needs a hierarchical deep network architecture with spatial pooling nor filters that are adapted to the statistics of natural images, we now focus on the aspects that are crucial for high quality texture synthesis. First, we evaluate whether the success of the random multi-scale network arises from the combination of filters on multiple scales or whether it is simply the increased size of its largest receptive fields (55× 55 vs. 11× 11) that leads to the improvement compared to the single-scale model. Thus, to investigate the influence of the spatial extend of the filters and the importance of combining multiple filter sizes in one model, we generate textures from multiple single-scale models, where each model has the same number of random filters as the multi-scale model (1024) but only uses filters from a single scale of the multi-scale model (Fig. 2). We find that while 3× 3 filters mainly capture the marginal distribution of the color channels, larger filters like 11 × 11 model small- to mid-scale structures (like small stones) but miss more long-range structures (larger stones are not well separated). Very large filters like 55× 55, on the other hand, are capable of modeling long-range structures but then miss much of the small- to midscale statistics (like the texture of the stone). Therefore we conclude that the combination of different scales in the multi-scale network is important for good texture synthesis since it allows to simultaneously model small-, mid- and long-range correlations of the textures. Finally we note that a further indispensable component for good texture models are the non-linearities: textures synthesised the multi-scale model without ReLU (Fig. 2, right column) are unable to capture the statistical dependencies of the texture.
The perceptual quality of the textures generated from models with only a single layer and random filters is quite remarkable and surpasses parametric methods like Portilla & Simoncelli (2000) that have been state-of-the-art two years ago (before the use of DNNs). The multi-scale model often rivals the current state of the art (Gatys et al., 2015a) as we show in Fig. 4 where we compare samples synthesized from 20 different textures for the random single- and multi-scale model, as well as VGG. The multi-scale model generates very competitive samples in particular for textures with extremely regular structures across the whole image (e.g. for the brick wall, the grids or the scales). In part, this effect can be attributed to the more robust optimization of the single-layer model that is less prone to local minima then the optimization in deeper models. This can be seen by initializing the VGG-based synthesis with textures from the single-layer model, which consistently yields superior synthesis results (see Appendix A, Fig. 5). In addition, for a few textures such as the grid structures, the VGG-based loss is paradoxically lower for samples from the multi-scale model then for the VGG-based model (which directly optimized the VGG-based loss). This suggests that the naive synthesis performed here favors images that are perceptually similar to the reference texture and thus looses variability (see sec. 7 for further discussion). Nonetheless, samples from the single-layer model still exhibit large perceptual differences, see Fig. 3. The VGG-based loss (7) appears to generally be an acceptable approximation of the perceptual differences between the reference and the synthesized texture. Only for a few textures, especially those with very regular men-made structures (e.g. the wall or the grids), the VGG-based loss fails to capture the perceptual advantage of the multi-scale synthesis.
7 DISCUSSION
We proposed a generative model of natural textures based on a single-layer convolutional neural network with completely random filters and showed that the model is able to qualitatively capture the perceptual differences between natural textures. Samples from the model often rival the current state-of-the-art (Gatys et al., 2015a) (Fig. 4, third vs fourth row), even though the latter relies on a high-performance deep neural network with features that are tuned to the statistics of natural images. Seen more broadly, this finding suggests that natural image generation does not necessarily depend on deep hierarchical representations or on the training of the feature maps. Instead, for texture synthesis, both aspects rather seem to serve as fine-tuning of the image representation.
One concern about the proposed single-layer multi-scale model is its computational inefficiency since it involves convolutions with spatially large filters (up to 55× 55). A more efficient way to achieve receptive fields of similar size would be to use a hierarchical multi-layer net. We conducted extensive experiments with various hierarchical architectures and while the synthesis is indeed significantly faster, the quality of the synthesized textures does not improve compared to a single-layer model. Thus for a minimal model of natural textures, deep hierarchical representations are not necessary but they can improve the efficiency of the texture synthesis.
Our results clearly demonstrate that Gram matrices computed from the feature maps of convolutional neural networks generically lead to useful summary statistics for texture synthesis. The Gram matrix on the feature maps transforms the representations from the convolutional neural network into a stationary feature space that captures the pairwise correlations between different features. If the number of feature maps is large, then the local structures in the image are well preserved in the projected space and the overlaps of the convolutional filtering add additional constraints. At the same time, averaging out the spatial dimensions yields sufficient flexibility to generate entirely new textures that differ from the reference on a patch by patch level, but still share much of the small- and long-range statistics.
The success of shallow convolutional networks with random filters in reproducing the structure of the reference texture is remarkable and indicates that they can be useful for parametric texture synthesis. Besides reproducing the stationary correlation structure of the reference image ("perceptual similarity") another desideratum of a texture synthesis is to exhibit a large variety between different samples generated from the same given image ("variability"). Hence, synthesis algorithms need to balance perceptual similarity and variability. This balance is determined by a complex interplay between the choice of summary statistics and the optimization algorithm used. For example the stopping criterion of the optimization algorithm can be adjusted to trade perceptual similarity for larger variability.
Finding the right balance between perceptual similarity and variability is challenging because we are currently lacking robust measures of these quantities. In this work we introduced VGG-loss as a measure of perceptual similarity, and, even though, it works much better than other common measures such as Structural Similarity Index (SSIM, Wang et al., 2004, see Appendix A, Figure 6) or Euclidean distance in the pixel space (not shown), it is still not perfect (Figure 4). Measuring variability is probably even more difficult: in principle it requires measuring the entropy of generated samples, which is intractable in a high-dimensional space. A different approach could be based on a psychophysical assessment of generated samples. For example, we could use an inpainting task (illustrated in Appendix A, Figure 7) to make human observers decide between actual texture patches and inpaited ones. Performance close to a chance-level would indicate that the texture model produces variable enough samples to capture the diversity of actual patches. The further exploration of variability measures lies, however, beyond the scope of this work.
In this paper we focused on maximizing perceptual similarity only, and it is worth pointing out that additional efforts will be necessary to find an optimal trade-off between perceptual similarity and variability. For the synthesis of textures from the random models considered here, the trade-off leans more towards perceptual similarity in comparison to Gatys et al. (2015a)(due to the simpler optimization) which also explains the superior performance on some samples. In fact, we found some anecdotal evidence (not shown) in deeper multi-layer random CNNs where the reference texture was exactly reconstructed during the synthesis. From a theoretical point of view this is likely a finite size effect which does not necessarily constitute a failure of the chosen summary statistics: for finite size images it is well possible that only the reference image can exactly reproduce all the summary
statistics. Therefore, in practice, the Gram matrices are not treated as hard constraints but as soft constraints only. More generally, we do not expect a perceptual distance metric to assign exactly zero to a random pair of patches from the same texture. Instead, we expect it to assign small values for pairs from the same texture, and large values for patches from different textures. Therefore, the selection of constraints is not sufficient to characterize a texture synthesis model but only determines the exact minima of the objective function (which are sought for by the synthesis). If we additionally consider images with small but non-zero distance to the reference statistics, then the set of equivalent textures increases substantially, and the precise composition of this set becomes critically dependent on the perceptual distance metric.
Mathematically, parametric texture synthesis models are described as ergodic random fields that have maximum entropy subject to certain constraints Zhu et al. (1997); Bruna & Mallat (2013); Zhu et al. (2000) (MaxEnt framework). Practical texture synthesis algorithms, however, always deal with finite size images. As discussed above, two finite-size patches from the same ergodic random field will almost never feature the exact same summary statistics. This additional uncertainty in estimating the constraints on finite length processes is not thoroughly accounted for by the MaxEnt framework (see discussion on its “ad hockeries” by Jaynes (Jaynes (1982))). Thus, a critical difference of practical implementations of texture synthesis algorithms from the conceptual MaxEnt texture modeling framework is that they genuinely allow a small mismatch in the constraints. Accordingly, specifying the summary statistics is not sufficient but a comprehensive definition of a texture synthesis model should specify:
1. A metric d(x,y) that determines the distance between any two arbitrary textures x,y.
2. A bipartition Px of the image space that determines which images are considered perceptually equivalent to a reference texture x. A simple example for such a partition is the -environment U (y) := {y : d(y,x) < } and its complement.
This definition is relevant for both under- as well as over-constrained models, but its importance becomes particularly obvious for the latter. According to the Minimax entropy principle for texture modeling suggested by Zhu et al Zhu et al. (1997), as many constraints as possible should be used to reduce the (Kullback-Leibler) divergence between the true texture model and its estimate. However, for finite spatial size, the synthetic samples become exactly equivalent to the reference texture (up to shifts) in the limit of sufficiently many independent constraints. In contrast, if we explicitly allow for a small mismatch between the summary statistics of the reference image and the synthesized textures, then the set of possible textures does not constitute a low-dimensional manifold but rather a small volume within the pixel space. Alternatively, instead of introducing an -environment it is also possible to extent the MaxEnt framework to allow for variability in the summary statistics (Joan Bruna, personal communication). It will be interesting to compare in the future to what extent the difference between the two approaches can lead to differences in the perceptual appearance of the textures.
Taken together we have shown that simple single-layer CNNs with random filters can serve as the basis for excellent texture synthesis models that outperform previous hand-crafted synthesis models and sometimes even rivals the current state-of-the-art. This finding repeals previous observations that suggested a critical role for the multi-layer representations in trained deep networks for natural texture generation. On the other hand, it is not enough to just use sufficiently many constraints as one would predict from the MaxEnt framework. Instead, for the design of good texture synthesis algorithms it will be crucial to find distance measures for which the -environment around the reference texture leads to perceptually satisfying results. In this way, building better texture synthesis models is inherently related to better quantitative models of human perception.
A APPENDIX

The framework of Gatys et al. demonstrated that correlation statistics (empirical Gram matrices) of deep feature responses provide an excellent characterisation of visual textures. This paper investigates in detail which kind of deep or shallow networks may work well in this framework. One of the main findings is that that very shallow nets, consisting of a single filter bank with random weights, work surprisingly well, and for simple and regular textures may produce results which are visually superior to complex data-adapted filters such as the ones in networks like VGG-19. More broadly, the paper contains an interesting and informative discussion on the strength and limitations on such methods for texture synthesis.
Figure 4 shows that the optimisation of images with respect to shallow filter banks may result in texture images that have a lower VGG-19 loss than optimising the VGG-19 objective directly. This is imputed to the difficulty of optimising the highly non-linear VGG-19 cost function, which is a reasonable explanation. In the new supplementary material, the authors show that better optimisation results can be obtained by initialising the VGG-19-based optimisation with the shallow network optimisation results, which is a useful complement to the original experiments.
The main limitation of the paper is that it does not systematically compare different methods against a quantifiable objective. It is trivial to define image statistics that would allow to simply generate an exact copy of any reference texture, hence with very good visual quality. Such trivial statistics would also be very shallow. The aim is instead to capture a texture distribution, and measuring how well a method meets this challenge remains an open problem. Hence, while the empirical results seem to confirm the intuition that simple statistics are good enough for texture synthesis both in terms of quality and diversity (when compared to more complex statistics), it is difficult to conclusively confirm that this is the case.
The authors indicate that diversity could be measured in terms of entropy. This is reasonable, but, as they acknowledge in their answers to questions, difficult to do in practice. Furthermore, this would still not account for the other aspect of the problem, namely visual quality. They also suggest to perform a psychophysical assessment, which may be the only practical way of addressing this problem, but deem that to be material for future work.
Overall, since evaluation of image generation is such an hard problem, I think the paper still has sufficient strengths to warrant publication in ICLR. Still, some form of psychophysical assessment would be useful to confirm the intuitions that, at present, can only be obtained by inspecting the figure sin the paper and in the supplementary material.


SHIFT AGGREGATE EXTRACT NETWORKS
The Shift Aggregate Extract Network (SAEN) is an architecture for learning representations on social network data. SAEN decomposes input graphs into hierarchies made of multiple strata of objects. Vector representations of each object are learnt by applying shift, aggregate and extract operations on the vector representations of its parts. We propose an algorithm for domain compression which takes advantage of symmetries in hierarchical decompositions to reduce the memory usage and obtain significant speedups. Our method is empirically evaluated on real world social network datasets, outperforming the current state of the art.
1 INTRODUCTION
Many different problems in various fields of science require the classification of structured data, i.e. collections of objects bond together by some kind of relation. A natural way to represent such structures is through graphs, which are able to encode both the individual objects composing the collection (as vertices) and the relationships between them (as edges). A number of approaches to the graph classification problem has been studied in graph kernel and neural network literature.
Graph kernels decompose input graphs in substructures such as shortest paths (Borgwardt & Kriegel, 2005), graphlets (Shervashidze et al., 2009) or neighborhood subgraph pairs (Costa & De Grave, 2010). The similarity between two graphs is then computed by comparing the respective sets of parts. Methods based on recursive neural networks unfold a neural network over input graphs and learn vector representations of their nodes employing backpropagation though structure (Goller & Kuchler, 1996). Recursive neural networks have been successfully applied to domains such as natural language (Socher et al., 2011) and biology (Vullo & Frasconi, 2004; Baldi & Pollastri, 2003). An advantage of recursive neural networks over graph kernels, is that the vector representations of the input graphs are learnt rather than handcrafted.
Learning on social network data can be considerably hard due to their peculiar structure: as opposed to chemical compounds and parse trees, the structure of social network graphs is highly irregular. Indeed in social networks it is common to have nodes in the same graph whose degree differs by orders of magnitude. This poses a significant challenge for the substructure matching approach used by some graph kernels as the variability in connectivity generates a large number of unique patterns leading to diagonally dominant kernel matrices.
We propose Shift Aggregate Extract Networks (SAEN), a neural network architecture for learning representations of input graphs. SAEN decomposes input graphs intoH-hierarchies made of multiple strata of objects. Objects in each stratum are connected by “part-of” relations to the objects to the stratum above.
In case we wish to classify graphs we can use an H-hierarchical decomposition in which the top stratum contains the graphG that we want to classify, while the intermediate strata contain subgraphs of G, subgraphs of subgraphs of G and so on, until we reach the bottom stratum which contains the vertices v of G.
Unlike R-convolution relations in kernel methods (which decompose objects into the set of their parts), H-hierarchical decompositions are deep as they can represent the parts of the parts of an object.
Recursive neural networks associate to the vertices of the input graphs vector representations imposing that they have identical dimensions. Moreover, the propagation follows the edge connectivity and weights are shared over the whole input graph. If we consider that vector representations of nodes (whose number of parents can differ by orders of magnitude) must share the same weights, learning on social network data with recursive neural networks might be nontrivial.
SAEN compensates the limitations of recursive neural networks by adding the following degrees of flexibility:
1. the SAEN computation schema unfolds a neural network over H-decompositions instead of the input graph, 2. SAEN imposes weight sharing and fixed size of the learnt vector representations on a per stratum basis instead of globally.
Indeed SAEN allows to use vector representations of different sizes for different strata of objects (e.g. graphs, subgraphs, subgraphs of subgraphs, edges, vertices etc.) The SAEN schema computes the vector representation of each object by applying shift, aggregate and extract operations on the vector representations of its parts.
Another contribution of this paper is the introduction of a domain compression algorithm, that we use in our experiments to reduce memory usage and runtime. Domain compression collapses objects in the same stratum of an H-hierarchical decomposition into a compressed one whenever these objects are indistinguishable for the SAEN computation schema. In particular objects made of the same sets of parts are indistinguishable. In order obtain a lossless compression an H-hierarchical decomposition we store counts on symmetries adopting some mathematical results from lifted linear programming (Mladenov et al., 2012). The domain compression algorithm is also reminiscent of the work of Sperduti & Starita (1997) in which common substructures of recursive neural networks are collapsed in order to reduce the computational cost.
2 SHIFT-AGGREGATE-EXTRACT NEURAL NETWORKS
We propose a neural network architecture that takes as input an undirected attributed graph G = (V,E,X) where V is the vertex set, E ⊆ V × V is the edge set, and X = {xv ∈ Rp}v∈V is a set of p-dimensional vertex attributes. When vertices do not have associated attributes (for example this happens in some of the social network datasets of § 4.1), we can set xv to some vertex invariant such as node centrality or betweenness.
2.1 H-HIERARCHICAL DECOMPOSITIONS
Most graph kernels decompose graphs into parts by using an R-convolution relation (Haussler, 1999). We extend this approach by decomposing graphs into a hierarchy of π-parametrized “part of” relations. Formally, anH-hierarchical decomposition is a pair ({Sl}Ll=0, {Rl,π}Ll=1) where: • {Sl}Ll=0 are disjoint sets of objects Sl called strata, or levels of the hierarchy. The bottom stratum S0 contains non-decomposable objects (e.g. individual vertices), while the other strata Sl, l = 1, . . . , L contain composite objects, oi ∈ Sl, whose parts oj ∈ Sl−1 belong to the preceding stratum, Sl−1.
• {Rl,π}Ll=1 is a set of l, π-parametrized Rl,π-convolution relations. A pair (oi, oj) ∈ Sl × Sl−1 belongs toRl,π iff “oj is part of oi with membership type π”. For notational convenience, the parts of oi are denoted asR−1l,π(oi) = {oj |(oj , oi) ∈ Rl,π}. The membership type π is used to represent the roles of the parts of an object. For example, we could decompose a graph as a multiset of π-neighborhood subgraphs 1 in which π is the radius of the neighborhoods (see Figure 1 on the left). Another possible use of the π membership type is to
1The r-neighborhood subgraph (or ego graph) of a vertex v in a graph G is the induced subgraph of G consisting of all vertices whose shortest-path distance from v is at most r.
distinguish the root from the other vertices in a rooted neighborhood subgraph (see Figure 1 on the right).
An H-hierarchical decomposition is a multilevel generalization of R-convolution relations, and it reduces to anR-convolution relation for L = 1.
2.2 SHIFT AGGREGATE EXTRACT SCHEMA FOR LEARNING REPRESENTATIONS
We propose Shift Aggregate Extract Network (SAEN) to learn vector representations for all the objects of all the strata {Sl}Ll=0 in an H-hierarchical decomposition. SAEN unfolds a neural network architecture over anH-hierarchical decomposition by using the Shift Aggregate Extract (SAE) schema.
According to the SAE schema the vector representation of each object in theH-hierarchical decomposition is either computed by applying a neural network on the vertex attributes (for the objects in bottom stratum) or defined in terms of the vector representations of its parts (for the other objects).
More formally, the SAE schema associates a dl-dimensional representation hi ∈ Rdl to each object oi ∈ Sl of theH-hierarchical decomposition according to the following formula:
hi =    f0(xvi ; Θ0) if oi ∈ S0 fl ( ∑ π∈Πl ∑ oj∈R−1l,π(oi) (zπ ⊗ hj)︸ ︷︷ ︸ Shift︸ ︷︷ ︸ Aggregate ; Θl )
︸ ︷︷ ︸ Extract
otherwise (1)
where fl(·; Θl), l = 0, . . . , L are multilayer neural networks with parameters Θl. With respect to the base case (first branch of Eq. 1) we have that each object oi in the bottom stratum S0 is in one-to-one correspondence with the vertices vi ∈ V of the graph that we are decomposing. Indeed the vector representations hi are computed by evaluating f0(·; Θ0) in correspondence of the vertex attributes xvi ∈ X . The recursion step (second branch of Eq. 1) follows the Shift Aggregate Extract (SAE) schema:
• Shift: each part representation hj ∈ Rdl−1 is remapped into a space R|Πldl−1| made of |Πl| slots, where each slot has dimension dl−1. This transformation shifts part representations hj by using the Kronecker product ⊗ between an indicator vector zπ ∈ R|Πl| and the vector representation hj of part oj ∈ Sl−1. The indicator vector zπ ∈ R|Πl| defined as zi = { 1 if i=π 0 otherwise. and it is used to
make sure that vector representations hj of object parts will fall in the same slot if and only if they have the same membership type π. • Aggregate: the shifted representations (zπ ⊗ hj) of the parts oj are then aggregated with a sum. • Extract: the aggregated representation is compressed to a dl-dimensional space by a Θlparametrized nonlinear map fl(·,Θl) : R|Πldl−1| → Rdl implemented with a multilayer neural network.
The shift and aggregate steps, that we have seen so far, are identical to those used in kernel design when computing the explicit feature of a kernel k(x, z) derived from a sum ∑ π∈Π kπ(x, z) of base kernels kπ(x, z), π ∈ Π. In principle, it would be indeed possible to turn SAEN into a kernel method by removing the extraction step E from the SAE schema. However, such an approach would increase the dimensionality of the feature space by a multiplicative factor |Πl| for each level l of the Hhierarchical decomposition, thus leading to an exponential number of features. When using SAEN, the feature space growth is prevented by exploiting a distributed representation (via a multilayered neural network) during the E step of the SAE schema. As a result, SAEN can easily cope with Hhierarchical decompositions consisting of multiple strata.
2.3 EXPLOITING SYMMETRIES FOR DOMAIN COMPRESSION
In this section we propose a technique, called domain compression, which allows to save memory and speedup the SAEN computation. Domain compression exploits symmetries inH-hierarchical decompositions by collapsing equivalent objects in each stratum. The greater the number of collapsed objects the highest the compression ratio.
Two objects a, b in a stratum Sl are collapsable a ∼ b if they share the same representation (i.e. ha = hb) for all the possible values of Θl. A compressed stratum S comp l is the quotient set Sl/∼ of stratum Sl w.r.t. the collapsibility relation ∼. We assume that the attributes of the elements in the bottom stratum S0 are categorical, so that the same vector representation can be shared by multiple elements with non-zero probability. 2 While objects in the bottom stratum S0 are collapsable when their attributes are identical, for all the other strata Sl, l = 1, . . . , L, objects are collapsable if they are made by the same sets of parts for all the membership types π.
In Figure 2 we provide a pictorial representation of the domain compression of an H-hierarchical decomposition (EGNN, described in § 4.2). On the left we show the H-hierarchical decomposition of a graph taken from the IMDB-BINARY dataset (see § 4.1) together with its compressed version on the right.
2.3.1 DOMAIN COMPRESSION ALGORITHM
In order to compress H-hierarchical decompositions we adapt the lifted linear programming technique proposed by Mladenov et al. (2012) to the SAEN architecture. If a matrix M ∈ Rn×p has
2 Vectors of real valued attributes could be discretized using clustering techniques. However, we leave discretization in SAEN to future works.
m ≤ n distinct rows it can be decomposed as the product DM comp where M comp is a compressed version of M in which the distinct rows of M appear exactly once. The Boolean decompression matrix, D, encodes the collapsibility relation among the rows of M so that Dij = 1 iff the ith row of M falls in the equivalence class j of ∼. A pseudo-inverse C of D can be computed by dividing the rows of D> by their sum (where D> is the transpose of D).
Example 1 If we look at matrix M in Eq. 2 we notice that row 1 and 4 share the encoding [0, 0, 0], rows 3 and 5 share the encoding [1, 1, 0] while the encoding [1, 0, 1] appears only once at row 2. Matrix M comp is the compressed version of M .
M =
0 0 0 1 0 1 1 1 0 0 0 0 1 1 0
  M comp = [ 0 0 0 1 0 1 1 1 0 ] D =
1 0 0 0 1 0 0 0 1 1 0 0 0 0 1
  C = [ 1/2 0 0 1/2 0 0 1 0 0 0 0 0 1/2 0 1/2 ] (2)
Matrix M can be expressed as the matrix product between the decompression matrix D and the compressed version of M comp (i.e. M = DM comp), while the matrix multiplication between the compression matrix C and the M leads to the compressed matrix M comp (i.e.M comp = CM ).
To apply domain compression we rewrite Eq. 1 in matrix form as follows:
Hl =
f0(X; Θ0)︸ ︷︷ ︸ |S0|×d0
if l = 0
fl   [ Rl,1, . . . , Rl,π, . . . , Rl,|Πl| ] ︸ ︷︷ ︸
|Sl|×|Πl||Sl−1|
  Hl−1 . . . 0 ... . . .
... 0 . . . Hl−1

︸ ︷︷ ︸ |Πl||Sl−1|×|Πl|dl−1
; Θl
︸ ︷︷ ︸ |Sl|×dl
otherwise (3)
where:
• Hl ∈ R|Sl|×dl is the matrix that represents the dl-dimensional encodings of the objects in Sl. The rows of Hl are the vector representations hi in Eq. 1, while the rows of Hl−1 are the vector representations hj in Eq. 1;
• X ∈ R|S0|×p is the matrix that represents the p-dimensional encodings of the vertex attributes in V (i.e. the rows of X are the xvi of Eq. 1);
• fl(·; Θl) is unchanged w.r.t. Eq. 1 and is applied to its input matrices row-wise; • Rl,π ∈ R|Sl|×|Sl−1| ∀π ∈ Πl are the matrix representations of the Rl,π-convolution relations of Eq. 1 whose elements are (Rl,π)ij = 1 if (oj , oi) ∈ Rl,π and 0 otherwise. Domain compression on Eq. 3 is performed by the DOMAIN-COMPRESSION procedure (see Algorithm 3) that takes as input the attribute matrix X and the part-of matrices Rl,π and returns their compressed versions Xcomp and the Rcompl,π respectively. The algorithm starts by invoking (line 1) the procedure COMPUTE-CD on X to obtain the compression and decompression matrices C0 and D0 respectively. The compression matrix C0 is used to compress X (line 2) then we start iterating over the levels l = 0, . . . , L of the H-hierarchical decomposition (line 4) and compress the Rl,π matrices. The compression of the Rl,π matrices is done by right-multiplying them by the decompression matrixDl−1 of the previous level l−1 (line 5). In this way we collapse the parts of relation Rl,π (i.e. the columns of Rl,π) as these were identified in stratum Sl−1 as identical objects (i.e. those objects corresponding to the rows of X or Rl−1,π collapsed during the previous step). The result is a list Rcol comp = [Rl,πDl−1, ∀π = 1, . . . , |Πl|] of column compressed Rl,π−matrices. We proceed collapsing equivalent objects in stratum Sl, i.e. those made of identical sets of parts: we find symmetries in Rcol comp by invoking COMPUTE-CD (line 6) and obtain a new pair Cl, Dl of compression, and decompression matrices respectively. Finally the compression matrix Cl is applied to the column-compressed matrices inRcol comp in order to obtain the Πl compressed matrices
DOMAIN-COMPRESSION(X,R) 1 C0, D0 = COMPUTE-CD(X) 2 Xcomp = C0X // Compress the X matrix. 3 Rcomp = {} // Initialize an empty container for compressed matrices. 4 for l = 1 to L 5 Rcol comp = [Rl,πDl−1, ∀π = 1, . . . , |Πl|] // column compression 6 Cl, Dl = COMPUTE-CD(Rcol comp) 7 for π = 1 to |Πl| 8 Rcompl,π = ClR col comp π // row compression 9 return Xcomp, Rcomp
Figure 3: DOMAIN-COMPRESSION
of stratum Sl (line 8). Algorithm 3 allows us to compute the domain compressed version of Eq. 3 which can be obtained by replacing: X with Xcomp = C0X , Rl,π with R comp l,π = ClRl,πDl−1 and Hl with H comp l . Willing to recover the original encodings Hl we just need to employ the decompression matrix Dl on the compressed encodings H comp l , indeed Hl = DlH comp l .
As we can see by substituting Sl with S comp l , the more are the symmetries (i.e. when |S comp l | |Sl|) the greater the domain compression will be.
3 RELATED WORKS
When learning with graph inputs two fundamental design aspects that must be taken into account are: the choice of the pattern generator and the choice of the matching operator. The former decomposes the graph input in substructures while the latter allows to compare the substructures.
Among the patterns considered from the graph kernel literature we have paths, shortest paths, walks (Kashima et al., 2003), subtrees (Ramon & Gärtner, 2003; Shervashidze et al., 2011) and neighborhood subgraphs (Costa & De Grave, 2010). The similarity between graphs G and G′ is computed by counting the number of matches between their common the substructures (i.e. a kernel on the sets of the substructures). The match between two substructures can be defined by using graph isomorphism or some other weaker graph invariant.
When the number of substructures to enumerate is infinite or exponential with the size of the graph (perhaps this is the case for random walks and shortest paths respectively) the kernel between the two graphs is computed without generating an explicit feature map. Learning with an implicit feature map is not scalable as it has a space complexity quadratic in the number of training examples (because we need to store in memory the gram matrix).
Other graph kernels such as the Weisfeiler-Lehman Subtree Kernel (WLST) (Shervashidze et al., 2011) and the Neighborhood Subgraph Pairwise Distance Kernel (NSPDK) (Costa & De Grave, 2010) deliberately choose a pattern generator that scales polynomially and produces an explicit feature map. However the vector representations produced by WLST and NSPDK are handcrafted and not learned.
A recent work by Yanardag & Vishwanathan (2015) proposes to uses pattern generators such as graphlets, shortest paths and WLST subtrees to transform input graphs into documents. The generated substructures are then treated as words and embedded in the Euclidean space with a CBOW or a Skip-gram model. The deep upgrade of existing graph kernels is performed by reweighing the counts of the substructures by the square root of their word-vector self similarity.
Another recent work by Niepert et al. (2016) upgrades the convolutional neural networks CNNs for images to graphs. While the receptive field of a CNN is usually a square window (Niepert et al., 2016) employ neighborhood subgraphs as receptive fields. As nodes in graphs do not have a specific temporal or spatial order, (Niepert et al., 2016) employ vertex invariants to impose an order on the nodes of the subgraphs/receptive fields.
4 EXPERIMENTAL EVALUATION
We answer to the following experimental questions: Q1 How does SAEN compare to the state of the art? Q2 Can SAEN exploit symmetries in social networks to reduce the memory usage and the runtime?
4.1 DATASETS
In order to answer the experimental questions we tested our method on six publicly available datasets first proposed by Yanardag & Vishwanathan (2015).
• COLLAB is a dataset where each graph represent the ego-network of a researcher, and the task is to determine the field of study of the researcher between High Energy Physics, Condensed Matter Physics and Astro Physics. • IMDB-BINARY, IMDB-MULTI are datasets derived from IMDB where in each graph the vertices represent actors/actresses and the edges connect people which have performed in the same movie. Collaboration graphs are generated from movies belonging to genres Action and Romance for IMDB-BINARYand Comedy, Romance and Sci-Fi for IMDB-MULTI, and for each actor/actress in those genres an ego-graph is extracted. The task is to identify the genre from which the ego-graph has been generated. • REDDIT-BINARY, REDDIT-MULTI5K, REDDIT-MULTI12K are datasets where each graph is derived from a discussion thread from Reddit. In those datasets each vertex represent a distinct user and two users are connected by an edge if one of them has responded to a post of the other in that discussion. The task in REDDIT-BINARYis to discriminate between threads originating from a discussion-based subreddit (TrollXChromosomes, atheism) or from a question/answers-based subreddit (IAmA, AskReddit). The task in REDDIT-MULTI5Kand REDDIT-MULTI12Kis a multiclass classification problem where each graph is labeled with the subreddit where it has originated (worldnews, videos, AdviceAnimals, aww, mildlyinteresting for REDDIT-MULTI5Kand AskReddit, AdviceAnimals, atheism, aww, IAmA, mildlyinteresting, Showerthoughts, videos, todayilearned, worldnews, TrollXChromosomes for REDDIT-MULTI12K).
4.2 EXPERIMENTS
In our experiments we chose an H-hierarchical decomposition called Ego Graph Neural Network (EGNN), that mimics the graph kernel NSPDK with the distance parameter set to 0.
Before applying EGNN we turn unattributed graphs (V,E) into attributed graphs (V,E,X) by annotating their vertices v ∈ V with attributes xv ∈ X . We label vertices v of G with their degree and encode this information into the attributes xv by employing the 1-hot encoding.
EGNN decomposes attributed graphs G = (V,E,X) into a 3 level H-hierarchical decomposition with the following strata (see Figure 1 for a pictorial representation of EGNN): • stratum S0 contains objects ov that are in one-to-one correspondence with the vertices v ∈ V . • stratum S1 contains vroot-rooted r-neighborhood subgraphs (i.e. ego graphs) e = (vroot, Ve, Ee) of radius r = 0, 1, . . . , R and has part-of alphabet Π1 = {ROOT, ELEM}. Objects ov ∈ S0 are “ELEM-part-of” ego graph e if v ∈ Ve \ {vroot}, while the are “ROOT-part-of” ego graph e if v = vroot. • stratum S2 contains the graph G that we want to classify and has part-of alphabet Π2 = {0, 1} which correspond to the radius of the ego graphs e ∈ S1 of which G is made of.
E1 We experimented with SAEN applying the EGNN H-decomposition on all the datasets. For each dataset, we manually chose the parameters of SAEN, i.e. the number of hidden layers for each stratum, the size of each layer and the maximum radius R. We used the Leaky ReLU (Maas et al.) activation function on all the units. We report the chosen parameters in Table A1 of the appendix. In all our experiments we trained the neural networks by using the Adam algorithm to minimize a cross entropy loss.
The classification accuracy of SAEN was measured with 10-times 10-fold cross-validation. We manually chose the number of layers and units for each level of the part-of decomposition; the number of epochs was chosen manually for each dataset and we kept the same value for all the 100 runs of the 10-times 10-fold cross-validation.
The mean accuracies and their standard deviations obtained by our method are reported in Table 4, where we compare these results with those obtained by Yanardag & Vishwanathan (2015) and by Niepert et al. (2016).
Although our method was conceived for social network data, it can also handle other types of graphs. For the sake of completeness in Table 5 we report the mean accuracies obtained with SAEN on the molecule and protein datasets studied in previous works (e.g. Niepert et al. (2016)).
E2 In Table 1 we show the file sizes of the preprocessed datasets before and after the compression together with the data compression ratio. 3 We also estimate the benefit of the relational compression from a computational time point of view and report the measurement of the runtime for 1 run with and without compression together with the speedup factor.
For the purpose of this experiment, all tests were run on a computer with two 8-cores Intel Xeon E5-2665 processors and 94 GB RAM. Uncompressed datasets which exhausted our server’s memory during the test are marked as “OOM” (out of memory) in the table, while those who exceeded the time limit of 100 times the time needed for the uncompressed version are marked as “TO” (timeout).
4.3 DISCUSSION
A1 As shown in Table 4, EGNN performs consistently better than the other two methods on all the social network datasets. This confirms that the chosenH-hierarchical decomposition is effective on this kind of problems. Also the results for molecule and protein datasets (see Table 5) are in line with the current state of the art. A2 The compression algorithm has proven to be effective in improving the computational cost of our method. Most of the datasets improved their runtimes by a factor of at least 4 while maintaining the
3The size of the uncompressed files are shown for the sole purpose of computing the data compression ratio. Indeed the last version of our code compresses the files on the fly.
same expressive power. Moreover, experiments on REDDIT-MULTI5K and REDDIT-MULTI12K have only been possible thanks to the size reduction operated by the algorithm as the script exhausted the memory while executing the training step on the uncompressed files.
5 CONCLUSIONS
We proposed SAEN, a novel architecture for learning vector representations of H-decompositions of input graphs. We applied SAEN for graph classification on 6 real world social network datasets, outperforming the current state of the art on 4 of them and obtaining state-of-the-art classification accuracy on the others. Another important contribution of this paper is the domain compression algorithm which greatly reduces memory usage and allowed us to speedup the training time of a factor of at least 4.
APPENDIX: SHIFT AGGREGATE EXTRACT NETWORKS
Francesco Orsini 12 , Daniele Baracchi 2 and Paolo Frasconi 2
1 Department of Computer Science
2 Department of Information Engineering
Katholieke Universiteit Leuven Università degli Studi di Firenze Celestijnenlaan 200A Via di Santa Marta 3
3001 Heverlee, Belgium I-50139 Firenze, Italy francesco.orsini@kuleuven.be daniele.baracchi@unifi.it
paolo.frasconi@unifi.it
A PARAMETERS USED IN THE EXPERIMENTS WITH EGNN
In Table A1 we report for each dataset: the radiuses r of the neighborhood subgraphs used in the EGNN decomposition and the number of units in the hidden layers for each stratum.
Figure A1: Parameters for the neural networks used in the experiments. DATASET RADIUSES HIDDEN UNITS
r S0 S1 S2 COLLAB 0, 1 15− 5 5− 2 5− 3 IMDB-BINARY 0, 1, 2 2 5− 2 5− 3− 1 IMDB-MULTI 0, 1, 2 2 5− 2 5− 3 REDDIT-BINARY 0, 1 10− 5 5− 2 5− 3− 1 REDDIT-MULTI5K 0, 1 10 10 6− 5 REDDIT-MULTI12K 0, 1 10 10 20− 11 MUTAG 0, 1, 2, 3 10 5− 5 5− 5− 1 PTC 0, 1 15 15 15− 1 NCI1 0, 1, 2, 3 15 15 15− 10− 1 PROTEINS 0, 1, 2, 3 3− 2 6− 5− 4 6− 3− 1 D&D 0, 1, 2, 3 10 5− 2 5− 3− 1

The paper contributes to recent work investigating how neural networks can be used on graph-structured data. As far as I can tell, the proposed approach is the following:
  1. Construct a hierarchical set of "objects" within the graph. Each object consists of multiple "parts" from the set of objects in the level below. There are potentially different ways a part can be part of an object (the different \pi labels), which I would maybe call "membership types". In the experiments, the objects at the bottom level are vertices. At the next level they are radius 0 (just a vertex?) and radius 1 neighborhoods around each vertex, and the membership types here are either "root", or "element" (depending on whether a vertex is the center of the neighborhood or a neighbor). At the top level there is one object consisting of all of these neighborhoods, with membership types of "radius 0 neighborhood" (isn't this still just a vertex?) or "radius 1 neighborhood".
  2. Every object has a representation. Each vertex's representation is a one-hot encoding of its degree. To construct an object's representation at the next level, the following scheme is employed:
    a. For each object, sum the representation of all of its parts having the same membership type.
    b. Concatenate the sums obtained from different membership types.
    c. Pass this vector through a multi-layer neural net.
I've provided this summary mainly because the description in the paper itself is somewhat hard to follow, and relevant details are scattered throughout the text, so I'd like to verify that my understanding is correct.
Some additional questions I have that weren't clear from the text: how many layers and hidden units were used? What are the dimensionalities of the representations used at each layer? How is final classification performed? What is the motivation for the chosen "ego-graph" representation?
The proposed approach is interesting and novel, the compression technique appears effective, and the results seem compelling. However, the clarity and structure of the writing is quite poor. It took me a while to figure out what was going on---the initial description is provided without any illustrative examples, and it required jumping around the paper to figure for example how the \pi labels are actually used. Important details around network architecture aren't provided, and very little in the way of motivation is given for many of the choices made. Were other choices of decomposition/object-part structures investigated, given the generality of the shift-aggregate-extract formulation? What motivated the choice of "ego-graphs"? Why one-hot degrees for the initial attributes?
Overall, I think the paper contains a useful contribution on a technical level, but the presentation needs to be significantly cleaned up before I can recommend acceptance.