PALEO: A PERFORMANCE MODEL FOR DEEP NEURAL NETWORKS
Although various scalable deep learning software packages have been proposed, it remains unclear how to best leverage parallel and distributed computing infrastructure to accelerate their training and deployment. Moreover, the effectiveness of existing parallel and distributed systems varies widely based on the neural network architecture and dataset under consideration. In order to efficiently explore the space of scalable deep learning systems and quickly diagnose their effectiveness for a given problem instance, we introduce an analytical performance model called PALEO. Our key observation is that a neural network architecture carries with it a declarative specification of the computational requirements associated with its training and evaluation. By extracting these requirements from a given architecture and mapping them to a specific point within the design space of software, hardware and communication strategies, PALEO can efficiently and accurately model the expected scalability and performance of a putative deep learning system. We show that PALEO is robust to the choice of network architecture, hardware, software, communication schemes, and parallelization strategies. We further demonstrate its ability to accurately model various recently published scalability results for CNNs such as NiN, Inception and AlexNet.
1 INTRODUCTION
Deep learning has been successfully applied in many areas including natural language processing and computer vision. The scale of modern datasets and the millions to billions of parameters in these deep networks pose new challenges when designing computational systems that leverage parallel and distributed computing. Indeed, several important open questions remain:
• How fast can we train or evaluate a model on a user’s given hardware? • For a given architecture, how can a user best leverage parallel and distributed computation? • How can we design a new neural network architecture that can be trained and evaluated efficiently
under common hardware setups?
In response to these fundamental questions, various software packages and systemshave been painstakingly developed, e.g. DistBelief (Dean et al., 2012), TensorFlow (Abadi et al., 2015), MXNet (Chen et al., 2015), SparkNet (Moritz et al., 2015), FireCaffe (Iandola et al., 2016). Moreover, expensive benchmarking efforts, e.g., Chintala et al. (2016), have performed brute-force profiling on some of these deep learning systems on a handful network architectures.
In this work we aim to tackle these questions by taking an analytical approach to model the performance of arbitrary learning systems. Our work hinges on the observation that a neural network architecture is a declarative specification of the forward and backward propagation steps required for training and deploying the network. However, given this specification, there is a rich design space of algorithms, hardware choices, and communications strategies to most efficiently execute these specifications. We build a novel performance model called PALEO1 that maps this declarative specification to arbitrary points in this design space to estimate the execution time of training and
1Open-sourced at https://github.com/TalwalkarLab/paleo.
deploying deep neural networks.2 PALEO applies broadly to a wide variety of neural network architectures and for arbitrary learning systems within this design space, and thus can serve as a valuable tool for practitioners and developers to answer the questions mentioned above.
2 BACKGROUND AND RELATED WORK
Training deep neural networks can be very time and resource consuming, and it is not uncommon for the training of a model to take days across tens or hundreds of machines. Several high-level strategies have been proposed to accelerate this process, and these strategies collectively define the design space considered by PALEO.
Hardware acceleration approaches are designed to accelerate the computation of the forward and backward passes and often make use of specialized hardware, such as GPUs (Coates et al., 2013), or more recently custom hardware designed specifically for deep learning (Jouppi, 2016). PALEO accepts constants associated with hardware as input (e.g., peak FLOPS, network bandwidth) and automatically adapts to changes in this input.
Software acceleration via specialized libraries, e.g., cuda-convnet (Krizhevsky, 2014a) and cuDNN (Chetlur et al., 2014), and highly-optimized algorithms for commonly used primitives, e.g., Chetlur et al. (2014) and Lavin (2016), can also be used to accelerate deep model training. PALEO dynamically picks among the best available implementation for each layer at execution time.
Parallelization is a natural approach to consider, and can involve training a neural network with many computational devices (e.g. CPUs, GPUs) on a single machine, or across a network. There are two major parallelization strategies when it comes to training deep neural network models at scale: data parallelism and model parallelism. In classical data parallel systems, each worker stores an identical copy of the model and computes gradients only on a shard of the training examples, and these gradients are aggregated to update the model. In contrast, model parallel systems shard the model itself across the workers, while the training data may be stored on each worker or sharded across the workers. PALEO models both data and model parallel settings.
Communication schemes have also been explored to accelerate incremental model updates across distributed workers. Three of the most common schemes are (Iandola et al., 2016; Zhao & Canny, 2013): (i) the OneToAll scheme has a 2KT communication time as a master node must communicate with all K workers individually, where T is the time for communicating data through one link in the network; (ii) the Tree AllReduce scheme takes 2 log2(K)T for weights to be aggregated and broadcasted to all workers following a tree topology; and (iii) the Butterfly AllReduce scheme in which all workers receive aggregated weights in log2(K)T using a butterfly network. We restrict the focus of PALEO to distributed communication schemes that return equivalent results to serial executions, and we thus do not consider the recently introduced butterfly mixing scheme of Zhao & Canny (2013), or non-deterministic asynchronous parameter servers.
3 PALEO
We now present PALEO, a model for the lean consumption of resources during the training of DNNs. PALEO decomposes the total execution time into computation time and communication time; both are estimated for each pass of a neural network’s evaluation given user specified choices within the design space of algorithms, hardware, and communications strategies. Figure 1 illustrates the overall idea. The computation time is calculated from factors including the size of the computation inputs imposed by the network architecture, the complexity of the algorithms and operations involved in the network layers, and the performance of the hardware to be used. The communication time is estimated based on the computational dependencies imposed by the network, the communication bandwidth of the hardware, and the assumed parallelization schemes. Once the network architecture and design space choices are fixed, all of the key factors in PALEO can be derived, and we can estimate execution time without actually implementing the entire network and/or an underlying software package.
2Training a neural network involves both forward and backward propagation, whereas deploying a trained network on a new data point involves only forward propagation. Thus, estimating the execution time of model training encompasses both model training and deployment, and is the focus of this work.
3.1 COMPUTATION MODELING
We first describe the computation model on a single machine. The computation in a neural network can be expressed as a directed graph N = 〈{u(i)}ni=1, {(u(i), u(j))}〉, where each node u(i) is associated with an operation f (i) on a device d(i); each directed edge (u(i), u(j)) represents the dependency that operation f (j) cannot be executed until f (i) is finished. We use Pa(u(j)) to represent the set of immediate parent nodes of u(j). We model each layer in the neural network as a node, and the connections between layers as edges. In the following text, we omit the superscript index when there is no ambiguity.
3.1.1 COMPUTATION TIME FOR A SINGLE LAYER
To model the runtime of a layer u, we consider the operation f and decompose the execution time of this operation into three terms (as shown in Figure 2a): the time to fetch the input produced by its parent layers R(Pa(u)); the time to perform the computation of f on the designated device d, i.e., C(f, d); and the time to write the outputs to the local memoryW(f, d). Assuming a sequential execution, the runtime for a node u can be written as a simple summation:
T (u) = R(Pa(u)) + C(f, d) +W(f, d). (1)
Among the three terms, the computation time C(f, d) is calculated as the FLOP (floating-point operation) counts of the operation divided by the computation speed (FLOPS; floating-point operation per second) of the device: C(f, d) = FLOPs(f)/speed(d). The IO times R(Pa(u)) and W(u) are calculated as the size of memory footprints involved in the computation divided by the IO bandwidth of the device. When inputs must be fetched from other devices, e.g. in the case of model parallelism, this IO bandwidth refers to the communication bandwidth between two devices. PALEO treats the speed and bandwidth of devices as parameters given to the model so that users can configure them to reflect user-specific configurations.
Using this per-layer model, we will next describe how to model the computation time of an entire network. We will subsequently we present FLOP counts for layer operations commonly used in modern DNNs in Section 4.
3.1.2 COMPUTATION TIME FOR NETWORKS
We first consider simple sequential structures where layers are constructed one after another, as in Figure 2b. The total execution time can be calculated as the sum of execution time of all layers T (N ) = ∑n i=1 T (u
(i)). While this calculation may seem trivial at first glance, it forms the foundation for modeling execution time for more complex architectures.
Parallel structures are not uncommon in DNNs; for example, the Inception model (Szegedy et al., 2015a) contains layers that can be evaluated simultaneously, and layers on different workers can run in parallel in model parallel setups (Dean et al., 2012). Figure 2c illustrates a parallel structure, where two convolutional layers (each followed by a pooling layer) are scheduled to be executed on two devices.
To model computation time of parallel structures, we identify synchronization barriers before and after every parallel structure and introduce a notation of supernode U = {G(i)}ki=1 as a set of disjoint subgraphs sandwiched by the synchronization barriers in the computation graph. When substituting the subgraphs with the supernode, the network is reduced to a sequential structure described above. For the supernode, the execution time T (U) is within the range [maxi T (G(i)), ∑ i T (G(i))], where the lower bound corresponds to perfect parallelization, the upper bound corresponds to sequential execution. Note that the execution time of a subgraph T (G(i)) can be calculated recursively.
3.1.3 COMPUTATION MODELING FOR LAYER OPERATIONS
In modern DNNs, the convolutional layer is one of the most commonly used and computationally intensive type of layer. For this reason, there has been many heavily optimized implementations (Chetlur et al., 2014; Vasilache et al., 2015; Lavin, 2016). Deriving plausible FLOP counts for other types of layers is a straightforward process, and in this section, we consider two leading implementations for convolutional operations: matrix multiplication and Fast Fourier Transform.
Following the notation used by Chetlur et al. (2014), a 2D convolutional layer during forward propagation3 takes an input feature map DN×C×H×W (which has a batch of N input feature maps with shape H ×W and C channels) and a set of convolutional filters FK×C×R×S (K filters with shape R×S and C channels). It produces N ×K feature maps each of shape P ×Q which can be calculated from the shapes of inputs and filters together with additional striding and padding parameters. The FLOP counts for the convolution operation can be expressed as 2KCRSNPQ. A commonly used implementation is to reduce convolution operations to matrix multiplications, which can be efficiently computed with well-optimized SGEMM routines on various platforms. Although these FLOP counts ignore auxiliary operations (e.g. indexing arithmetic in efficient implementations), they nonetheless provide a good estimate of FLOP counts for matrix multiplication implementations.
Another implementation is based on Fast Fourier Transform (Vasilache et al., 2015): both input feature maps and filters are transformed into the frequency domain, then element-wise multiplications are performed followed by an inverse Fourier transform. This implementation introduces computation and memory overhead in discrete Fourier transforms, but reduces the computation complexity to O(NCKHW +(NC+CK+NK)HW log(HW )). Convolutional layers with large filters or a
3Our arguments generalize to N-dimensional settings, and similar arguments apply for the backward pass.
large problem size can benefit from FFT implementations. When counting FLOPs, it is not possible to get exact counts without knowing the underlying implementation details. In PALEO, we adopt the commonly used FFT complexity 5n log2 n as the FLOP counts for complex-valued transformations of size n (Cooley & Tukey, 1965). To account for the IO overhead caused by auxiliary memories, PALEO estimates the memory size required for complex-valued matrices in the frequency domain and incorporates it into the data reading and writing terms. For FFT-based implementations with tilings, PALEO estimates the number of tiles from the convolution specifications.
The choice of algorithm – matrix multiplication or FFT – is problem specific, as it depends on the filter size, strides, input size of the convolutional layers, and memory workspace. In order to derive reasonable estimations for user-specific DNNs comparable to real executions, it is important for PALEO to make decisions comparable to real-world systems. Two common approaches are employed in existing DNN software frameworks and libraries to choose between these algorithms: (i) using predefined heuristics based on offline benchmarks; (ii) autotuning to empirically evaluate available algorithms on the given specification. Since autotuning is tied to platform and software implementations, for maximum generality PALEO by default takes the first approach. In particular, PALEO uses heuristics from cuDNN to make algorithm choices while also accounting for user preferences.
3.2 COMMUNICATION MODELING
We now describe our modeling for communication among multiple workers. Let |D| be the size of data to be communicated between two workers, and define B as the bandwidth of the communication channel. Then the communication time can simply be written as Tcomm = |D|/B. By using different bandwidth configurations, PALEO works for both scale-up setups (multiple GPUs on one machine) and scale-out setups (multiple machines in a cluster). Moreover, in data-parallel settings, an AllReduce operation is performed to synchronize model parameters across all workers after every backward pass. PALEO considers three communications schemes: OneToAll, Tree AllReduce, and Butterfly AllReduce. The communication time under these three schemes is described in Section 2.
3.3 PLATFORM PERCENT OF PEAK
Thus far, we have assumed that deep learning software platforms make perfect use of their underlying hardware. That is, that the CPUs and GPUs are operating at “peak FLOPS”, and that network and IO links are fully saturated. This has allowed our model to be platform independent.
However, this assumption is unreasonable in practice. For instance, achieving peak FLOPS is a difficult proposition, usually requiring customized libraries developed by organizations with intimate knowledge of the underlying hardware, e.g., Intel’s MKL (int, 2009), ATLAS (Whaley & Petitet, 2005), and cuDNN. Even these specially tuned libraries may fall short of peak execution by as much as 40% (atl). Further, any computation done outside the scope of PALEO (e.g. job scheduling, data copying) will exacerbate the observed inefficiency in practice. Sometimes such inefficiencies are warranted from the perspective of ease of programmability or maintenance of the learning platforms.
Rather than trying to measure and capture every source of inefficiency in every learning framework, we take a small number of representative deep learning workloads which contain convolutions, pooling, dropout, and fully connected layers and run them for a short time on a single GPU. Given observed total throughput and estimated total throughput on this benchmark we fit a scaling constant to estimate a platform percent of peak (PPP) parameter which captures the average relative inefficiency of the platform compared to peak FLOPS. Highly specialized frameworks (e.g. cuDNN) will in general have a computational PPP that is close to 100%, while frameworks with higher overheads may have PPP constants closer to 50% or less.
We follow a similar benchmarking procedure to estimate PPP for the communication link for TensorFlow. For the FireCaffe experiments, we estimate the communication PPP based on the empirical results for communication reported in Table 4 of the paper.
4 EXPERIMENTS
We now present empirical results which illustrate that PALEO is robust to the choice of network architecture, hardware, communication schemes, and parallelization strategies.
4.1 LAYER-WISE EVALUATION
We first compare PALEO-estimated runtimes with actual runtimes measured from TensorFlow4 (Abadi et al., 2015) execution in two popular CNN architectures: the one-tower variant of AlexNet (Krizhevsky, 2014b) and the 16-layer VGG network (Simonyan & Zisserman, 2014). PALEO uses cuDNN heuristics to choose algorithms and the auto-tuning mechanism in TensorFlow is disabled. Experiments are run on a NVIDIA TITAN X GPU with a 4 GB workspace limit.
For convolutional and fully connected layers, we evaluate forward computation, backward computation with respect to layer inputs, and backward computation with respect to filters separately (see Figure 4 in the appendix for the plots of layer-by-layer comparison.) Table 1 shows a comparison of full forward pass and backward pass with all layers included. PALEO’s per layer estimates are quite close to the actual TensorFlow execution, with only one layer – ‘fc6’ – consistently being underestimated by PALEO.5 In spite of this issue with ‘fc6’, our full pass estimates are remarkably accurate.
4.2 CASE STUDIES
We now revisit the questions posed at the beginning of the paper and demonstrate how PALEO can help in answering them. In this subsection we present three case studies. We extract experiment setups including network architectures, hardware specifications, communication schemes, and parallelization strategies from selected publications focusing on scalability of CNNs. We then plug those configurations into PALEO and compare the simulated scalability results with the reported results in the original publications. Table 2 summaries the configurations of PALEO in these experiments.
4TensorFlow 0.9 with cuDNN 4 backend. 5Examining the TensorFlow execution with the NVIDIA profiler revealed that TensorFlow spent two-thirds of its reported ‘fc6’ time in transforming data layout between NHWC and NCHW when calling the underlying cuBLAS primitives.
6Total time of forward pass, backward pass, and parameter update for one mini-batch on one worker. 7Reported times for Cases 1 and 3 are derived approximately from information in the publications. For Case
2 no run time information is provided.
4.2.1 CASE 1: NIN WITH FIRECAFFE
FireCaffe (Iandola et al., 2016) adopts the Tree AllReduce communication scheme when training a NiN model (Lin et al., 2013) in data parallel settings with up to 128 servers on the Titan supercomputer. They report a 38× speedup for NiN with batch size 1024 relative to single-GPU performance. Tabel 3 shows the results from PALEO compared with the results reported by FireCaffe.
4.2.2 CASE 2: INCEPTION WITH TENSORFLOW
Murray et al. (2016) reported their results in synchronously training the Inception model (Szegedy et al., 2015b) with TensorFlow and achieved a 56× speedup with 100 workers. They apply a weak scaling strategy with batch size 256 to keep GPUs saturated. Although Murray et al. (2016) leveraged a distributed parameter server rather than one of the three communications schemes considered in PALEO, the communication cost of Butterfly AllReduce can be viewed as a lower bound (Zhao & Canny, 2013). To account for the fact that they train with worker nodes each of which have 8 GPUs, we assumes a linear speedup for GPUs on the same host. Figure 3a shows a comparison between reported speedups and PALEO estimated speedups. For absolute runtime, in one of the experiments, their model completes 20 epochs of training after 100 hours when using 8 Tesla K40’s and a batch size 256. PALEO projects a 111 hours runtime under the same setting.
4.2.3 CASE 3: ALEXNET WITH HYBRID PARALLELISM
Krizhevsky (2014b) describes a hybrid model and data parallelism approach for training AlexNet using up to 8 GPUs with a weak scaling strategy. In his setup, each of the two CPUs connects to 4 GPUs, the communication bandwidth is penalized by 50% across the two groups as mentioned in the paper. Table 4 shows the comparison between PALEO’s projection and the original result, which are quite similar. Moreover, whereas Krizhevsky (2014b) does not quantify the speedup of hybrid parallelism relative to strict data parallelism, PALEO simulates training the entire network with only data parallelism (see last two columns of Table 4) in order to estimate this speedup.
4.3 HYPOTHETICAL SETUPS
In this subsection, we use PALEO in two hypothetical setups to analyze the scalability of AlexNet and a GAN model under different communication schemes.
4.3.1 ALEXNET IN A CLOUD-BASED SETUP
In this study, we present an analysis of data parallel training of AlexNet. We assume a modern cloud setup with a cluster of servers each equipped with a NVIDIA K80 GPU connected to a 20 Gbps network. In contrast to the Inception model with 23 million parameter, the one-tower variant of AlexNet has 50 million parameters and therefore doubles communication workload when training with data parallelism.
We show strong scaling for all three communication schemes in Figure 3c. Even when assuming a fairly large batch size of 2048 which is beneficial in distributed settings, we see very modest speedups. The OneToAll scheme achieves a max speedup of less than a 2× using 4 workers, while the communication-efficient Butterfly AllReduce scheme achieves a max speedup of roughly 5× when using 32 workers. The weak scaling results, shown in Figure 3b, show drastically improved scaling results, as we observe nearly linear speedups as we increase the number of workers. However, it is important to note that we are increasing the effective batch size as we increase the number of workers, and it is well-known that training with large effective batch-sizes can yield models with substandard accuracy (Breuel, 2015).
4.3.2 GAN ARCHITECTURE
PALEO can be applied to architectures other than CNNs. We profile a generative adversarial network (GAN) inspired by Radford et al. (2015) for the LSUN dataset with the same hardware assumptions as the previous case study. Table 5 shows that PALEO estimations are close to empirical TensorFlow run time for both the discriminator and generator networks. Figure 3d plots the estimated speedups for training the model with a batch size 256 on up to 128 workers under strong scaling. Without communication-intensive fully-connected layers, while training this GAN architecture is more scalable than AlexNet, PALEO still only predicts an 8× sub-linear speedup with 64 workers.
5 CONCLUSION
We introduced PALEO – an analytical performance model for exploring the space of scalable deep learning systems. By extracting computational requirements carried by neural network architectures and mapping them to the design space of software, hardware, and communication strategies, PALEO can effectively and accurately model the expected scalability and performance of a putative deep learning system.

In PALEO the authors propose a simple model of execution of deep neural networks. It turns out that even this simple model allows to quite accurately predict the computation time for image recognition networks both in single-machine and distributed settings.
The ability to predict network running time is very useful, and the paper shows that even a simple model does it reasonably, which is a strength. But the tests are only performed on a few networks of very similar type (AlexNet, Inception, NiN) and only in a few settings. Much broader experiments, including a variety of models (RNNs, fully connected, adversarial, etc.) in a variety of settings (different batch sizes, layer sizes, node placement on devices, etc.) would probably reveal weaknesses of the proposed very simplified model. This is why this reviewer considers this paper borderline -- it's a first step, but a very basic one and without sufficiently large experimental underpinning.
More experiments were added, so I'm updating my score.


The question of how procedural knowledge is represented and inferred is a fundamental problem in machine learning and artificial intelligence. Recent work on program induction has proposed neural architectures, based on abstractions like stacks, Turing machines, and interpreters, that operate on abstract computational machines or on execution traces. But the recursive abstraction that is central to procedural knowledge is perhaps most naturally represented by symbolic representations that have syntactic structure, such as logical expressions and source code. Combining abstract, symbolic reasoning with continuous neural reasoning is a grand challenge of representation learning. As a step in this direction, we propose a new architecture, called neural equivalence networks, for the problem of learning continuous semantic representations of mathematical and logical expressions. These networks are trained to represent semantic equivalence, even of expressions that are syntactically very different. The challenge is that semantic representations must be computed in a syntax-directed manner, because semantics is compositional, but at the same time, small changes in syntax can lead to very large changes in semantics, which can be difficult for continuous neural architectures. We perform an exhaustive evaluation on the task of checking equivalence on a highly diverse class of symbolic algebraic and boolean expression types, showing that our model significantly outperforms existing architectures.
1 INTRODUCTION
Representing and learning knowledge about the world requires not only learning declarative knowledge about facts but also procedural knowledge, knowledge about how to do things, which can be complex yet difficult to articulate explicitly. The goal of building systems that learn procedural knowledge has motivated many recent architectures for learning representations of algorithms (Graves et al., 2014; Reed & de Freitas, 2016; Kaiser & Sutskever, 2016). These methods generally learn from execution traces of programs (Reed & de Freitas, 2016) or input-output pairs generated from a program (Graves et al., 2014; Kurach et al., 2015; Riedel et al., 2016; Grefenstette et al., 2015; Neelakantan et al., 2015).
However, the recursive abstraction that is central to procedural knowledge is perhaps most naturally represented not by abstract models of computation, as in that work, but by symbolic representations that have syntactic structure, such as logical expressions and source code. One type of evidence for this claim is the simple fact that people communicate algorithms using mathematical formulae and pseudocode rather than Turing machines. Yet, apart from some notable exceptions (Alemi et al., 2016; Piech et al., 2015; Allamanis et al., 2016; Zaremba & Sutskever, 2014), symbolic representations of procedures have received relatively little attention within the machine learning literature as a source of information for representing procedural knowledge.
In this paper, we address the problem of learning continuous semantic representations (SEMVECs) of symbolic expressions. The goal is to assign continuous vectors to symbolic expressions in such a way that semantically equivalent, but syntactically diverse expressions are assigned to identical (or
highly similar) continuous vectors, when given access to a training set of pairs for which semantic equivalence is known. This is an important but hard problem; learning composable SEMVECs of symbolic expressions requires that we learn about the semantics of symbolic elements and operators and how they map to the continuous representation space, thus encapsulating implicit knowledge about symbolic semantics and its recursive abstractive nature.
Our work in similar in spirit to the work of Zaremba & Sutskever (2014), who focus on learning expression representations to aid the search for computationally efficient identities. They use recursive neural networks (TREENN)1 (Socher et al., 2012) for modelling homogenous, single-variable polynomial expressions. While they present impressive results, we find that the TREENN model fails when applied to more complex symbolic polynomial and boolean expressions. In particular, in our experiments we find that TREENNs tend to assign similar representations to syntactically similar expressions, even when they are semantically very different. The underlying conceptual problem is how to develop a continuous representation that follows syntax but not too much, that respects compositionality while also representing the fact that a small syntactic change can be a large semantic one.
To tackle this problem, we propose a new architecture, called neural equivalence networks (EQNETs). EQNETs learn how syntactic composition recursively composes SEMVECs, like a TREENN, but are also designed to model large changes in semantics as the network progresses up the syntax tree. As equivalence is transitive, we formulate an objective function for training based on equivalence classes rather than pairwise decisions. The network architecture is based on composing residual-like multi-layer networks, which allows more flexibility in modeling the semantic mapping up the syntax tree. To encourage representations within an equivalence class to be tightly clustered, we also introduce a training method that we call subexpression forcing, which uses an autoencoder to force the representation of each subexpression to be predictable from its syntactic neighbors. Experimental evaluation on a highly diverse class of symbolic algebraic and boolean expression types shows that EQNETs dramatically outperform existing architectures like TREENNs and RNNs.
To summarize, the main contributions of our work are: (a) We formulate the problem of learning continuous semantic representations (SEMVECs) of symbolic expressions and develop benchmarks for this task. (b) We present neural equivalence networks (EQNETs), a neural network architecture that learns to represent expression semantics onto a continuous semantic representation space and how to perform symbolic operations in this space. (c) We provide an extensive evaluation on boolean and polynomial expressions, showing that EQNETs perform dramatically better than state-of-the-art alternatives. Code and data are available at groups.inf.ed.ac.uk/cup/semvec.
2 MODEL
In this work, we are interested in learning semantic, composable representations of mathematical expressions (SEMVEC) and learn to generate identical representations for expressions that are semantically equivalent, i.e. they belong to the same equivalence class. Equivalence is a stronger property than similarity that is habitually learned by neural networks, since equivalence is additionally a transitive relationship.
Problem Hardness. Finding the equivalence of arbitrary symbolic expressions is a NP-hard problem or worse. For example, if we focus on boolean expressions, reducing an expression to the representation of the false equivalence class amounts to proving its non-satisfiability — an NPcomplete problem. Of course, we do not expect to circumvent an NP-complete problem with neural networks. A network for solving boolean equivalence would require an exponential number of nodes in the size of the formula if P 6= NP . Instead, our goal is to develop architectures whose inductive biases allow them to efficiently learn to solve the equivalence problems for expressions that are similar to a smaller number of expressions in a given training set. This requires that the network learn identical representations for expressions that may be syntactically different but semantically equivalent and also discriminate between expressions that may be syntactically very similar but are non-equivalent. Appendix A shows a sample of such expressions that illustrate the hardness of this problem.
1To avoid confusion, we use TREENN for recursive neural networks and retain RNN for recurrent neural networks.
rc2
rc1
a
a
c Combine l̄0
rp
SubexpForce
rc1
rc2
rp r̃c1
r̃c2
r̃p Combine
r̃c1–rc1, r̃c2–rc2, r̃p–rp
x̃ l̄1
l̄out
/ ‖·‖2
SemVec
(a) Architectural diagram of EQNETs. Example parse tree shown is of the boolean expression (a ∨ c) ∧ a.
COMBINE (rc0 , . . . , rck , τn) l̄0 ← [rc0 , . . . , rck ] l̄1 ← σ ( Wi,τn · l̄0 ) l̄out ←Wo0,τn · l̄0 +Wo1,τn · l̄1 return l̄out/
∥∥l̄out∥∥2 (b) COMBINE of EQNET.
SUBEXPFORCE (rc0 , . . . , rck , rn, τn) x← [rc0 , . . . , rck ] x̃← tanh (Wd · tanh (We,τn · [rn,x] · n)) x̃← x̃ · ‖x‖2 / ‖x̃‖2 r̃n ← COMBINE(x̃, τn) return − ( x̃>x + r̃>n rn
) (c) Loss function used for subexpression forcing
Figure 1: EQNET architecture.
Notation and Framework. We employ the general framework of recursive neural networks (TREENN) (Socher et al., 2012; 2013) to learn to compose subtree representations into a single representation. The TREENNs we consider operate on tree structures of the syntactic parse of a formula. Given a tree T , TREENNs learn distributed representations by recursively computing the representations of its subtrees. We denote the children of a node n as ch(n) which is a (possibly empty) ordered tuple of nodes. We also use par(n) to refer to the parent node of n. Each node in our tree has a type, e.g. a terminal node could be of type “a” referring to the variable a or of type “and” referring to a node of the logical and (∧) operation. We refer to the type of a node n as τn. At a high level, TREENNs retrieve the representation of a tree T rooted at node ρ, by invoking TREENET(ρ) that returns a vector representation rρ ∈ RD, i.e., a SEMVEC, using the function TREENET (current node n)
if n is not a leaf then rn ← COMBINE(TREENET(c0), . . . , TREENET(ck), τn), where (c0, . . . , ck) = ch(n) else rn ← LOOKUPLEAFEMBEDDING(τn)
return rn The general framework of TREENET allows two points of variation, the implementation of LOOKUPLEAFEMBEDDING and COMBINE. The traditional TREENNs (Socher et al., 2013) define LOOKUPLEAFEMBEDDING as a simple lookup operation within a matrix of embeddings and COMBINE as a single-layer neural network. As discussed next, these will both prove to be serious limitations in our setting.
2.1 NEURAL EQUIVALENCE NETWORKS
We now define the neural equivalence networks (EQNET) that learn to compose representations of equivalence classes into new equivalence classes (Figure 1a). Our network follows the TREENN architecture, that is, our EQNETs are implemented using the TREENET, so as to model the compositional nature of symbolic expressions. However, the traditional TREENNs (Socher et al., 2013) use a single-layer neural network at each tree node. During our preliminary investigations and in Section 3, we found that single layer networks are not expressive enough to capture many operations, even a simple XOR boolean operator, because representing these operations required high-curvature operations in the continuous semantic representation space. Instead, we turn to multi-layer neural networks. In particular, we define the COMBINE in Figure 1b. This uses a two-layer MLP with a residual-like connection to compute the SEMVEC of each parent node in that syntax tree given that of its children. Each node type τn, e.g., each logical operator, has a different set of weights. We experimented with deeper networks but this did not yield any improvements. However, as TREENN
become deeper, they suffer from optimization issues, such as diminishing and exploding gradients. This is essentially because of the highly compositional nature of tree structures, where the same network (i.e. the COMBINE non-linear function) is used recursively, causing it to “echo” its own errors and producing unstable feedback loops. We observe this problem even with only two-layer MLPs, as the overall network can become quite deep when using two layers for each node in the syntax tree.
We resolve this issues in a few different ways. First, we constrain each SEMVEC to have unit norm. That is, we set LOOKUPLEAFEMBEDDING(τn) = Cτn/ ‖Cτn‖2 , and we normalize the output of the final layer of COMBINE in Figure 1b. The normalization step of l̄out and Cτn is somewhat similar to layer normalization (Ba et al., 2016), although applying layer normalization directly did not work for our problem. Normalizing the SEMVECs partially resolves issues with diminishing and exploding gradients, and removes a spurious degree of freedom in the semantic representation. As simple as this modification may seem, we found that it was vital to obtaining effective performance, and all of our multi-layer TREENNs converged to low-performing parameters without it.
However this modification is not sufficient, since the network may learn to map expressions from the same equivalence class to multiple SEMVECs in the continuous space. We alleviate this problem using a method that we call subexpression forcing that guides EQNET to cluster its output to one location per equivalence class. We encode each parent-children tuple [rc0 , . . . , rck , rn] containing the (computed) representations of the children and parent node into a low-dimensional space using a denoising autoencoder. We then seek to minimize the reconstruction error of the child representations (r̃c0 , . . . , r̃ck ) as well as the reconstructed parent representation r̃n that can be computed from the reconstructed children. Thus more formally, we minimize the return value of SUBEXPFORCE in Figure 1c where n is a binary noise vector with κ percent of its elements set to zero. Note that the encoder is specific to the type of τn. Although our SUBEXPFORCE may seem similar to the recursive autoencoders of Socher et al. (2011) it differs significantly in form and purpose, since it acts as an autoencoder on the whole parent-children representation tuple and the encoding is not used within the computation of the parent representation. In addition, this constraint has two effects. It forces each parent-children tuple to “live” in a low-dimensional space, providing a clustering-like behavior. Second, it implicitly joins distinct locations that belong to the same equivalence class. To illustrate the latter point, imagine two semantically equivalent c′0 and c ′′ 0 child
nodes of different nodes that have two geometrically distinct representations ∥∥rc′0 − rc′′0 ∥∥2 and COMBINE(rc′0 , . . . ) ≈ COMBINE(rc′′0 , . . . ). In some cases due to the autoencoder noise, the differences between the input tuple x′,x′′ that contain rc′0 and rc′′0 will be non-existent and the decoder will be forced to predict a single location r̃c0 (possibly different from rc′0 and rc′′0 ). Then, when minimizing the reconstruction error, both rc′0 and rc′′0 will be attracted to r̃c0 and eventually should merge.
2.2 TRAINING
We train EQNETs from a dataset of expressions whose semantic equivalence is known. Given a training set T = {T1 . . . TN} of parse trees of expressions, we assume that the training set is partitioned into equivalence classes E = {e1 . . . eJ}. We use a supervised objective similar to classification; the difference between classification and our setting is that whereas standard classification problems consider a fixed set of class labels, in our setting the number of equivalence classes in the training set will vary withN . Given an expression tree T that belongs to the equivalence class ei ∈ E , we compute the probability
P (ei|T ) = exp
( TREENN(T )>qei + bi )∑ j exp ( TREENN(T )>qej + bj
) (1) where qei are model parameters that we can interpret as representations of each equivalence classes that appears in the training class, and bi are bias terms. Note that in this work, we only use information about the equivalence class of the whole expression T , ignoring available information about subexpressions. This is without loss of generality, because if we do know the equivalence class of a subexpression of T , we can simply add that subexpression to the training set. Directly maximizing P (ei|T ) would be bad for EQNET since its unit-normalized outputs cannot achieve high probabilities within the softmax. Instead, we train a max-margin objective that maximizes
classification accuracy, i.e.
LACC(T, ei) = max ( 0, arg max ej 6=ei,ej∈E logP (ej |T )− logP (ei|T ) +m ) (2)
where m > 0 is a scalar margin. And therefore the optimized loss function for a single expression tree T that belongs to equivalence class ei ∈ E is
L(T, ei) = LACC(T, ei) + µ |Q| ∑ n∈Q SUBEXPFORCE(ch(n), n) (3)
where Q = {n ∈ T : | ch(n)| > 0}, i.e. contains the non-leaf nodes of T and µ ∈ (0, 1] a scalar weight. We found that subexpression forcing is counterproductive early in training, before the SEMVECs begin to represent aspects of semantics. So, for each epoch t, we set µ = 1− 10−νt with ν ≥ 0. Instead of the supervised objective that we propose, an alternative option for training EQNET would be a Siamese objective (Chopra et al., 2005) that learns about similarities (rather than equivalence) between expressions. In practice, we found the optimization to be very unstable, yielding suboptimal performance. We believe that this has to do with the compositional and recursive nature of the task that creates unstable dynamics and the fact that equivalence is a stronger property than similarity.
3 EVALUATION
Datasets. We generate datasets of expressions grouped into equivalence classes from two domains. The datasets from the BOOL domain contain boolean expressions and the POLY datasets contain polynomial expressions. In both domains, an expression is either a variable, a binary operator that combines two expressions, or a unary operator applied to a single expression. When defining equivalence, we interpret distinct variables as referring to different entities in the domain, so that, e.g., the polynomials c · (a · a+ b) and f · (d · d+ e) are not equivalent. For each domain, we generate “simple” datasets which use a smaller set of possible operators and “standard” datasets which use a larger set of more complex operators. We generate each dataset by exhaustively generating all parse trees up to a maximum tree size. All expressions are then simplified into a canonical from in order to determine their equivalence class and are grouped accordingly. Table 1 shows the datasets we generated. We also present in Appendix A some sample expressions. For the polynomial domain, we also generated ONEV-POLY datasets, which are polynomials over a single variable, since they are similar to the setting considered by Zaremba & Sutskever (2014) — although ONEV-POLY is still a little more general because it is not restricted to homogeneous polynomials. Learning SEMVECs for boolean expressions is already a hard problem; with n boolean variables, there are 22 n
equivalence classes (i.e. one for each possible truth table). We split the datasets into training, validation and test sets. We create two test sets, one to measure generalization performance on equivalence classes that were seen in the training data (SEENEQCLASS), and one to measure generalization to unseen equivalence classes (UNSEENEQCLASS). It is easiest to describe UNSEENEQCLASS first. To create the UNSEENEQCLASS, we randomly select 20% of all the equivalence classes, and place all of their expressions in the test set. We select equivalence classes only if they contain at least two expressions but less than three times the average number of expressions per equivalence class. We thus avoid selecting very common (and hence trivial to learn) equivalence classes in the testset. Then, to create SEENEQCLASS, we take the remaining 80% of the equivalence classes, and randomly split the expressions in each class into training, validation, SEENEQCLASS test in the proportions 60%–15%–25%. We provide the datasets online.
Baselines. To compare the performance of our model, we train the following baselines. TF-IDF: learns a representation given the tokens of each expression (variables, operators and parentheses). This can capture topical/declarative knowledge but is unable to capture procedural knowledge. GRU refers to the token-level gated recurrent unit encoder of Bahdanau et al. (2015) that encodes the token-sequence of an expression into a distributed representation. Stack-augmented RNN refers to the work of Joulin & Mikolov (2015) which was used to learn algorithmic patterns and uses a stack as a memory and operates on the expression tokens. We also include two recursive neural network (TREENN) architectures. The 1-layer TREENN which is the original TREENN also used
by Zaremba & Sutskever (2014). We also include a 2-layer TREENN, where COMBINE is a classic two-layer MLP without residual connections. This shows the effect of SEMVEC normalization and subexpression forcing.
Hyperparameters. We tune the hyperparameters of the baselines and EQNET using Bayesian optimization (Snoek et al., 2012), optimizing on a boolean dataset with 5 variables and maximum tree size of 7 (not shown in Table 1). We use the average k-NN (k = 1, . . . , 15) statistics (described next) as an optimization metric. The selected hyperparameters are detailed in Appendix C.
3.1 QUANTITATIVE EVALUATION
Metric. To evaluate the quality of the learned representations we count the proportion of k nearest neighbors of each expression (using cosine similarity) that belong to the same equivalence class. More formally, given a test query expression q in an equivalence class c we find the k nearest neighbors Nk(q) of q across all expressions, and define the score as
scorek(q) = |Nk(q) ∩ c| min(k, |c|) . (4)
To report results for a given testset, we simply average scorek(q) for all expressions q in the testset.
Evaluation. Figure 2 presents the average scorek across the datasets for each model. Table 1 shows score5 of UNSEENEQCLASS for each dataset. Detailed plots can be found in Appendix B. It can be clearly seen that EQNET performs better for all datasets, by a large margin. The only exception is POLY5, where the two-layer TREENN performs better. However, this may have to do with the small size of the dataset. The reader may observe that the simple datasets (containing fewer operations and variables) are easier to learn. Understandably, introducing more variables increases the size of the represented space reducing performance. The tf-idf method performs better in settings where more variables are included, because it captures well the variables and operations used. Similar observations can be made for sequence models. The one and two layer TREENNs have mixed performance; we believe that this has to do with exploding and diminishing gradients due to the deep and highly compositional nature of TREENNs. Although Zaremba & Sutskever (2014) consider a different problem to us, they use data similar to the ONEV-POLY datasets with a traditional TREENN architecture. Our evaluation suggests that EQNETs perform much better within the ONEV-POLY setting.
¬(c ⊕ (a ∧ ((a ⊕ c) ∧ b))) ((c ∨ (¬b))⇒ a) ∧ (a ⇒ a) ((b ⊕ (¬c)) ∧ b)⊕ (a ∨ b)
((b · a)− a) · b a − ((a + b) · a) ((c · b) · c) · a b + ((b · b) · b)
Figure 3: Visualization of score5 for all expression nodes for three BOOL10 and four POLY8 test sample expressions using EQNET. The darker the color, the lower the score, i.e. white implies a score of 1 and dark red a score of 0.
Evaluation of Compositionality. We evaluate whether the EQNETs have successfully learned to compute compositional representations, rather than overfitting to expression trees of a small size. We evaluate this by considering a type of transfer setting, in which we train on simpler datasets, but tested on more complex ones; for example, training on the training set of BOOL5 but testing on the testset of BOOL8. We average over 11 different train-test pairs (full list in Figure 6) and present the results in Figure 2b-i and Figure 2b-ii (note the differences in scale to the two figures on the left). These graphs again show that EQNETs are dramatically better than any of the other methods, and indeed, performance is only a bit worse than in the non-transfer setting.
Impact of EQNET Components EQNETs differ from traditional TREENNs in two major components, which we analyze here. First, SUBEXPFORCE has a positive impact on performance. When training the network with and without subexpression forcing, on average, the area under the curve (AUC) of the scorek decreases by 16.8% on the SEENEQCLASS and 19.7% on the UNSEENEQCLASS. This difference is smaller in the transfer setting of Figure 2b-i and Figure 2b-ii, where AUC decreases by 8.8% on average. However, even in this setting we observe that SUBEXPFORCE helps more in large and diverse datasets. The second key difference to traditional TREENNs is the output normalization at each layer. Comparing our model to the one-layer and two-layer TREENNs again, we find that output normalization results in important improvements (the two-layer TREENNs have on average 60.9% smaller AUC).
3.2 QUALITATIVE EVALUATION
Table 2 and Table 3 shows expressions whose SEMVEC nearest neighbor is of an expression of another equivalence class. Manually inspecting boolean expressions, we find that EQNET confusions happen more when a XOR or implication operator is involved. In fact, we fail to find any confused expressions for EQNET not involving these operations in BOOL5 and in the top 100 expressions in BOOL10. As expected, tf-idf confuses expressions with others that contain the same operators and
variables ignoring order. In contrast, GRU and TREENN tend to confuse expressions with very similar symbolic representation differing in one or two deeply nested variables or operators. In contrast, EQNET tends to confuse fewer expressions (as we previously showed) and the confused expressions tend to be more syntactically diverse and semantically related.
Figure 3 shows a visualization of score5 for each node in the expression tree. One may see that as EQNET knows how to compose expressions that achieve good score, even if the subexpressions achieve a worse score. This suggests that for common expressions, (e.g. single variables and monomials) the network tends to select a unique location, without merging the equivalence classes or affecting the upstream performance of the network. Larger scale interactive t-SNE visualizations can be found online.
Figure 4 presents two PCA visualizations of the learned embeddings of simple expressions and their negations/negatives. It can be easily discerned that the black dots and their negations (in red) are easily discriminated in the semantic representation space. Figure 4b shows this property in a very clear manner: left-right discriminates between polynomials with a and −a, top-bottom between polynomials that contain b and −b and the diagonal y = x between c and −c. We observe a similar behavior in Figure 4a for boolean expressions.
4 RELATED WORK
Researchers have proposed compilation schemes that can transform any given program or expression to an equivalent neural network (Gruau et al., 1995; Neto et al., 2003; Siegelmann, 1994). One can consider a serialized version of the resulting neural network as a representation of the expression. However, it is not clear how we could compare the serialized representations corresponding to two expressions and whether this mapping preserves semantic distances.
Recursive neural networks (TREENN) (Socher et al., 2012; 2013) have been successfully used in NLP with multiple applications. Socher et al. (2012) show that TREENNs can learn to compute the values of some simple propositional statements. EQNET’s SUBEXPFORCE may resemble recursive autoencoders (Socher et al., 2011) but differs in form and function, encoding the whole parent-children tuple to force a clustering behavior. In addition, when encoding each expression our architecture does not use a pooling layer but directly produces a single representation for the expression.
Mou et al. (2016) use tree convolutional neural networks to classify code into 106 student submissions tasks. Although their model learns intermediate representations of the student tasks, it is a way of learning task-specific features in the code, rather than of learning semantic representations of programs. Piech et al. (2015) also learn distributed matrix representations of programs from student submissions. However, to learn the representations, they use input and output program states and do not test over program equivalence. Additionally, these representations do not necessarily represent program equivalence, since they do not learn the representations over the exhaustive set of all possible input-output states. Allamanis et al. (2016) learn variable-sized representations of
source code snippets to summarize them with a short function-like name. This method aims to learn summarization features in code rather than to learn representations of symbolic expression equivalence.
More closely related is the work of Zaremba & Sutskever (2014) who use a recursive neural network (TREENN) to guide the tree search for more efficient mathematical identities, limited to homogeneous single-variable polynomial expressions. In contrast, EQNETs consider at a much wider set of expressions, employ subexpression forcing to guide the learned SEMVECs to better represent equivalence, and do not use search when looking for equivalent expressions. Alemi et al. (2016) use RNNs and convolutional neural networks to detect features within mathematical expressions and speed the search for premise selection during automated theorem proving but do not explicitly account for semantic equivalence. In the future, SEMVECs may find useful applications within this work.
Our work is also related to recent work on neural network architectures that learn controllers/programs (Gruau et al., 1995; Graves et al., 2014; Joulin & Mikolov, 2015; Grefenstette et al., 2015; Dyer et al., 2015; Reed & de Freitas, 2015; Neelakantan et al., 2015; Kaiser & Sutskever, 2016). In contrast to this work, we do not aim to learn how to evaluate expressions or execute programs with neural network architectures but to learn continuous semantic representations (SEMVECs) of expression semantics irrespectively of how they are syntactically expressed or evaluated.
5 DISCUSSION & CONCLUSIONS
In this work, we presented EQNETs, a first step in learning continuous semantic representations (SEMVECs) of procedural knowledge. SEMVECs have the potential of bridging continuous representations with symbolic representations, useful in multiple applications in artificial intelligence, machine learning and programming languages.
We show that EQNETs perform significantly better than state-of-the-art alternatives. But further improvements are needed, especially for more robust training of compositional models. In addition, even for relatively small symbolic expressions, we have an exponential explosion of the semantic space to be represented. Fixed-sized SEMVECs, like the ones used in EQNET, eventually limit the capacity that is available to represent procedural knowledge. In the future, to represent more complex procedures, variable-sized representations would seem to be required.
ACKNOWLEDGMENTS
This work was supported by Microsoft Research through its PhD Scholarship Programme and the Engineering and Physical Sciences Research Council [grant number EP/K024043/1]. We thank the University of Edinburgh Data Science EPSRC Centre for Doctoral Training for providing additional computational resources.
A SYNTHETIC EXPRESSION DATASETS
Below are sample expressions within an equivalence class for the two types of datasets we consider.
BOOL8 (¬a) ∧ (¬b) (¬a ∧ ¬c) ∨ (¬b ∧ a ∧ c) ∨ (¬c ∧ b) (¬a) ∧ b ∧ c
a¬((¬a)⇒ ((¬a) ∧ b)) c⊕ (((¬a)⇒ a)⇒ b) ¬((¬b) ∨ ((¬c) ∨ a)) ¬((b ∨ (¬(¬a))) ∨ b) ¬((b⊕ (b ∨ a))⊕ c) ((a ∨ b) ∧ c) ∧ (¬a) (¬a)⊕ ((a ∨ b)⊕ a) ¬((¬(b ∨ (¬a)))⊕ c) (¬((¬(¬b))⇒ a)) ∧ c
(b⇒ (b⇒ a)) ∧ (¬a) ((b ∨ a)⊕ (¬b))⊕ c) (c ∧ (c⇒ (¬a))) ∧ b ((¬a)⇒ b)⇒ (a⊕ a) (¬((b⊕ a) ∧ a))⊕ c b ∧ (¬(b ∧ (c⇒ a)))
False (¬a) ∧ (¬b) ∨ (∧c) ¬a ∨ b (a⊕ a) ∧ (c⇒ c) (a⇒ (¬c))⊕ (a ∨ b) a⇒ ((b ∧ (¬c)) ∨ b) (¬b) ∧ (¬(b⇒ a)) (a⇒ (c⊕ b))⊕ b ¬(¬((b ∨ a)⇒ b)) b ∧ ((a ∨ a)⊕ a) b⊕ (a⇒ (b⊕ c)) (¬a)⊕ (¬(b⇒ (¬a)))
((¬b) ∧ b)⊕ (a⊕ a) (b ∨ a)⊕ (x⇒ (¬a)) b ∨ (¬((¬b) ∧ a)) c ∧ ((¬(a⇒ a)) ∧ c) b⊕ ((¬a) ∨ (c⊕ b)) ¬((a⇒ (a⊕ b)) ∧ a)
POLY8 −a− c c2 b2c2
(b− a)− (c+ b) (c · c) + (b− b) (b · b) · (c · c) b− (c+ (b+ a)) ((c · c)− c) + c c · (c · (b · b)) a− ((a+ a) + c) ((b+ c)− b) · c (c · b) · (b · c) (a− (a+ a))− c c · (c− (a− a)) ((c · b) · c) · b (b− b)− (a+ c) c · c ((c · c) · b) · b
c b · c b− c c− ((c− c) · a) (c− (b− b)) · b (a− (a+ c)) + b c− ((a− a) · c) (b− (c− c)) · c (a− c)− (a− b) ((a− a) · b) + c (b− b) + (b · c) (b− (c+ c)) + c
(c+ a)− a c · ((b− c) + c) (b− (c− a))− a (a · (c− c)) + c (b · c) + (c− c) b− ((a− a) + c)
B DETAILED EVALUATION
Figure 5 presents a detailed evaluation for our k-NN metric for each dataset. Figure 6 shows the detailed evaluation when using models trained on simpler datasets but tested on more complex ones, essentially evaluating the learned compositionality of the models. Figure 9 show how the performance varies across the datasets based on their characteristics. As expected as the number of variables increase, the performance worsens (Figure 9a) and expressions with more complex operators tend to have worse performance (Figure 9b). In contrast, Figure 9c suggests no obvious correlation between performance and the entropy of the equivalence classes within the datasets. The results for UNSEENEQCLASS look very similar and are not plotted here.
C MODEL HYPERPARAMETERS
The optimized hyperparameters are detailed in Table 4. All hyperparameters were optimized using the Spearmint (Snoek et al., 2012) Bayesian optimization package. The same range of values was used for all common model hyperparameters.
5 10
B O
O L L5 →
B O
O L 8
5 10
B O
O L 5→
B O
O L 8
5 10
B O
O L 5→
B O
O L 10
5 10
P O
LY 8→
S IM
P P
O LY
8
5 10
S IM
P P
O LY
5→ S
IM P
P O
LY 10
5 10
S IM
P P
O LY
8→ S
IM P
P O
LY 10
5 10
P O
LY 5→
S IM
P P
O LY
10
5 10
P O
LY 8→
S IM
P P
O LY
10
5 10
O N
E V
-P O
LY 10 →
O N
E V
-P O
LY 13
5 10
P O
LY 8→
O N
E V
-P O
LY 13
5 10 0.0
0.2
0.4
0.6
0.8
1.0
P O
LY 5→
P O
LY 8
(a) SEENEQCLASS evaluation using model trained on simpler datasets. Caption is “model trained on”→“Test dataset”.
5 10
B O
O L L5 →
B O
O L 8
5 10
B O
O L 5→
B O
O L 8
5 10
B O
O L 5→
B O
O L 10
5 10
P O
LY 8→
S IM
P P
O LY
8
5 10
S IM
P P
O LY
5→ S
IM P
P O
LY 10
5 10
S IM
P P
O LY
8→ S
IM P
P O
LY 10
5 10
P O
LY 5→
S IM
P P
O LY
10
5 10
P O
LY 8→
S IM
P P
O LY
10
5 10
O N
E V
-P O
LY 10 →
O N
E V
-P O
LY 13
5 10
P O
LY 8→
O N
E V
-P O
LY 13
5 10 0.0
0.2
0.4
0.6
0.8
1.0
P O
LY 5→
P O
LY 8
(b) Evaluation of compositionality. UNSEENEQCLASS evaluation using model trained on simpler datasets. Caption is “model trained on”→“Test dataset”.
tf-idf GRU StackRNN TreeNN-1Layer TreeNN-2Layer EqNet
Figure 6: Evaluation of compositionality. Evaluation of scorex (y axis) for x = 1, . . . , 15. The markers are shown every five ticks of the x-axis to make the graph more clear. TREENN refers to the model of Socher et al. (2012).

This work proposes to compute embeddings of symbolic expressions (e.g., boolean expressions, or polynomials) such that semantically equivalent expressions are near each other in the embedded space.  The proposed model uses recursive neural networks where the architecture is made to match that of the parse tree of a given symbolic expression.  To train the model parameters, the authors create a dataset of expressions where semantic equivalence relationships are known and minimize a loss function so that equivalent expressions are closer to each other than non-equivalent expressions via a max-margin loss function.  The authors also use a “subexpression forcing” mechanism which, if I understand it correctly, encourages the embeddings to respect some kind of compositionality.
Results are shown on a few symbolic expression datasets created by the authors and the proposed method is demonstrated to outperform baselines pretty convincingly.  I especially like the PCA visualization where the action of negating an expression is shown to correspond roughly to negating the embedding in its vector space — it is a lot like the man - woman + queen = king type embeddings that we see in the word2vec and glove style papers.
The weakest part of the paper is probably that the setting seems somewhat contrived — I can’t really think of a real setting where it is easy to have a training set of known semantic equivalences, but still more worth it to use a neural network to do predictions.   The authors have also punted on dealing with variable names, assuming that distinct variables refer to different entities in the domain.  This is understandable, as variable names add a whole new layer of complexity on an already difficult problem, but also seems high limiting.  For example, the proposed methods would not be useable in an “equation search engine” unless we were able to accurately canonicalize variable names in some way.
Other miscellaneous points:
* Regarding problem hardness, I believe that the problem of determining if two expressions are equivalent is actually undecidable — see the “word problem for Thue systems”.  Related to this, I was not able to figure out how the authors determine ground truth equivalence in their training sets.  They say that expressions are simplified into a canonical form and grouped, but this seems to not be possible in general, so one question is — is it possible that equivalent expressions in the training data would have been mapped to different canonical forms?  Would it have been easier/possible to construct and compare truth tables?
* The “COMBINE” operation uses what the authors describe as a residual-like connection.  Looking at the equations, the reason why this is not actually a residual connection is because of the weight matrix that is multiplied by the lower level l_0 features.  A true residual connection would have passed the features through unchanged (identity connection) and would have also been better at fighting gradient explosion…. so is there a reason why this was used rather than an identity connection?
* In table 3, the first tf-idf entry: a + (c+a) * c seems equivalent to a + (c * (a+c))
* Vertical spacing between Figure 4 caption and body of text is very small and looks like the caption continues into the body of the text.