Stochastic gradient descent (SGD) is a well-known method for regression and classification tasks. However, it is an inherently sequential algorithm — at each step, the processing of the current example depends on the parameters learned from the previous examples. Prior approaches to parallelizing SGD, such as HOGWILD! and ALLREDUCE, do not honor these dependences across threads and thus can potentially suffer poor convergence rates and/or poor scalability. This paper proposes SYMSGD, a parallel SGD algorithm that retains the sequential semantics of SGD in expectation. Each thread in this approach learns a local model and a probabilistic model combiner that allows the local models to be combined to produce the same result as what a sequential SGD would have produced, in expectation. This SYMSGD approach is applicable to any linear learner whose update rule is linear. This paper evaluates SYMSGD’s accuracy and performance on 9 datasets on a shared-memory machine shows up-to 13× speedup over our heavily optimized sequential baseline on 16 cores.
1 INTRODUCTION
Stochastic Gradient Descent (SGD) is an effective method for many regression and classification tasks. It is a simple algorithm with few hyper-parameters and its convergence rates are well understood both theoretically and empirically. However, its performance scalability is severely limited by its inherently sequential computation. SGD iteratively processes its input dataset where the computation at each iteration depends on the model parameters learned from the previous iteration.
Current approaches for parallelizing SGD do not honor this inter-step dependence across threads. Each thread learns a local model independently and combine these models in ways that can break sequential behavior. For instance, threads in HOGWILD! Recht et al. (2011) racily update a shared global model without holding any locks. In parameter-server Li et al. (2014a), each thread (or machine) periodically sends its model deltas to a server that applies them to a global model. In ALLREDUCE Agarwal et al. (2014), threads periodically reach a barrier where they compute a weightedaverage of the local models. Although these asynchronous parallel approaches reach the optimal solution eventually, they can produce a model that is potentially different from what a sequential SGD would have produced after processing a certain number of examples. Our experiments indicate that this makes their convergence rate slower than sequential SGD in terms of total number of examples studied. Our experiments show that all these algorithms either do not scale or their accuracy on the same number of examples falls short of a sequential baseline.
To address this problem, this paper presents SYMSGD, a parallel SGD algorithm that seeks to retain its sequential semantics. The key idea is for each thread to generate a sound combiner that allows the local models to be combined into a model that is the same as the sequential model. This paper describes a method for generating sound combiners for a class of SGD algorithms in which the inter-step dependence is linear in the model parameters. This class includes linear regression, linear regression with L2 regularization, and polynomial regression. While logistic regression is not in this class, our experiments show that linear regression performs equally well in classification tasks as logistic regression for the datasets studied in this paper. Also, this approach works even if the SGD
computation is non-linear on the input examples and other parameters such as the learning rate; only the dependence on model parameters has to be linear.
Generating sound combiners can be expensive. SYMSGD uses random projection techniques to reduce this overhead but still retaining sequential semantics in expectation. We call this approach probabilistically sound combiners. Even though SYMSGD is expected to produce the same answer as the sequential SGD, controlling the variance introduced by the random projection requires care — a large variance can result in reduced accuracy. This paper describes the factors that affect this variance and explores the ensuing design trade-offs.
The resulting algorithm is fast, scales well on multiple cores, and achieves the same accuracy as sequential SGD on sparse and dense datasets. When compared to our optimized sequential baseline, SYMSGD achieves a speedup of 3.5× to 13× on 16 cores, with the algorithm performing better for denser datasets. Moreover, the cost of computing combiners can be efficiently amortized in a multiclass regression as a single combiner is sufficient for all of the classes. Finally, SYMSGD (like ALLREDUCE) is deterministic, producing the same result for a given dataset, configuration, and random seed. Determinism greatly simplifies the task of debugging and optimizing learning.
2 SOUND AND PROBABILISTIC MODEL COMBINERS
Stochastic gradient descent (SGD) is a robust method for finding the parameters of a model that minimize a given error function. Figure 1 shows an example of a (convex) error function over two dimensions x and y reaching the minimum at parameter w∗. SGD starts from some, not necessarily optimal, parameter wg (as shown in Figure 1), and repeatedly modifies w by taking a step along the gradient of the error function for a randomly selected example at the currentw. The magnitude of the step is called the learning rate and is usually denoted by α. The gradient computed from one example is not necessarily the true gradient at w. Nevertheless, SGD enjoys robust convergence behavior by moving along the “right” direction over a large number of steps. This is shown pictorially in Figure 1, where SGD processes examples in dataset D1 to reach w1 from wg . Subsequently, SGD starts from w1 and processes a different set D2 to reach wh. There is a clear dependence between the processing ofD1 and the processing ofD2 — the latter starts from w1, which is only determined after processing D1. Our goal is to parallelize SGD despite this dependence.
State of the art parallelization techniques such as HOGWILD! and ALLREDUCE approach this problem by processing D1 and D2 starting from the same model wg (let us assume that there only two processors for now), and respectively reaching w1 and w2. Then, they combine their local models into a global model, but do so in an ad-hoc manner. For instance, ALLREDUCE computes a weighted average of w1 and w2, where the per-feature weights are chosen so as to prefer the processor that has larger update for that feature. This weighted average is depicted pictorially as wa. Similarly, in HOGWILD!, the two processors race to update the global model with their respective local model without any locking. (HOGWILD! performs this udpate after every example, thus the size of D1 and D2 is 1.) Both approaches do not necessarily reach wh, the model that a sequential SGD would have reached on D1 and D2. While SGD is algorithmically robust to errors, such ad-hoc combinations can result in slower convergence or poor performance, as we demonstrate in Section 4.
Sound Combiner: The goal of this paper is to soundly combine local models. Looking at Figure 1, a sound combiner combines local models w1 and w2, respectively generated from datasets D1 and D2, into a global model wh that is guaranteed to be the same as the model achieved by the sequential
SGD processing D1 and then D2. In effect, a sound combiner allows us to parallelize the sequential computation without changing its semantics.
If we look at the second processor, it starts its computation at wg , while in a sequential execution it would have started at w1, the output of the first processor. To obtain sequential semantics, we need to “adjust” its computation from wg to w1. To do so, the second processor performs its computation starting fromwg +∆w, where ∆w is an unknown symbolic vector. This allows the second processor to both compute a local model (resulting from the concrete part) and a sound combiner (resulting from the symbolic part) that accounts for changes in the initial state. Once both processors are done learning, second processor finds wh by setting ∆w to w1 − wg where w1 is computed by the first processor. This parallelization approach of SGD can be extended to multiple processors where all processor produce a local model and a combiner (except for the first processor) and the local models are combined sequentially using the combiners.
When the update to the model parameters is linear in a SGD computation, then the dependence on the unknown ∆w can be concisely represented by a combiner matrix, as formally described in Section 3. Many interesting machine learning algorithms, such as linear regression, linear regression with L2 regularization, and polynomial regression already have linear update to the model parameters (but not necessarily linear on the input example).
Probabilistically Sound Combiner: The main problem with generating a sound combiner is that the combiner matrix has as many rows and columns as the total number of features. Thus, it can be effectively generated only for datasets with modest number of features. Most interesting machine learning problems involve learning over tens of thousands to billions of features, for which maintaining a combiner matrix is clearly not feasible.
We solve this problem through dimensionality reduction. Johnson-Lindenstrauss (JL) lemma Johnson & Lindenstrauss (1984) allows us to project a set of vectors from a high-dimensional space to a random low-dimensional space while preserving distances. We use this property to reduce the size of the combiner matrix without losing the fidelity of the computation — our parallel algorithm produces the same result as the sequential SGD in expectation.
Of course, a randomized SGD algorithm that generates the exact result in expectation is only useful if the resulting variance is small enough to maintain accuracy and the rate of convergence. We observe that for the variance to be small, the combiner matrix should have small singular values. Interestingly, the combiner matrix resulting from SGD is dominated by the diagonal entries as the learning rate has to be small for effective learning. We use this property to perform the JL projection only after subtracting the identity matrix. Also, other factors that control the singular values are the learning rate, number of processors, and the frequency of combining local models. This paper explores this design space and demonstrates the feasibility of efficient parallelization of SGD that retains the convergence properties of sequential SGD while enjoying parallel scalability.
3 PARALLEL SYMSGD ALGORITHM
Consider a training dataset (Xn×f , yn×1), where f is the number of features, n is the number of examples in the dataset, the ith row of matrix X , Xi, represents the features of the ith example, and yi is the dependent value (or label) of that example. A linear model seeks to find a
w∗ = arg min w∈Rf n∑ i=0 Q(Xi · w, yi)
that minimizes an error function Q. For linear regression, Q(Xi · w, yi) = (Xi · w − yi)2. When (Xi, yi) is evident from the context, we will simply refer to the error function as Qi(w).
SGD iteratively finds w∗ by updating the current model w with a gradient of Qr(w) for a randomly selected example r. For the linear regression error function above, this amounts to the update
wi = wi−1 − α∇Qr(wi−1) = wi−1 − α(Xr · wi−1 − yr)XTr (1) Here, α is the learning rate that determines the magnitude of the update along the gradient. As it is clear from this equation, wi is dependent on wi−1 which creates a loop-carried dependence and consequently makes parallelization of SGD across iterations using naı̈ve approaches impossible.
The complexity of SGD for each iteration is as follows. Assume thatXr has z non-zeros. Therefore, the computation in Equation 1 requires O(z) amount of time for the inner product computation, Xr ·wi−1, and the sameO(z) amount of time for scalar-vector multiplication, α(Xr ·wi−1−yr)XTr . If the updates to the weight vector happen in-place meaning thatwi andwi−1 share the same memory location, the computation in Equation 1 takes O(z) amount of time.
3.1 SYMBOLIC STOCHASTIC GRADIENT DESCENT
This section explains a new approach to parallelize the SGD algorithm despite its loop-carried dependence. As shown in Figure 1, the basic idea is to start each processor (except the first) on a concrete model w along with a symbolic unknown ∆w that captures the fact that the starting model can change based on the output of the previous processor. If the dependence on ∆w is linear during an SGD update, which is indeed the case for linear regression, then the symbolic dependence on ∆w on the final output can be captured by an appropriate matrix Ma→b that is a function of the input examples Xa, . . . , Xb processed (ya, . . . , yb do not affect this matrix). Specifically, as Lemma A.1 in the Appendix shows, this combiner matrix is given by
Ma→b = a∏ i=b (I − αXTi ·Xi) (2)
In effect, the combiner matrix above is the symbolic representation of how a ∆w change in the input will affect the output of a processor. Ma→b is referred by M when the inputs are not evident.
The parallel SGD algorithm works as follows (see Figure 1). In the learning phase, each processor i starting from w0, computes both a local model li and a combiner matrix Mi. In a subsequent reduction phase, each processor in turn computes its true output using
wi = li +Mi · (wi−1 − w0) (3) Lemma A.1 ensures that this combination of local models will produce the same output as what these processors would have generated had they run sequentially. We call such combiners sound.
One can compute a sound model combiner for other SGD algorithms provided the loop-carried dependence on w is linear. In other words, there should exist a matrix Ai and vector bi in iteration i such that wi = Ai · wi−1 + bi. Note that Ai and bi can be nonlinear in terms of input datasets.
3.2 DIMENSIONALITY REDUCTION OF A SOUND COMBINER
The combiner matrixM generate above can be quite large and expensive to compute. The sequential SGD algorithm maintains and updates the weight vector w, and thus requires O(f) space and time, where f is the number of features. In contrast, M is a f × f matrix and consequently, the space and time complexity of parallel SGD is O(f2). In practice, this would mean that we would need O(f) processors to see constant speedups, an infeasible proposition particularly for datasets that can have thousands if not millions of features.
SYMSGD resolves this issue by projecting M into a smaller space while maintaining its fidelity. This projection is inspired by the Johnson-Lindenstrauss (JL) lemma Johnson & Lindenstrauss (1984) and follows the treatment of Achlioptas Achlioptas (2001). Lemma 3.1. 1 Let A be a random f × k matrix with
aij = dij/ √ k
where aij is the element of A at the ith row and jth column and dij is independently sampled from a random distribution D with IE[D] = 0 and Var[D] = 1. Then
IE[A ·AT ] = If×f
The matrix A from Lemma 3.1 projects from Rf → Rk where k can be much smaller than f . This allows us to approximate Equation 3 as
wi ≈ li +Mi ·A ·AT (wi−1 − w0) (4) 1See proof in Appendix A.2.
Lemma 3.1 guarantees that the approximation above is unbiased.
IE[li +Mi ·A ·AT (wi−1 − w0)] = li +Mi · IE[A ·AT ](wi−1 − w0) = wi
This allows an efficient algorithm that only computes the projected version of the combiner matrix while still producing the same answer as the sequential algorithm in expectation. We call such combiners probabilistically sound.
Algorithm 1: SYMSGD learning a local model and a model combiner. 1 <vector,matrix,matrix> SymSGD( 2 float α, vector: w0, X1..Xn, 3 scalar: y1..yn) { 4 vector w = w0; 5 matrix A = 1√
k random(D,f,k);
6 matrix MA = A; 7 for i in (1..n) { 8 w = w - α(Xi·w - yi)XiT; 9 MA = MA - α Xi·(XiTMA); }
10 return <w,MA,A>; }
Algorithm 2: SYMSGD combining local models using model combiners. 1 vector SymSGDCombine(vector w0, 2 vector w, vector l, 3 matrix MA, matrix A) { 4 parallel { 5 matrix NA = MA - A; 6 w = l+w-w0+NA·AT(w-w0); 7 } 8 return w; }
Algorithm 1 shows the resulting symbolic SGD learner. The random function in line 5 returns a f × k matrix with elements chosen independently from the random distribution D according to Lemma 3.1. When compared to the sequential SGD, the additional work is the computation of MA in Line 9. It is important to note that this algorithm maintains the invariant that MA = M · A at every step. This projection incurs a space and time overhead of O(z × k) where z is the number of non-zeros in Xi. This overhead is acceptable for small k and infact in our experiments in Section 4, k is between 7 to 15 across all benchmarks. Most of the overhead for such a small k is hidden by utilizing SIMD hardware within a processor (SymSGD with one thread is only half as slow as the sequential SGD as discussed in Section 4.1). After learning a local model and a probabilistically sound combiner in each processor, Algorithm 2 combines the resulting local model using the combiners, but additionally employs the optimizations discussed in Section 3.3.
Note that the correctness and performance of SYMSGD do not depend on the sparsity of a dataset and as Section 4 demonstrates, it works for very sparse and completely dense datasets. Also, note that X1, . . . ,Xn may contain a subset of size f’ of all f features. Our implementation of Algorithm 1 takes advantage of this property and allocates and initializes A for only the observed features. This optimization is omitted from the pseudo code in Algorithm 1 for the sake of simplicity.
3.3 CONTROLLING THE VARIANCE
While the dimensionality reduction discussed above is expected to produce the right answer, this is useful only if the variance of the approximation is acceptably small. Computing the variance is involved and is discussed in the associated technical report SymSGDTR. But we discuss the main result that motivates the rest of the paper.
Consider the approximation ofM ·∆w with v = M ·A ·AT ·∆w. Let C(v) be the covariance matrix of v. The trace of the covariance matrix tr(C(v)) is the sum of the variance of individual elements of v. Let λi(M) by the ith eigenvalue of M and σi(M) = √ λi(MTM) the ith singular value of M . Let σmax(M) be the maximum singular value of M . Then the following holds SymSGDTR:
‖∆w‖22 k ∑ i σ2i (M) ≤ tr(C(v)) ≤ ‖∆w‖22 k ( ∑ i σ2i (M) + σ 2 max(M))
The covariance is small if k, the dimension of the projected space, is large. But increasing k proportionally increases the overhead of the parallel algorithm. Similarly, covariance is small if the projection happens on small ∆w. Looking at Equation 4, this means that wi−1 should be as close to w0 as possible, implying that processors should communicate frequently enough such that their
models are roughly in sync. Finally, the singular values of M should be as small as possible. The next section describes a crucial optimization that achieves this.
Taking Identity Off: Expanding Equation 2, we see that the combiner matrices are of the form
I − αR1 + α2R2 − α3R3 + · · ·
where Ri matrices are formed from the sum of products of Xj · XTj matrices. Since α is a small number, this sum is dominated by I . In fact, for a combiner matrix M generated from n examples, M − I has at most n non-zero singular values SymSGDTR. We use these observation to lower the variance of dimensionality reduction by projecting matrix N = M − I instead of M . Appendix A.3 empirically shows the impact of this optimization. Rewriting Equations 3 and 4, we have
wi = li + (Ni + I) · (wi−1 − w0) = li + wi−1 − w0 +Ni · (wi−1 − w0) ≈ li + wi−1 − w0 +Ni ·A ·AT · (wi−1 − w0) (5)
Lemma 3.1 guarantees that the approximation above is unbiased. Algorithm 2 shows the pseudo code for the resulting probabilistically sound combination of local models. The function SymSGDCombine is called iteratively to combine the model of the first processor with the local models of the rest. Note that each model combination is executed in parallel (Line 4) by parallelizing the underlying linear algebra operations.
An important factor in controlling the singular values of N is the frequency of model combinations which is a tunable parameter in SYMSGD. As it is shown in Appendix A.3, the fewer the number of examples learned, the smaller the singular values of N and the less variance (error) in Equation 5.
Implementation For the implementation of SymSGD function, matrix M and weight vector w are stored next to each other. This enables better utilization of vector units in the processor and improves the performance of our approach significantly. Also, most of datasets are sparse and therefore, SGD and SymSGD only copy the observed features from w0 to their learning model w. Moreover, for the implementation of matrix A, we used Achlioptas (2001) theorem to minimize the overhead of creating A. In this approach, each element of A is independently chosen from { 13 ,− 1 3 , 0} with probability { 16 , 1 6 , 2 3}, respectively.
4 EVALUATION
All experiments described in this section were performed on an Intel Xeon E5-2630 v3 machine clocked at 2.4 GHz with 256 GB of RAM. The machine has two sockets with 8 cores each, allowing us to study the scalability of the algorithms across sockets. We disabled hyper-threading and turbo boost. We also explicitly pinned threads to cores in a compact way which means that thread i + 1 was placed as close as possible to thread i. The machine runs Windows 10. All of our implementations were compiled with Intel C/C++ compiler 16.0 and relied heavily on OpenMP primitives for parallelization and MKL for efficient linear algebra computations. And, finally, to measure runtime, we use the average of five independent runs on an otherwise idle machine.
There are several algorithms and implementations that we used for our comparison: Vowpal Wabbit Langford et al. (2007), a widely used public library, Baseline, a fast sequential implementation, HW-Paper, the implementation from Recht et al. (2011), HW-Release, an updated version, HogWild, which runs Baseline in multiple threads without any synchronization, and ALLREDUCE, the implementation from Agarwal et al. (2014). Each of these algorithms have different parameters and settings and we slightly modified to ensure a fair comparison; see Appendix A.4 for more details.
When studying the scalability of a parallel algorithm, it is important to compare the algorithms against an efficient baseline Bailey (1991); McSherry et al. (2015). Otherwise, it is empirically not possible to differentiate between the scalability achieved from the parallelization of the inefficiencies and the scalability inherent in the algorithm. We spent a significant effort to implement a well-tuned sequential algorithm which we call Baseline in our comparisons. Baseline is between 1.97 to 7.62 (3.64 on average) times faster than Vowpal Wabbit and it is used for all speedup graphs in this paper.
Datasets Table 1 describes the datasets used for evaluation. The number of features, training instances, test instances, classes and the sparsity of each dataset is shown in Table 1. We used Vowpal
Wabbit with the configurations discussed in Appendix A.4 to measure the maximum accuracies that can be achieved using linear and logistic regression and the result is presented in columns 8 and 9 of Table 1. In the case of aloi dataset, even after 500 passes (the default for our evaluation was 100 passes) the accuracies did not saturate to the maximum possible and we reported that both linear and logistic achieved at least 80% accuracy. The last two columns show the maximum speedup of SYMSGD and HOGWILD! over the baseline.
Parameters Hyper-parameters searching is essential for performance and accuracy. The learning rate, α, for each dataset was selected by searching for a constant value among {.5, .05, .005, . . . } where Baseline reached close to maximum accuracy for each benchmark. The parameters for the projection size, k, and the frequency of model combination were searched to pick the best performing configuration. The parameters for ALLREDUCE were similarly searched.
4.1 RESULTS
Figure 2 shows the accuracy and speedup measurements on three benchmarks: rcv1.binary, a sparse binary dataset, rcv1.multiclass, a sparse multiclass dataset, and epsilon, a dense binary dataset. The results for the other six benchmarks are presented in Appendix A.5.
Sparse Binary, rcv1.binary: Figure 2a compares the scalability of all the algorithms studied in this paper. HW-Paper is around six times slower than HW-Release. While this could potentially be a result of us running HW-Release on a Ubuntu VM, our primary aim of this comparison was to ensure that HogWild is a competitive implementation of HOGWILD!. Thus, we remove HW-Paper and HW-Release in our subsequent comparisons.
SYMSGD is half as slow as the Baseline on one thread as it performs lot more computation, but scales to a 3.5× speedup to 16 cores. Note, this represents a roughly 7× strong-scaling speedup with respect to its own performance on one thread. Analysis of the hardware performance counters shows the current limit to SYMSGD’s scalability arises from load-imbalance across barrier synchronization, which provides an opportunity for future improvement.
Figure 2d shows the accuracy as a function of the number of examples processed by different algorithms. SYMSGD “stutters” at the beginning, but it too matches the accuracy of Baseline. The initial stuttering happens because the magnitude of the local models on each processor are large during the first set of examples. This directly affect the variance of the combiner matrix approximation. However, as more examples are given to SYMSGD, the magnitude of the local models are smaller and thus SYMSGD better matches the Baseline accuracy. One way to avoid this stuttering is to combine models more frequently (lower variance) or running single threaded for the first few iterations.
HogWild does approach sequential accuracy, however, it does so at the cost of scalablity (i.e., see Figure 2a (a)). Likewise, ALLREDUCE scales slightly better but does so at the cost of accuracy.
Sparse Multiclass, rcv1.multiclass: Figure 2b shows the scalability on rcv1.multiclass. Since this is a multiclass dataset, SYMSGD is competitive with the baseline on one thread as it is able to amortize the combiner matrix computation across all of the classes (M is the same across different classes). Thus, it enjoys much better scalability of 7× when compared to rcv1.binary. HogWild scales similar to SYMSGD up-to 8 threads but suffers when 16 threads across multiple sockets are used. Figure 2e shows that SYMSGD meets the sequential accuracy after an initial stutter. ALLREDUCE suffers from accuracy.
Dense Binary, epsilon: Figure 2c in Appendix A.5 shows that SYMSGD achieves a 7× speedup over the baseline on 16 cores. This represents a 14× strong scaling speedup over SYMSGD on one thread. As HOGWILD! is not designed for dense workloads, its speedup suffers when 16 cores across multiple sockets are used. This shows that SYMSGD scales to both sparse and dense datasets. Similarly, ALLREDUCE suffers from accuracy.
5 RELATED WORK
Most schemes for parallelizing SGD learn local models independently and communicate to update the global model. The algorithms differ in how and how often the update is performed. These choices determine the applicability of the algorithm to shared-memory or distributed systems.
To the best of our knowledge, our approach is the only one that retain the semantics of the sequential SGD algorithm. While some prior work provides theoretical analysis of the convergence rates that justify a specific parallelization, convergence properties of SYMSGD simply follow from the sequential SGD algorithm. On the other hand, SYMSGD is currently restricted to class of SGD computations where the inter-step dependence is linear in the model parameters.
Given a tight coupling of the processing units, Langford et al. Langford et al. (2009) suggest on a round-robin scheme to update the global model allowing for some staleness. However, as the SGD computation per example is usually much smaller when compared to the locking overhead, HOGWILD! Recht et al. (2011) improves on this approach to perform the update in a “racy” manner. While HOGWILD! is theoretically proven to achieve good convergence rates provided the dataset is sparse enough and the processors update the global model fast enough, our experiments show that the generated cache-coherence traffic limits its scalability particularly across multiple sockets. Moreover, as HOGWILD! does not update the model atomically, it potentially loses correlation among more frequent features resulting in loss of accuracy. Lastly, unlike SYMSGD, which works for both sparse and dense datasets, HOGWILD! is expclitly designed for sparse data. Recently, Sallinen et al. (2016) proposed applying lock-free HOGWILD! approach to mini-batch. However, mini-batch converges slower than SGD and also they did not study multi-socket scaling.
Zinkevich et al. Zinkevich et al. (2010) propose a MapReduce-friendly framework for SGD. The basic idea is for each machine/thread to run a sequential SGD on its local data. At the end, the global model is obtained by averaging these local models. Alekh et al. Agarwal et al. (2014) extend this approach by using MPI AllReduce operation. Additionally, they use the adagrad Duchi et al. (2011) approach for the learning rates at each node and use weighted averaging to combine local models
with processors that processed a feature more frequently having a larger weight. Our experiments on our datasets and implementation shows that it does not achieve the sequential accuracy.
Several distributed frameworks for machine learning are based on parameter server Li et al. (2014b;a) where clients perform local learning and periodically send the changes to a central parameter server that applies the changes. For additional parallelism, the models themselves can be split across multiple servers and clients only contact a subset of the servers to perform their updates.
6 CONCLUSION
With terabytes of memory available on multicore machines today, our current implementation has the capability of learning from large datasets without incurring the communication overheads of a distributed system. That said, we believe the ideas in this paper apply to distributed SGD algorithms and how to pursue in future work.
Many machine learning SGD algorithms require a nonlinear dependence on the parameter models. While SYMSGD does not directly apply to such algorithms, it is an interesting open problem to devise linear approximations (say using Taylor expansion) to these problems and subsequently parallelize with probabilistically sound combiners. This is an interesting study for future work.
A APPENDIX
A.1 COMBINER MATRIX
Lemma A.1. If the SGD algorithm for linear regression processes examples (Xa, ya), (Xa+1, ya+1), . . . , (Xb, yb) starting from model ws to obtain wb, then its outcome starting on model ws + ∆w is given by wb + Ma→b · ∆w where the combiner matrix Ma→b is given by
Ma→b = a∏ i=b (I − αXTi ·Xi)
Proof. The proof follows from a simple induction. Starting from ws, let the models computed by SGD after processing (Xa, ya), (Xa+1, ya+1), . . . , (Xb, yb) respectively be wa, wa+1, . . . wb. Consider the base case of processing of (Xa, ya). Starting from ws + ∆w, SGD computes the model w′a using Equation 1 (reminder: wi = wi−1 − α(Xi · wi−1 − yi)XTi ):
w′a = ws + ∆w − α(Xa · (ws + ∆w)− ya)XTa = ws + ∆w − α(Xa · ws − ya)XTa − α(Xa ·∆w)XTa = ws − α(Xa · ws − ya)XTa + ∆w − α(Xa ·∆w)XTa = wa + ∆w − α(Xa ·∆w)XTa (6) = wa + ∆w − αXTa (Xa ·∆w) (7) = wa + ∆w − α(XTa ·Xa) ·∆w (8) = wa + (I − αXTa ·Xa) ·∆w
Step 6 uses Equation 1, Step 7 uses the fact that Xa ·∆w is a scalar (allowing it to be rearranged), and Step 8 follows from associativity property of matrix multiplication.
The induction is very similar and follows from replacing ∆w with Ma→i−1∆w and the property that
Ma→i = (I − αXTi ·Xi) ·Ma→i−1
A.2 PROOF OF LEMMA 3.1 Proof. Let’s call B = A · AT . Then bij , the element of B at row i and column j, is ∑
s aisajs. Therefore, IE[bij ] = ∑k s=1 IE[aisajs] = ( 1√ k )2 ∑k s=1 IE[disdjs] = 1 k ∑k s=1 IE[disdjs]. For i 6= j, IE[bij ] = 1 k ∑k s=1 IE[dis]IE[djs] because dij are chosen independently. Since IE[D] = 0 and dis, djs ∈ D, IE[dis] = IE[djs] = 0 and consequently, IE[bij ] = 0. For i = j, IE[bii] = 1 k ∑ s IE[disdis] = 1 k ∑ s IE[d 2 is]. Since IE[D
2] = 1 and dis ∈ D, IE[d2is] = 1. As a result, IE[bii] = 1 k ∑k s=1 IE[d 2 is] = 1 k ∑k s=1 1 = 1.
A.3 EMPIRICAL EVALUATING SINGULAR VALUES OF M
Figure 3 empirically demonstrates the benefit of taking identity off. This figure plots the singular values of M for the rcv1.binary dataset (described in Section 4) after processing 64, 128, 192, 256 examples for two different learning rates. As it can be seen, the singular values are close to 1. However, the singular values of N = M − I are roughly the same as those of M minus 1 and consequently, are small. Finally, the smaller α, the closer the singular values of M are to 1 and the singular values of N are to 0. Also, note that the singular values of M decrease as the numbers of examples increase and therefore, the singular values of N increase. As a result, the more frequent the models are combined, the less variance (and error) is introduced into Equation 5.
A.4 ALGORITHM DETAILS AND SETTINGS
This section provides details of all algorithms we used in this paper. Each algorithm required slight modification to ensure fair comparison.
Vowpal Wabbit: Vowpal Wabbit Langford et al. (2007) is one of the widely used public libraries for machine learning algorithms. We used this application as a baseline for accuracy of different datasets and as a comparison of logistic and linear regression and also an independent validation of the learners without any of our implementation bias. Vowpal Wabbit applies accuracy-improving optimizations such as adaptive and individual learning steps or per feature normalize updates. While all of these optimizations are applicable to SYMSGD, we avoided them since the focus of this paper is the running time performance of our learner. The non-default flags that we used are: --sgd, --power t 0, --holdout off, --oaa nc for multiclass datasets where nc is the number of classes, --loss function func where func is squared or logistic. For learning rate, we searched for α, the learning rate, in the set of {.1, .5, .01, .05, .001, .005, . . . } and used --learning rate α. We went through dataset 100 times for each dataset (--passes 100) and saved the learned model after each pass (--save per pass). At the end, for linear and logistic regressions, we reported the maximum accuracies achieved among different passes and different learning rates.
Baseline: Baseline uses a mixture of MKL Intel and manually vectorized implementations of linear algebra primitives in order to deliver the fastest performance. Baseline processes up-to 3.20 billion features per second at 6.4 GFLOPS.
HOGWILD!: HOGWILD! Recht et al. (2011) is a lock-free approach to parallelize SGD where multiple thread apply Equation 1 simultaneously. Although this approach may have race condition
when two threads process instances with a shared feature but the authors discuss that this does not hurt the accuracy significantly for sparse datasets. There are multiple implementations of this approach that we studied and evaluated in this section. Below is a description of each:
• HW-Paper: This is the implementation used to report the measurements in Recht et al. (2011) which is publicly available Hogwild. This code implements SVM algorithm. Therefore, we modified the update rule to linear regression. The modified code was compiled and run on our Windows machine described above using an Ubuntu VM since the code is configured for Linux systems.
• HW-Release: This is an optimized implementation that the authors built after the HOGWILD! paper Recht et al. (2011) was published. Similar to HW-Paper, we changed the update rule accordingly and executed it on the VM.
• HogWild: We implemented this version which runs Baseline by multiple threads without any synchronizations. This code runs natively on Windows and enjoys all the optimizations applied to our Baseline such as call to MKL library and manual vectorization of linear algebra primitives.
ALLREDUCE: ALLREDUCE Agarwal et al. (2014) is an approach where each thread makes a copy from the global model and applies the SGD update rule to the local model for certain number of instances. Along with the local model, another vector g is computed which indicates the confidence in an update for the weight of a feature in the local model. After the learning phase, the local weight vectors are averaged based on the confidence vectors from each thread. We implemented this approach similarly using MKL calls and manual vectorization.
A.5 SPEEUPS ON REMAINING BENCHMARKS

This paper propose a parallel mechanism for stochastic gradient descent method (SGD) in case of gradient can be computed via linear operations (including least square linear regression and polynomial regression problems). The motivation is to recover the same effect compared with sequential SGD, by using a proposed sound combiner. To make such combiner more efficient, the authors also use a randomized projection matrix to do dimension reduction. Experiments shows the proposed method has better speedup than previous methods like Hogwild! and Allreduce.
I feel that there might be some fundamental misunderstanding on SGD.
''The combiner matrixM  generate above can be quite large and expensive to compute. The sequential SGD algorithm maintains and updates the weight vector w , and thus requires O(f)  space and time, where f  is the number of features. In contrast,M  is a f f  matrix and consequently, the space and time complexity of parallel SGD is O(f^2) . In practice, this would mean that we would need O(f) processors to see constant speedups, an infeasible proposition particularly for datasets that can have
thousands if not millions of features."
I do not think one needs O(f^2) space and complexity for updating M_i * v, where v is an f-dimensional vector. Note that M_i is a low rank matrix in the form of (I - a_i a_i'). The complexity and space can be reduced to O(f) if compute it by O(v - a_i (a_i' v)) equivalently. If M_i is defined in the form of the product of n number of rank 1 matrices. The complexity and space complexity is O(fn). In the context of this paper, n should be much smaller than f. I seriously doubt that all author's assumptions, experiments, and strategies in this paper are based on this incorrect assumption on space and complexity of SGD.
Why one can have speedup is unclear for me. It is unclear what computations are in parallel and why this sequential algorithms can bring speedup if M_i*v is computed in the most efficient way.
I suggest authors to make the following changes to make this paper more clear and theoretically solid
- provide computational complexity per step of the proposed algorithm
- convergence rate analysis (convergence analysis is not enough): we would like to see how the dimension reduction can affect the complexity.


LEARNING GRAPHICAL STATE TRANSITIONS
Graph-structured data is important in modeling relationships between multiple entities, and can be used to represent states of the world as well as many data structures. Li et al. (2016) describe a model known as a Gated Graph Sequence Neural Network (GGS-NN) that produces sequences from graph-structured input. In this work I introduce the Gated Graph Transformer Neural Network (GGTNN), an extension of GGS-NNs that uses graph-structured data as an intermediate representation. The model can learn to construct and modify graphs in sophisticated ways based on textual input, and also to use the graphs to produce a variety of outputs. For example, the model successfully learns to solve almost all of the bAbI tasks (Weston et al., 2016), and also discovers the rules governing graphical formulations of a simple cellular automaton and a family of Turing machines.
1 INTRODUCTION
Many different types of data can be formulated using a graph structure. One form of data that lends itself to a graphical representation is data involving relationships (edges) between entities (nodes). Abstract maps of places and paths between them also have a natural graph representation, where places are nodes and paths are edges. In addition, many data structures can be expressed in graphical form, including linked lists and binary trees.
Substantial research has been done on producing output when given graph-structured input (Kashima et al., 2003; Shervashidze et al., 2011; Perozzi et al., 2014; Bruna et al., 2013; Duvenaud et al., 2015). Of particular relevance to this work are Graph Neural Networks (Gori et al., 2005; Scarselli et al., 2009), or GNNs, which extend recursive neural networks by assigning states to each node in a graph based on the states of adjacent nodes. Recently Li et al. (2016) have modified GNNs to use gated state updates and to produce output sequences. The resulting networks, called GG-NNs and GGS-NNs, are successful at solving a variety of tasks with graph-structured input.
The current work further builds upon GG-NNs and GGS-NNs by allowing graph-structured intermediate representations, as well as graph-structured outputs. This is accomplished using a more flexible graph definition, along with a set of graph transformations which take a graph and other information as input and produce a modified version of the graph. This work also introduces the Gated Graph Transformer Neural Network model (GGT-NN), which combines these transformations with a recurrent input model to incrementally construct a graph given natural language input, and can either produce a final graph representing its current state, or use the graph to produce a natural language output.
Extending GG-NNs in this way opens up a wide variety of applications. Since many types of data can be naturally expressed as a graph, it is possible to train a GGT-NN model to manipulate a meaningful graphical internal state. In this paper I demonstrate the GGT-NN model on the bAbI task dataset, which contains a set of stories about the state of the world. By encoding this state as a graph and providing these graphs to the model at training time, a GGT-NN model can be trained to construct the correct graph from the input sentences and then answer questions based on this internal graph. I also demonstrate that this architecture can learn complex update rules by training it to model a simple 1D cellular automaton and arbitrary 4-state Turing machines. This requires the network to learn how to transform its internal state based on the rules of each task.
2 BACKGROUND
2.1 GRU
Gated Recurrent Units (GRU) are a type of recurrent network cell introduced by Cho et al. (2014). Each unit uses a reset gate r and an update gate z, and updates according to
r(t) = σ ( Wrx (t) + Urh (t−1) + br ) z(t) = σ ( Wzx (t) + Uzh (t−1) + bz ) h̃(t) = φ ( Wx + U ( r(t) h(t−1) ) + b ) h(t) = z h(t−1) + (1− z) h̃(t)
where σ is the logistic sigmoid function, φ is an activation function (here tanh is used), x(t) is the input vector at timestep t, h(t) is the hidden output vector at timestep t, and W,U,Wr,Ur,Wz, Uz , b, br and bz are learned weights and biases. Note that denotes elementwise multiplication.
2.2 GG-NN AND GGS-NN
The Gated Graph Neural Network (GG-NN) is a form of graphical neural network model described by Li et al. (2016). In a GG-NN, a graph G = (V, E) consists of a set V of nodes v with unique values and a set E of directed edges e = (v, v′) ∈ V × V oriented from v to v′. Each node has an annotation xv ∈ RN and a hidden state hv ∈ RD, and each edge has a type ye ∈ {1, · · · ,M}. GG-NNs operate by first initializing the state hv of each node to correspond to the annotation xv . Then, a series of propagation steps occur. In each step, information is transferred between nodes across the edges, and the types of edge determine what information is sent. Each node sums the input it receives from all adjacent nodes, and uses that to update its own internal state, in the same manner as a GRU cell. Finally, the states of all nodes are used either to create a graph-level aggregate output, or to classify each individual node.
GGS-NNs extend GG-NNs by performing a large number of propagation-output cycles. At each stage, two versions of the GG-NN propagation process are run. The first is used to predict an output for that timestep, and the second is used to update the annotations of the nodes for the next timestep. This allows GGS-NNs to predict a sequence of outputs from a single graph.
3 DIFFERENTIABLE GRAPH TRANSFORMATIONS
In this section, I describe some modifications to the graph structure to make it fully differentiable, and then propose a set of transformations which can be applied to a graph structure in order to transform it. In particular, I redefine a graph G = (V, C) ∈ Γ as a set V of nodes v, and a connectivity matrix C ∈ R|V|×|V|×Y , where Y is the number of possible edge types. As before, each node has an annotation xv ∈ RN and a hidden state hv ∈ RD. However, there is an additional constraint that∑N j=1 xv,j = 1. One can then interpret xv,j as the level of belief that v should have type j out of N possible node types. Each node also has a strength sv ∈ [0, 1]. This represents the level of belief that node v should exist, where sv = 1 means the node exists, and sv = 0 indicates that the node should not exist and thus should be ignored.
Similarly, elements of C are constrained to the range [0, 1], and thus one can interpret Cv,v′,y as the level of belief that there should be a directed edge of type y from v to v′. (Note that it is possible for there to be edges of multiple types between the same two nodes v and v′, i.e. it is possible for Cv,v′,y = Cv,v′,y′ = 1 where y 6= y′.) Figure 1 shows the values of xv , sv , hv , and C corresponding to a particular graphical structure.
There are five classes of graph transformation:
a) Node addition (Tadd), which modifies a graph by adding new nodes and assigning them annotations xv and strengths sv based on an input vector. b) Node state update (Th), which modifies the internal state of each node using an input vector (similar to a GRU update step). Optionally, different input can be given to nodes of each type, based on direct textual references to specific node types. This version is called a direct reference update (Th,direct). c) Edge update (TC), which modifies the edges between each pair of nodes based on the internal states of the two nodes and an external input vector. d) Propagation (Tprop), which allows nodes to trade information across the existing edges and then update their internal states based on the information received. e) Aggregation (Trepr), which uses an attention mechanism to select relevant nodes and then generates a graph-level output.
Each transformation has its own trainable parameters. Together, these transformations can be combined to process a graph in complex ways. An overview of these operations is shown in Figure 2. For details about the implementation of each of these transformations, see Appendix B.
4 GATED GRAPH TRANSFORMER NEURAL NETWORK (GGT-NN)
In this section I introduce the Gated Graph Transformer Neural Network (GGT-NN), which is constructed by combining a series of these transformations. Depending on the configuration of the transformations, a GGT-NN can take textual or graph-structured input, and produce textual or graph-
Algorithm 1 Graph Transformation Pseudocode
1: G ← ∅ 2: for k from 1 to K do 3: G ← Th(G, i(k)) 4: if direct reference enabled then 5: G ← Th,direct(G,D(k)) 6: end if 7: if intermediate propagation enabled then 8: G ← Tprop(G) 9: end if
10: haddG ← Trepr(G)
11: G ← Tadd(G, [i(k) haddG ]) 12: G ← TC(G, i(k)) 13: end for 14: G ← T queryh (G, i query) 15: if direct reference enabled then 16: G ← T queryh,direct(G,D query) 17: end if 18: G ← T queryprop (G) 19: hanswerG ← T query repr (G) 20: return foutput(hanswerG )
structured output. Here I describe one particular GGT-NN configuration, designed to build and modify a graph based on a sequence of input sentences, and then produce an answer to a query.
When run, the model performs the following: For each sentence k, each word is converted to a one-hot vector w(k)l , and the sequence of words (of length L) is passed through a GRU layer to produce a sequence of partial-sentence representation vectors p(k)l . The full sentence representation vector i(k) is initialized to the last partial representation vector p(k)L . Furthermore, a direct-reference input matrix D(k) is set to the sum of partial representation vectors corresponding to the words that directly reference a node type, i.e. D(k)n = ∑ l∈Rn p (k) l where Rn is the set of words in sentence k that directly refer to node type n. This acts like an attention mechanism, by accumulating the partial representation vectors for the words that directly reference each type, and masking out the vectors corresponding to other words.
Next, a series of graph transformations are applied, as depicted in Algorithm 1. Depending on the task, direct reference updates and per-sentence propagation can be enabled or disabled. The output function foutput will depend on the specific type of answer desired. If the answer is a single word, foutput can be a multilayer perceptron followed by a softmax operation. If the answer is a sequence of words, foutput can use a recurrent network (such as a GRU) to produce a sequence of outputs. Note that transformations with different superscripts (Th and T queryh , for instance) refer to similar transformations with different learned weights.
Since the processing of the input and all of the graph transformations are differentiable, at this point the network output can be compared with the correct output for that query and used to update the network parameters, including both the GRU parameters used when processing the input and the internal weights associated with each transformation.
4.1 SUPERVISION
As with many supervised models, one can evaluate the loss based on the likelihood of producing an incorrect answer, and then minimize the loss by backpropagation. However, based on initial experiments, the model appeared to require additional supervision to extract meaningful graphstructured data. To provide this additional supervision, I found it beneficial to provide the correct graph at each timestep and train the network to produce that graph. This occurs in two stages, first when new nodes are proposed, and then when edges are adjusted. For the edge adjustment, the edge loss between a correct edge matrix C∗ and the computed edge matrix C is given by
Ledge = −C∗ · ln(C)− (1− C∗) · ln(1− C).
The node adjustment is slightly more complex. Multiple nodes are added in each timestep, but the order of those nodes is arbitrary, and only their existence is important. Thus it should be possible for the network to determine the optimal ordering of the nodes. In fact, this is important because there is no guarantee that the nodes will be ordered consistently in the training data.
Vinyals et al. (2016) demonstrate a simple method for training a network to output unordered sets: the network produces a sequence of outputs, and these outputs are compared with the closest order-
ing of the training data, i.e., the ordering of the training data which would produce the smallest loss when compared with the network output. Vinyals et al. show that when using this method, the network arbitrarily chooses an ordering which may not be the optimal ordering for the task. However, in this case any ordering should be sufficient, and I found the arbitrary orderings selected in this way to work well in practice. In particular, letting s∗π(v) and x ∗ π(v) denote the correct strength and annotations of node v under ordering π, the loss becomes
Lnode = −max π |Vnew|∑ v=|Vold|+1 s∗π(v) ln(sv) + (1− s ∗ π(v)) ln(1− sv) + x ∗ π(v) · ln(xv).
At this point the correct values C∗, s∗π(v) and x ∗ π(v) are substituted into the graph for further processing. Note that only the edges and the new nodes are replaced by the supervision. The hidden states of all existing nodes are propagated without adjustment.
4.2 OTHER TRANSFORMATION CONFIGURATIONS
The structure described in Algorithm 1 is designed for question-answering tasks. However, due to the composability of the individual graph transformations, other configurations could be used to solve other tasks that operate on structured data.
For instance, if a task consists of tracking relationships between a fixed set of objects, one could construct a version of the model that does not use the new-nodes transformation (Tadd), but instead only modifies edges. If the task was to extract information from an existing graph, a structure similar to the GGS-NNs could be built by using only the propagation and aggregation transformations. If the task was to construct a graph based on textual input, the query processing steps could be omitted, and instead the final graph could be returned for processing. And if information should be gathered from a sequence of graphs instead of from a single graph, the query processing steps could be modified to run in parallel on the full sequence of graphs and extract information from each graph. This last modification is demonstrated in Appendix D.
5 EXPERIMENTS
5.1 BABI TASKS
I evaluated the GGT-NN model on the bAbI tasks, a set of simple natural-language tasks, where each task is structured as a sequence of sentences followed by a query (Weston et al., 2016). The generation procedure for the bAbI tasks includes a “Knowledge” object for each sentence, representing the current state of knowledge after that sentence. I exposed this knowledge object in graph format, and used this to train a GGT-NN in supervised mode. The knowledge object provides names for each node type, and direct reference was performed based on these names: if a word in the sentence matched a node type name, it was parsed as a direct reference to all nodes of that type. For details on this graphical format, see Appendix C.
5.1.1 ANALYSIS AND RESULTS
I trained two versions of the GGT-NN model for each task: one with and one without direct reference. Tasks 3 and 5, which involve a complex temporal component, were trained with intermediate propagation, whereas all of the other tasks were not because the structure of the tasks made such complexity unnecessary. Most task models were configured to output a single word, but task 19 (pathfinding) used a GRU to output multiple words, and task 8 (listing) was configured to output a strength for each possible word to allow multiple words to be selected without having to consider ordering.
Results are shown in Tables 1 and 2. The GGT-NN model was able to reach 95% accuracy in all but one of the tasks, and reached 100% accuracy in eleven of them (see Table 2). Additionally, for fourteen of the tasks, the model was able to reach 95% accuracy using 500 or fewer of the 1000 training examples (see Table 1).
The only task that the GGT-NN was unable to solve with 95% accuracy was task 17 (Positional Reasoning), for which the model was not able to attain a high accuracy. Task 17 has a larger number
of possible entities than the other tasks: each entity consists of a color (chosen from five options) and a shape (chosen from four shapes), for a total of 20 unique entities that must be represented separately. Additionally, the stories are much shorter than those in other tasks (2 facts for each set of 8 questions). It is likely that these additional complexities caused the network performance to suffer.
For comparison, accuracy on the bAbI tasks is also included for a simple sequence-to-sequence LSTM model and for a variety of existing state-of-the-art approaches (see Table 2): a simple sequence-to-sequence LSTM model, as implemented in Weston et al. (2016), a modified Memory Network model (MemNN, Weston et al., 2016), End-To-End Memory Network (MemN2N, Sukhbaatar et al., 2015), Recurrent Entity Network (EntNet, Henaff et al., 2016), Neural Turing Machine (NTM, Graves et al., 2014), Dynamic NTM (D-NTM, Gulcehre et al., 2016), a larger version of the MemN2N model with weight tying and nonlinearity (MemN2N*, Sukhbaatar et al., 2015), Differentiable Neural Computer (DNC, Graves et al., 2016), and Dynamic Memory Network (DMN+, Xiong et al., 2016). Although the GGT-NN model was trained using only 1,000 training examples, results using 10,000 examples have also been reproduced here for comparison. Also, it is important to note that the GGT-NN and MemNN models were trained with strong supervision: the GGT-NN model was trained with full graph information, and the MemNN model was trained with information on which sentences were relevant to the query. All other models were trained end-to-end without additional supervision.
Since the GGT-NN and MemNN models are both strongly supervised, it is interesting to note that each approach outperforms the other on a subset of the tasks. In particular, the GGT-NN model with direct reference attains a higher level of accuracy on the following tasks, with an improvement of 0.4-64% depending on the task: task 5 (0.4%), task 7 (15%), task 8 (9%), task 17 (0.5%), task 18 (2.9%), and task 19 (64%). This may indicate that a graphical representation is superior to a list of sentence memories for solving these tasks. On the other hand, the MemNN model outperforms the GGT-NN model (0.1-2.9% greater accuracy) on tasks 3, 4, 10, 12, 14, and 15.
Of particular interest is the performance on task 19, the pathfinding task, for which the GGT-NN model with direct reference performs better than all but one of the other models (DMN+), and shows a large improvement over the performance of the MemNN model. This is reasonable, since pathfinding is a task that is naturally suited to a graphical representation. The shortest path between two nodes can be easily found by sending information across all paths away from one of the nodes in a distributed fashion, which the GGT-NN model allows. Note that the preexisting GGS-NN model (discussed in Section 2.2) was also able to successfully learn the pathfinding task, but required the input to be preprocessed into graphical form even when evaluating the model, and thus could not be directly evaluated on the textual form of any of the bAbI tasks (Li et al., 2016). The current results demonstrate that the proposed GGT-NN model is able to solve the pathfinding task when given textual input.
Similarly, both variants of the GGT-NN model show improvement over many other models on task 16, the induction task. Solving the induction task requires being able to infer relationships based on similarities between entities. (One example from this task: Lily is a swan. Lily is white. Bernhard is green. Greg is a swan. What color is Greg? A:white.) In a graphical setting, this can be done by following a sequence of edges (Greg → swan → Lily → white), and the performance of the GGT-NN model indicates that this task is particularly suited to such a representation.
In general, the GGT-NN model with direct reference performs better than the model without it. The model with direct reference reaches 95% accuracy on 19/20 of the bAbI tasks, while the model without direct reference reaches that level of accuracy on 9/20 of the tasks (see Table 2). Additionally, when compared to the direct-reference model, the model without direct reference requires more training examples in order to reach the accuracy threshold (see Table 1). This indicates that, although the model can be used without direct reference, adding direct reference greatly improves the training of the model.
5.2 RULE DISCOVERY TASKS
To demonstrate the power of GGT-NN to model a wide variety of graph-based problems, I applied the GGT-NN to two additional tasks. In each task, a sequence of data structures were transformed into a graphical format, and the GGT-NN was tasked with predicting the data for the next timestep
based on the current timestep. No additional information was provided as textual input; instead, the network was tasked with learning the rules governing the evolution of the graph structure over time.
5.2.1 CELLULAR AUTOMATON TASK
The first task used was a 1-dimensional cellular automaton, specifically the binary cellular automaton known as Rule 30 (Wolfram, 2002). Rule 30 acts on an infinite set of cells, each with a binary state (either 0 or 1). At each timestep, each cell deterministically changes state based on its previous state and the states of its neighbors. In particular, the update rules are
Current neighborhood 111 110 101 100 011 010 001 000 Next value 0 0 0 1 1 1 1 0
Cell states can be converted into graphical format by treating the cells as a linked list. Each of the cells is represented by a node with edges connecting it to the cell’s neighbors, and a value edge is used to indicate whether the cell is 0 or 1. This format is described in more detail in Appendix C.
5.2.2 TURING MACHINES
The second task was simulating an arbitrary 2-symbol 4-state Turing machine. A Turing machine operates on an infinite tape of cells, each containing a symbol from a finite set of possible symbols. It has a head, which points at a particular cell and can read and write the symbol at that cell. It also has an internal state, from a finite set of states. At each timestep, based on the current state and the contents of the cell at the head, the machine writes a new symbol, changes the internal state, and can move the head left or right or leave it in place. The action of the machine depends on a finite set of rules, which specify the actions to take for each state-symbol combination. Note that the version of Turing machine used here has only 2 symbols, and requires that the initial contents of the tape be all 0 (the first symbol) except for finitely many 1s (the second symbol).
When converting a Turing machine to graphical format, the tape of the machine is modeled as a linked list of cells. Additionally, each state of the machine is denoted by a state node, and edges between these nodes encode the transition rules. There is also a head node, which connects both to the current cell and to the current state of the machine. See Appendix C for more details.
5.2.3 ANALYSIS AND RESULTS
The GGT-NN model was trained on 1000 examples of the Rule 30 automaton with different initial states, each of which simulated 7 timesteps of the automaton, and 20,000 examples of Turing
machines with different rules and initial tape contents, each of which simulated 6 timesteps of the Turing machine. Performance was then evaluated on 1000 new examples generated with the same format. The models were evaluated by picking the most likely graph generated by the model, and comparing it with the correct graph. The percent accuracy denotes the fraction of the examples for which these two graphs were identical at all timesteps. In addition to evaluating the performance on identical tasks, the generalization ability of the models was also assessed. The same trained models were evaluated on versions of the task with 20 and 30 timesteps of simulation.
Results are shown in Table 3. The models successfully learned the assigned tasks, reaching high levels of accuracy for both tasks. Additionally, the models show the ability to generalize to large inputs, giving a perfect output in the majority of extended tasks. For visualization purposes, Figure 3 shows the model at various stages of training when evaluated starting with a single 1 cell.
6 COMPARISON WITH RELATED WORK
Many methods have been proposed for combining neural networks with graphs. These methods generally require the input to the network to be in graphical format. For instance, GNNs and GGS-NNs take a graph as input, and propagate information between nodes according to the graph structure (Gori et al., 2005; Scarselli et al., 2009; Li et al., 2016). Similarly, graph convolutional networks extract information from an existing graph structure by using approximations to spectral graph convolutions (Kipf & Welling, 2016). These methods are similar to GGT-NNs in that they all store information in the nodes of a graph and use edges to determine how information flows. However, they all use a graph with fixed structure, and can only accept graphical data. The GGT-NN model, on the other hand, allows the graph structure to be built and modified based on unstructured input.
Giles et al. (1992) describe a method for extracting a finite state machine from a trained recurrent neural network by quantizing the hidden states of the network, recording all possible state transitions, and using them to construct a minimal directed graph representing the state machine. This method, however, requires postprocessing of the network to extract the graph, and is limited to extracting graphs that represent state machines. Additionally, although the FSM extraction method described by Giles et al. (1992) and the GGT-NN model both produce graphs using neural networks, the goals are different: the FSM extraction method aims to learn a single graph that can classify sequences, whereas the GGT-NN model aims to learn a neural network that can manipulate graphs.
The lifted relational neural network (LRNN) is another approach to working with structured data (Sourek et al., 2015). LRNNs require the input to be formatted as a combination of weighted predicate logic statements, encompassing both general rules and specific known facts. For each training example, the statements are used to construct a “ground neural network”, with a connection pattern determined by the dependencies between the statements. LRNNs can learn to extract information by adjusting the weights of each statement, but require the rules to be composed by hand based on the task structure. Furthermore, unlike in GGT-NNs, a LRNN has no internal state associated with the objects it describes (which are instead represented by single neurons), and the relationships between objects cannot be constructed or modified by the network.
Multiple recent architectures have included differentiable internal states. Memory Networks, as described in Weston et al. (2014), and the fully differentiable end-to-end memory networks, described in Sukhbaatar et al. (2015), both utilize a differentiable long-term memory component, consisting of a set of memories that are produced by encoding the input sentences. To answer a query, an attention mechanism is used to select a subset of these memories, and the resulting memories are processed to produce the desired output. Differentiable Neural Computers (DNCs), described in Graves et al. (2016), interact with a fixed-size memory using a set of read and write “heads”, which can be moved within the memory either by searching for particular content or by following temporal “links of association” that track the order in which data was written.
Memory networks and DNCs share with the GGT-NN model the ability to iteratively construct an internal state based on textual input, and use that internal state to answer questions about the underlying structured data. However, in these models, the structure of the internal state is implicit: although the network can store and work with structured data, the actual memory consists of a set of vectors that cannot be easily interpreted, except by monitoring the network access patterns. The GGT-NN model, on the other hand, explicitly models the internal state as a graph with labeled
nodes and edges. This allows the produced graph to be extracted, visualized, and potentially used in downstream applications that require graph-structured input.
Hierarchical Attentive Memory (HAM) is a memory-based architecture that consists of a binary tree built on top of an input sequence (Andrychowicz & Kurach, 2016). A recurrent controller accesses the HAM module by performing a top-down search through the tree, at each stage choosing to attend to either the left or right subtrees. Once this process reaches a leaf, the value of the leaf is provided to the controller to use in predicting the next output, and this leaf’s value can be updated with a new value. This architecture is especially suited toward sequence-based tasks, and has been shown to generalize to longer sequences very efficiently due to the tree structure. However, it is unclear whether a HAM module would work well with non-sequential structured data, since the tree structure is fixed by the network.
One advantage of the GGT-NN model over existing works is that it can process data in a distributed fashion. Each node independently processes its surroundings, which can be beneficial for complex tasks such as pathfinding on a graph. This is in contrast to memory networks, DNCs, and HAM modules, which are restricted to processing only a fixed number of locations in a given timestep. On the other hand, the distributed nature of the GGT-NN model means that it is less time and space efficient than these other networks. Since every node can communicate with every other node, the time and space required to run a GGT-NN step scales quadratically with the size of the input. A DNC or memory network, on the other hand, either scales linearly (since it attends to all stored data or memories) or is constant (if restricted to a fixed-size memory), and a HAM module scales logarithmically (due to the tree structure).
7 CONCLUSION
The results presented here show that GGT-NNs are able to successfully model a wide variety of tasks using graph-structured states and potentially could be useful in solving many other types of problems. The specific GGT-NN model described here can be used as-is for tasks consisting of a sequence of input sentences and graphs, optionally followed by a query. In addition, due to the modular nature of GGT-NNs, it is possible to reconfigure the order of the transformations to produce a model suitable for a different task.
The GGT-NN architecture has a few advantages over the architectures described in existing works. In contrast to other approaches to working with structured data, GGT-NNs are designed to work with unstructured input, and are able to modify a graphical structure based on the input. And in contrast to memory networks or DNCs, the internal state of the network is explicitly graph structured, and complex computations can be distributed across the nodes of the graph.
One downside of the current model is that the time and space required to train the model increase very quickly as the complexity of the task increases, which limits the model’s applicability. It would be very advantageous to develop optimizations that would allow the model to train faster and with smaller space requirements, such as using sparse edge connections, or only processing some subset of the nodes at each timestep. Another promising direction of future work is in reducing the level of supervision needed to obtain meaningful graphs, for example by combining a few examples that have full graph-level supervision with a larger set of examples that do not have graph-level information, or using additional regularization to enable the GGT-NN model to be trained without any graph information.
There are exciting potential uses for the GGT-NN model. One particularly interesting application would be using GGT-NNs to extract graph-structured information from unstructured textual descriptions. More generally, the graph transformations provided here may allow machine learning to interoperate more flexibly with other data sources and processes with structured inputs and outputs.
ACKNOWLEDGMENTS
I would like to thank Harvey Mudd College for computing resources. I would also like to thank the developers of the Theano library, which I used to run my experiments. This work used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation grant number ACI-1053575.
APPENDIX A BACKGROUND ON GG-NNS AND GGS-NNS
This section gives additional background on the implementation of GG-NNs and GGS-NNs, described by Li et al. (2016).
Recall from section 2.2 that GG-NNs represent a graph G = (V, E) as a set V of nodes v with unique values 1, . . . , |V| and a set E of directed edges e = (v, v′) ∈ V × V oriented from v to v′. Each node has an annotation xv ∈ RN and a hidden state hv ∈ RD. Additionally, each edge has a type ye ∈ {1, · · · ,M}.
Initially, h(1)v is set to the annotation xv padded with zeros. Then nodes exchange information for some fixed number of timesteps T according to the propagation model
h (1) v = [x>v ,0] > a (t) v = A>v:[h (t−1)> 1 · · ·h (t−1)> |V| ] > (1)
z (t) v = σ(Wza (t) v + Uh (t−1) v ) r (t) v = σ(Wra (t) v + Urh (t−1) v )
h̃ (t) v = tanh(Wa (t) v + U(r (t) v h(t−1)v )) h(t)v = (1− z(t)v ) h(t−1)v + z(t)v h̃(t)v .
Here a(t)v represents the information received by each node from its neighbors in the graph, and the matrix A ∈ RD|V|×2D|V| has a specific structure that determines how nodes communicate. The first half of A, denoted A(out) ∈ RD|V|×D|V|, corresponds to outgoing edges, whereas the second half A(in) ∈ RD|V|×D|V| corresponds to incoming edges. Each edge type y corresponds to specific forward and backward propagation matrices Py,P ′ y ∈ RD×D which determine how to propagate information across an edge of that type in each direction. The D×D-sized submatrix of A(out) in position i, j contains Py if an edge of type y connects nodes ni to nj , or 0 if no such edge connects in that direction. Similarly, the D × Dsized submatrix of the matrix A(in) in position i, j contains P′y if an edge of type y connects nodes nj to ni, or 0 if no such edge connects in that direction. Av: ∈ RD×2D|V| is the submatrix of A corresponding to node v. Thus, multiplication by Av: in 1 is equivalent to taking the following sum:
a(t)v = ∑ v′∈V ( M∑ y=1 sedge(v, v ′, y) Py + sedge(v′, v, y) P′y ) h (t−1) v′ (2)
where sedge(v, v′, y) is 1 if e = (v, v′) ∈ E and ye = y, and 0 otherwise. The output from a GG-NN is flexible depending on the task. For node selection tasks, a node score ov = g(h (T ) v ,xv) is given for each node, and then a softmax operation is applied. Graph-level outputs are obtained by combining an attention mechanism i and a node representation function j, both implemented as neural networks, to produce the output representation
hG = tanh (∑ v∈V σ(i(h (T ) v ,xv)) tanh(j(h(T )v ,xv)) ) (3)
Gated Graph Sequence Neural Networks (GGS-NN) are an extension of GG-NNs to sequential output o(1), . . . ,o(K). At each output step k, the annotation matrix X is given by X (k) = [x
(k) 1 , . . . ,x (k) |V|] > ∈ R|V|×LV . A GG-NN Fo is trained to predict an output sequence o(k) from X (k), and another GG-NN FX is trained to predict X (k+1) from X (k). Prediction of the output at each step is performed as in a normal GG-NN, and prediction of X (k+1) from the set of all final hidden statesH(k,T ) (after T propagation steps of FX) occurs according to the equation
x (k+1) v = σ ( j(h (k,T ) v ,x (k) v ) ) .
APPENDIX B GRAPH TRANSFORMATION DETAILS
In this section I describe in detail the implementations of each type of differentiable graph transformation.1 A diagram of the implementation of each transformation is shown in Figure 4. Note that it is natural to think of these transformations as operating on a single graphical state, and each modifying the state in place. However, in the technical descriptions of these transformations, the operations will be described as functions that take in an old graph and produce a new one, similarly to unrolling a recurrent network over time.
B.1 NODE ADDITION
The node addition transformation Tadd : Γ × Rα → Γ takes as input a graph G and an input vector a ∈ Rα, and produces a graph G′ with additional nodes. The annotation and strength of each new node is determined by a function fadd : Rα × Rβ → R × RN × Rβ , where α is the length of the input vector, β is the length of the internal state vector, and as before N is the number of node types. The new nodes are then produced according to
(s|VG |+i,x|VG |+i,hi) = fadd(a,hi−1), (4)
starting with h0 initialized to some learned initial state, and recurrently computing sv and xv for each new node, up to some maximum number of nodes. Based on initial experiments, I found that implementing fadd as a GRU layer followed by 2 hidden tanh layers was effective, although other recurrent networks would likely be similarly effective. The node hidden states hv are initialized to zero. The recurrence should be computed as many times as the maximum number of nodes that
1The code for each transformation, and for the GGT-NN model itself, is available at https://github. com/hexahedria/gated-graph-transformer-network.
might be produced. The recurrent function fadd can learn to output sv = 0 for some nodes to create fewer nodes, if necessary.
Note that in order to use information from all of the existing nodes to produce the new nodes, the input to this transformation should include information provided by an aggregation transformation Trepr, described in section B.5.
B.2 NODE STATE UPDATE
The node state update transformation Th : Γ×Rα → Γ takes as input a graph G and an input vector a ∈ Rα, and produces a graph G′ with updated node states. This is accomplished by performing a GRU-style update for each node, where the input is a concatenation of a and that node’s annotation vector xv and the state is the node’s hidden state, according to
rv = σ (Wr[a xv] + Urhv + br) , zv = σ (Wz[a xv] + Uzhv + bz) ,
h̃′v = tanh (W[a xv] + U (r hv) + b) , h′v = zv hv + (1− zv) h̃′v
B.2.1 DIRECT REFERENCE UPDATE
For some tasks, performance can be improved by providing information to nodes of a particular type only. For instance, if the input is a sentence, and one word of that sentence directly refers to a node type (e.g., if nodes of type 1 represent Mary, and Mary appears in the sentence), it can be helpful to allow all nodes of type 1 to perform an update using this information. To accomplish this, Th can be modified to take node types into account. (This modification is denoted Th,direct.) Instead of a single vector a ∈ Rα, the direct-reference transformation takes in A ∈ RN×α, where An ∈ Rα is the input vector for nodes with type n. The update equations then become
av = xvA
rv = σ (Wr[av xv] + Urhv + br) , zv = σ (Wz[av xv] + Uzhv + bz) ,
h̃′v = tanh (W[av xv] + U (r hv) + b) , h′v = zv hv + (1− zv) h̃′v
B.3 EDGE UPDATE
The edge update transformation TC : Γ×Rα → Γ takes a graph G and an input vector a ∈ Rα, and produces a graph G′ with updated edges. For each pair of nodes (v, v′), the update equations are
cv,v′ = fset(a,xv,hv,xv′ ,hv′) rv,v′ = freset(a,xv,hv,xv′ ,hv′) C′v,v′ = (1− Cv,v′) cv,v′ + Cv,v′ (1− rv,v′).
The functions fset, freset : Rα×2N×2D → [0, 1]Y are implemented as neural networks. (In my experiments, I used a simple 2-layer fully connected network.) cv,v′,y gives the level of belief in [0, 1] that an edge from v to v′ of type y should be created if it does not exist, and rv,v′,y gives the level of belief in [0, 1] that an edge from v to v′ of type y should be removed if it does. Setting both to zero results in no change for that edge, and setting both to 1 toggles the edge state.
B.4 PROPAGATION
The propagation transformation Tprop : Γ → Γ takes a graph G = G(0) and runs a series of T propagation steps (as in GG-NN), returning the resulting graph G′ = G(T ). The GG-NN propagation step is extended to handle node and edge strengths, as well as to allow more processing to occur to
the information transferred across edges. The full propagation equations for step t are
a(t)v = ∑ v′∈V sv′ M∑ y=1 Cv,v′,y f fwdy (xv′ ,h (t−1) v′ ) + Cv′,v,y f bwd y (xv′ ,h (t−1) v′ ) (5)
z(t)v = σ(Wz[a (t) v xv] + Uh (t−1) v + bz) (6)
r(t)v = σ(Wr[a (t) v xv] + Urh (t−1) v + br) (7)
h̃ (t) v = tanh(W[a (t) v xv] + U(r (t) v h(t−1)v ) + bh) (8)
h(t)v = (1− z(t)v ) h(t−1)v + z(t)v h̃ (t) v . (9)
Equation 5 has been adjusted in the most significant manner (relative to Equation 2). In particular, sv′ restricts propagation so that nodes with low strength send less information to adjacent nodes, sedge has been replaced with C to allow edges with fractional strength, and the propagation matrices Py,P ′ y have been replaced with arbitrary functions f fwd y , f bwd y : RN × RD → Rα, where α is the length of the vector a. I used a fully connected layer to implement each function in my experiments. Equations 6, 7, and 8 have also been modified slightly to add a bias term.
B.5 AGGREGATION
The aggregation transformation Trepr : Γ→ Rα produces a graph-level representation vector from a graph. It functions very similarly to the output representation of a GG-NN (equation 3), combining an attention mechanism with a node representation function, but is modified slightly to take into account node strengths. As in GG-NN, both i and j are neural networks, and in practice a single fully connected layer appears to be adequate for both.
hG = tanh (∑ v∈V svσ(i(h (T ) v ,xv)) tanh(j(h(T )v ,xv)) ) .
APPENDIX C GRAPH FORMAT DETAILS
C.1 BABI TASKS
The knowledge graph object used during generation of the bAbI tasks is structured as a dictionary relating entities to each other with specific relationship types. Entities are identified based on their names, and include people (John, Mary, Sandra), locations (bedroom, kitchen, garden), objects (football, apple, suitcase), animals (mouse, wolf, cat), and colors (white, yellow, green), depending on the particular task. Relationships between entities are also expressed as strings, and are directed: if John is holding the milk there is an “is in” relationship from “milk” to “John”; if Sandra is in the bedroom there is an “is in” relationship from “Sandra” to “bedroom”; if Lily is green there is a “has color” relationship from “Lily” to “green”, etc.
The transformation from the knowledge object to a graph is straightforward: each entity used is assigned to a new node type, and relationships between entities are represented as edges between the corresponding nodes. To avoid confusion from overloaded relationships (such as “is in” being used to represent an object being held by a person as well as a person being in a room), relation names are given a distinct edge type depending on the usage context. For instance, when a person is carrying an object, the generic “is in” relationship becomes an edge of type “gettable is in actor”.
Some of the graph representations had to be modified in order to ensure that they contained all of the necessary information. For instance, task 3 requires the network to remember where items were in the past, but the knowledge object only contained references to their current locations. In these cases, a linked list structure was added to the knowledge object to allow the history information to be represented in the graph.
In particular, each time an item changed locations, a new “record” node was added, with a “previous” edge to the previous history node and a “value” edge to the current location of the item. Each item then connected to the most recent history node using a “history-head” edge. This ensures that the history of each node is present in the graph.
In a few of the tasks, specific entities had multi-word representations. While this works for normal input, it makes it difficult to do direct reference, since direct reference is checked on an individual word level. These tasks were modified slightly so that the entities are referred to with single words (e.g. “red square” instead of “red square”).
An example of a graph produced from the bAbI tasks is given in Figure 5.
C.2 CELLULAR AUTOMATON
The cellular automaton task was mapped to graphical format as follows: Nodes have 5 types: zero, one, init-cell, left-cell, and right-cell. Edges have 2 types: value, and next-r. There is always exactly one “zero” node and one “one” node, and all of the cell nodes form a linked list, with a “value” edge connecting to either zero or one, and a “next-r” edge pointing to the next cell to the right (or no edge for the rightmost cell).
At the start of each training example, there are 13 timesteps with input of the form “init X” where X is 0 or 1. These timesteps indicate the first 13 initial cells. Afterward, there are 7 “simulate” inputs. At each of these timesteps, one new left-cell node is added on the left, one new right-cell node is added on the right, and then all cells update their value according to the Rule 30 update rules.
An example of the graphical format for the cellular automaton task is given in Figure 6.
C.3 TURING MACHINE
For the Turing machine task, nodes were assigned to 8 types: state-A, state-B, state-C, state-D, head, cell, 0, and 1. Edges have 16 types: head-cell, next-left, head-state, value, and 12 types of the form rule-R-W-D, where R is the symbol read (0 or 1), W is the symbol written (0 or 1), and D is the direction to move afterward (Left, Right, or None). State nodes are connected with rule edges, which together specify the rules governing the Turing machine. Cell nodes are connected to adjacent cells with next-left edges, and to the symbol on the tape with value edges. Finally, the head node is connected to the current state with a head-state edge, and to the current cell of the head with a head-cell edge.
At the start of each training example, each of the rules for the Turing machine are given, in the form “rule state-X R W state-Y D”. Next, the initial state is given in the format “start state-X”, and the initial contents of the tape (of length 4) are given sequentially in the format “input symbol-X”, with the position for the head to start marked by “input symbol-X head”. Finally, there are 6 “run” inputs, after each of which the head node updates its edges and the cell at the head updates its value according to the rules of the Turing machine. If the head leaves the left or right of the tape, a new node is introduced there.
An example of the graphical format for the Turing machine task is given in Figure 7.
APPENDIX D GRAPH SEQUENCE INPUT
The model described in Section 4 conditions the output of the model on the final graph produced by the network. This is ideal when the graph represents all of the necessary knowledge for solving the task. However, it may also be desirable for each graph to represent a subset of knowledge corresponding to a particular time, and for the output to be based on the sequence of graphs produced. For instance, in the third bAbI task (which requires reasoning about the temporal sequence of events) each graph could represent the state of the word at that particular time, instead of representing the full sequence of events prior to that time. In Appendix C, section C.1, I describe a transformation to the tasks which allows all information to be contained in the graph. But this adds complexity to the graphical structure. If it were possible for the model to take into account the full sequence of graphs, instead of just the final one, we could maintain the simplicity of the graph transformation.
To this end, I present an extension of the GGT-NN model that can produce output using the full graphical sequence. In the extended model, the graphical output of the network after each input sentence is saved for later use. Then, when processing the query, the same set of query transformations are applied to every intermediate graph, producing a sequence of representation vectors hanswer1 , . . . ,h answer K . These are then combined into a final summary representation vector h answer summary
Direct reference No direct reference Task Accuracy Accuracy
3 - Three Supporting Facts 90.3% 65.4% 5 - Three Arg. Relations 89.8% 74.2%
using a recurrent network such as a GRU layer, from which the output can be produced. The modified pseudocode for this is shown in Algorithm 2.
I evaluated the extended model on bAbI tasks 3 and 5, the two tasks which asked questions about a sequence of events. (Note that although Task 14 also involves a sequence of events, it uses a set of discrete named time periods and so is not applicable to this modification.) The model was trained on each of these tasks, without the extra record and history nodes used to store the sequence, instead simply using the sequence of graphs to encode the relevant information. Due to the simpler graphs produced, intermediate propagation was also disabled.
Results from training the model are shown in Table 4. The accuracy of the extended model appears to be slightly inferior to the original model in general, although the extended direct-reference model of task 5 performs slightly better than its original counterpart. One possible explanation for the inferiority of the extended model is that the increased amount of query processing made the model more likely to overfit on the training data. Even so, the extended model shows promise, and could be advantageous for modeling complex tasks for which preprocessing the graph would be impractical.

The paper proposes an extension of the Gated Graph Sequence Neural Network by including in this model the ability to produce complex graph transformations. The underlying idea is to propose a method that will be able build/modify a graph-structure as an internal representation for solving a problem, and particularly for solving question-answering problems in this paper. The author proposes 5 different possible differentiable transformations that will be learned on a training set, typically in a supervised fashion where the state of the graph is given at each timestep. A particular occurence of the model is presented that takes a sequence as an input a iteratively update an internal graph state to a final prediction, and which can be applied for solving QA tasks (e.g BaBi) with interesting results.
The approach  in this paper is really interesting since the proposed model is able to maintain a representation of its current state as a complex graph, but still keeping the property of being differentiable and thus easily learnable through gradient-descent techniques. It can be seen as a succesfull attempt to mix continuous and symbolic representations. It moreover seems more general that the recent attempts made to add some 'symbolic' stuffs in differentiable models (Memory networks, NTM, etc...) since the shape of the state is not fixed here and can evolve. My main concerns is about the way the model is trained i.e by providing the state of the graph at each timestep which can be done for particular tasks (e.g Babi) only, and cannot be the solution for more complex problems. My other concern is about the whole content of the paper that would perhaps best fit a journal format and not a conference format, making the article still difficult to read due to its density.