Sum-Product networks (SPNs) are expressive deep architectures for representing probability distributions, yet allowing exact and efficient inference. SPNs have been successfully applied in several domains, however always as black-box distribution estimators. In this paper, we argue that due to their recursive definition, SPNs can also be naturally employed as hierarchical feature extractors and thus for unsupervised representation learning. Moreover, when converted into Max-Product Networks (MPNs), it is possible to decode such representations back into the original input space. In this way, MPNs can be interpreted as a kind of generative autoencoder, even if they were never trained to reconstruct the input data. We show how these learned representations, if visualized, indeed correspond to “meaningful parts” of the training data. They also yield a large improvement when used in structured prediction tasks. As shown in extensive experiments, SPN and MPN encoding and decoding schemes prove very competitive against the ones employing RBMs and other stacked autoencoder architectures.
1 INTRODUCTION
On a high level, the generative approach to machine learning can be described as follows: Given a set of samples D, drawn (usually i.i.d.) from an unknown distribution p∗ over random variables (RVs) X, recover p∗ from D. To a certain extent, generative learning (GL) can be seen as the “kingclass” paradigm in machine learning. It is well known that an optimal predictor – given an additional loss function – can just be derived from p∗. For example, assuming that Y is a class variable and X are observed features, the classifier with minimal expected 0/1-loss is given by argmaxy p ∗(y,X).
It is therefore not surprising that GL and representation learning (RL) (Bengio et al., 2012) are highly related, as both aim at “formally understanding” data. GL can be described as a “black-box” approach, since we are usually interested in the capability of some model pθ to capture the underlying distribution p∗. In RL, however, one may be interested in interpreting the “inner parts” of pθ as abstract features of the original raw data. Both perspectives can be seen in the seminal RL approaches (Hinton & Salakhutdinov, 2006; Bengio et al., 2006), as the activations of generatively trained models are employed as data representations for initializing deep architectures.
As another simple example, consider a Bayes classifier, which estimates the joint distribution p(Y,X) by using the class-prior p(Y ) and class-conditionals p(X |Y ). In a purist GL view, we estimate p(Y ) and p(X |Y ) to compute p(Y,X) = p(X |Y ) p(Y ) ∝ p(Y |X). In an RL approach, however, we would recognize that the parts of our model p(X |Y ) (or also p(Y |X)) can be interpreted as a kind of soft one-hot encoding for Y , and would use them as features in a discriminative approach. The same argument holds for an unsupervised learning scenario, i.e. when Y is unobserved: we would deal with latent mixture models for which p(X |Y ) are the mixture components. In summary, we
note that any generative model – depending on its structure and semantics – might be a potential feature extractor and thus a potential useful representation learner.
In this paper, we investigate a particular promising candidate for this approach, namely Sum-Product Networks (SPNs) (Poon & Domingos, 2011), recently proposed deep probabilistic networks, admitting exact but tractable inference for several kinds of probabilistic queries. SPNs have been successfully applied to computer vision (Gens & Domingos, 2012; Amer & Todorovic, 2015), speech (Peharz et al., 2014b; Zöhrer et al., 2015) and language modeling (Cheng et al., 2014). In these works, however, SPNs have been used only as black-box distribution estimators.
Here we exploit SPNs for RL. One way to interpret SPNs is as a hierarchically structured generalization of mixture models: they are nested arrangements of factorized distributions (product nodes) and mixture distributions (weighted sum nodes) defining distributions over subsets of X. Due to this peculiarity, representations extracted from an SPN by evaluating the network nodes extend the idea of using mixture components as features, as in the motivating example above, in a recursive way. In Vergari et al. (2016) some initial approaches to encode embeddings via an SPN were proposed, showing how these model can constitute an interesting alternative or addition to other popular generative feature extractors such as RBMs (Hinton & Salakhutdinov, 2006; Marlin et al., 2010). The advantages of employing SPNs for RL are that one can “easily” learn both structure and parameters by leveraging the SPN’s recursive probabilistic semantics (Gens & Domingos, 2013), rather than imposing an a-priori structure or using an ad-hoc weight learning algorithm, as usually done for other deep architectures. Rich hierarchical features can be obtained even by such a simple generative learning scheme. Indeed, in an SPN each node can be seen as a probabilistic part-based feature extractor. Visualizations of the filters learned by SPNs trained on images data confirm that these networks are able to learn meaningful representations at different levels of abstraction.
In this work we provide a way to decode the learned representations back to their original space by employing a Max-Product Network (MPN) (Poon & Domingos, 2011). Our decoding procedure leverages the Most Probable Explanation (MPE) (Darwiche, 2009) inference routine for SPNs and incorporates an imputation mechanism for missing components in a representation to be decoded. To a certain extent, an MPN can be exploited as a kind of generative autoencoder. We continue the work of Vergari et al. (2016) by adding other ways to leverage SPN representations, again for “free”, i.e. without training the network with the aim to reconstruct its input. Additionally, we characterize conditions when MPNs can be considered perfect encoder-decoders under the proposed scheme. As a final contribution, we evaluate the meaningfulness and usefulness of SPN and MPN representations in an extensive set of structured output prediction tasks. Having devised a decoding procedure allows us to explore different learning scenarios, e.g. building embeddings for the input features, for the labels or for both. We demonstrate that these encoding and decoding schemes, “cheaply” obtained by a generative SPN, show surprisingly competitive performances when compared to those extracted from RBMs, probabilistic autoencoders (Germain et al., 2015) and deep autoencoders tailored for label embeddings (Wicker et al., 2016) in all the learning scenarios evaluated.
2 SUM-PRODUCT NETWORKS
Let RVs be denoted by upper-case letters, e.g. X , Y and let corresponding lower-case letters denote their values, e.g. x ∼ X . Similarly, boldface notation denotes sets of RVs and their combined values, e.g. X, Y and x, y. For Y ⊆ X and a sample x, we denote with x|Y the restriction of x to Y. An SPN S over a set of RVs X is a probabilistic model defined via a rooted DAG. The leaves of the graph (the SPN’s inputs) are computationally tractable, possibly unnormalized distributions over a sub-set of X. When n is a leaf of S, let φn denote its associated distribution. The inner nodes compute either weighted sums or products over their children. Let ch(n) be the set of children for a particular node n. For a sum node n and a child c ∈ ch(n), we associate a nonnegative weight wnc with the outgoing sum-edge n→ c. The set of all sum weights in S (the network parameters) is denoted as w. Furthermore, let S⊕ (resp. S⊗) be the set of all sum (resp. product) nodes in S.
Let Sn denote the sub-network rooted at node n and parametrized by wn. Each node n in S defines a probability distribution pwn over its scope by normalizing the output of Sn. Consequently, the distribution of S over all X is defined as the root normalized output. Sn(x|sc(n)), or short-hand Sn(x), indicates the output value of node n when X = x is observed as the network input.
Let sc(∗) denote the scope, a labeling function associating to each node n a subset of X, i.e. sc(n) ⊆ X. For a leaf n, sc(n) is the set of RVs over which φn is defined. The scope of an inner node n is defined as sc(n) = ⋃ c∈ch(n) sc(c). The scope gives rise to some fundamental properties of an SPN: S is complete if ∀n ∈ S⊕ and ∀c1, c2 ∈ ch(n) : sc(c1) = sc(c2). S is decomposable if ∀n ∈ S⊗ and ∀c1, c2 ∈ ch(n), c1 6= c2 : sc(c1) ∩ sc(c2) = ∅ (an example in Figure 1a). While inference in unconstrained SPNs is intractable, marginalization in complete and decomposable SPNs reduces to performing the marginalization task at the leaves and evaluating the inner nodes as usual (Poon & Domingos, 2011; Peharz et al., 2015). An SPN is locally normalized when it holds that ∀n ∈ S⊕ : ∑c∈ch(n) wnc = 1 and all leaves represent normalized distributions. For complete, decomposable and locally normalized SPNs, the distributions of all nodes are already correctly normalized distributions. In the following, we only consider this class of SPNs.
While marginalization can be tackled in time linear in the network size, the problem of finding a Most Probable Explanation (MPE) is generally NP-hard in SPNs (Peharz et al., 2016). Given two sets of RVs Q,O ⊂ X, Q ∪O = X and Q ∩O = ∅, inferring an MPE assignment is defined as finding
x∗|Q = argmax q∼Q p(o,q). (1)
However, MPE can be solved exactly in selective SPNs (Peharz et al., 2014b; 2016), i.e. SPNs where it holds that for each sample x at most one child of each sum node is non-zero. MPE in selective SPNs is solved via MPEAssignment (Poon & Domingos, 2011), which evaluates the network twice. First one builds an MPN M from S by replacing each node n ∈ S⊕ by a max node n ∈Mmax computing maxc∈ch(n) wncMc(x) and each leaf distribution by a maximizing distribution (Peharz et al., 2016) (Figure 1b). One then computes M(x|O) – the MPE probability of the query p(x|O) – by evaluating M bottom-up (Figure 1c). Stage two consists of a top-down traversal of M . Starting from the root, one follows the maximal child branch for each max node and all child branches of a product node. Each partial input configuration determines a unique tree path. The MPE assignment x∗ is obtained by collecting the MPE solutions (w.r.t. Q) of the leaves in the path (Figure 1d). For selective SPNs, the corresponding MPNs compute precisely the same value for each node, since sums and maxes are equivalent when applied to all zeros but one nonnegative value. In the non-selective case, MPNs can be seen as a (lower-bounding) approximation of SPNs, and are thus also an interesting candidate for RL, as showed in the next sections. Furthermore, while MPEAssignment solution for general SPNs is not exact, it is still employable as a reasonable and common approximation (Peharz et al., 2016).
SPNs and MPNs can be interpreted as very peculiar deep Neural Networks (ANNs) that are labeled, constrained and fully probabilistic (Vergari et al., 2016). They are labeled networks because of the scope function, which enables a direct encoding of the input (Bengio et al., 2012). Their topology is constrained because of the completeness and decomposability properties, hence connections are sparse and not dense. Differently from other distribution estimators like NADEs (Larochelle &
Murray, 2011) and MADEs (Germain et al., 2015), they are fully probabilistic: not only the network outputs emit valid probability values, but each inner node as well, due to their recursive definition.
The semantics of SPNs enable the design of simple and yet surprisingly effective structure learning algorithms (Dennis & Ventura, 2012; Peharz et al., 2013; Gens & Domingos, 2013). Many recent attempts and variants (Rooshenas & Lowd, 2014; Adel et al., 2015; Vergari et al., 2015) build upon the currently most prominent algorithm LearnSPN, a greedy top-down SPN learner introduced in (Gens & Domingos, 2013). LearnSPN proceeds by recursively decomposing a given data matrix along its rows (i.e. samples), generating sum nodes and estimating their weights, and its columns (i.e. RVs), generating products. To a certain extent, LearnSPN can be interpreted as a recursive data crawler, extracting peculiar features from a data matrix, which potentially only live in a particular data cluster and/or in a certain subset of RVs. This may be one of the few cases when the structure and parameters of an ANN can be learned without directly optimizing a loss function.
3 LEARNING REPRESENTATIONS WITH SPNS AND MPNS
In this section we discuss how to exploit an SPN S or its corresponding MPN M for RL, after structure and parameters are generatively learned over X, following Vergari et al. (2016). We are interested in encoding each sample xi ∼ X into a continuous vector representation ei in a new d-dimensional space, i.e. an embedding ei ∈ EX ⊆ Rd, where ei = fS(xi) (SPNs) or ei = fM (xi) (MPNs). We usually refer to SPNs, since most of the time similar consideration hold also for MPNs.
For ANNs, the common approach is to use the hidden neuron activations of the upper layers as the learned representations for f . As argued above, SPN nodes are particular interpretable due to their clear probabilistic semantics. Given an SPN S and a set of nodes N = {nj}dj=1 ⊂ S, we construct our embedding as eij = Snj (x|sc(nj)) = pwnj (x|sc(nj)), where a reasonable selection criterion for N is given below. Each value represents the probability to see that sample according to a marginal distribution over a node scope. Thus, the so-constructed embedding is a point in the geometric space induced by a collection of proper probability densities.
SPN nodes can also be seen as part-based filters operating over sub-spaces given by the node scopes. Sum nodes can be interpreted as filters built by weighted averages over filters sharing the same scope, and product nodes can be seen as compositions of filters over non-overlapping scopes. From the perspective of the internal mechanisms of LearnSPN-like algorithms, each filter captures a different aspect of sub-population and sub-space of the data. Thereby, the scope information induces a hierarchy of filters at different levels of abstraction.
To confirm this interpretation, we visualize the features extracted from nodes in an SPN learned on image samples (Vergari et al., 2016). For ANNs, the feature filtered by a hidden neuron can be visualized as the image in the input space that maximally activates that neuron (Erhan et al., 2009). In SPNs this corresponds to solving MPE for the sub-SPN rooted at a particular node, and restricted to the node’s scope. As stated in Section 2, we employ MPEAssignment as an approximation to this generally hard problem. Figure 2 shows some of the MPE solutions/filter activations for an SPN trained on a binarized version of MNIST (Larochelle et al., 2007) (see Appendix C for details). Note that they resemble part-based features at different levels of complexity: from small blobs (Figure 2a) to shape contours (Figures 2b and 2c), to full digits comprising background parts (Figure 2d). The missing background pixels, visualized in a checkerboard pattern, is due to those pixels being out of the scope for those nodes. This pattern locality is an SPN peculiarity: although also fully connected ANNs typically show locality (e.g. edge filters), the locality information is explicitly encoded in SPNs via the node scopes. This suggests that the scope information alone may already be able to convey a meaningful representation of “object parts”, e.g. see the ‘O’ shapes in Figure 2. Also, note that filters appear qualitatively different from most classical ANNs, which motivates to combine SPN features with those from other deep architectures, an approach worth further investigation.
While in classical deep ANNs the layer depth is usually associated with the level of abstraction of its filters (Erhan et al., 2009; Zeiler & Fergus, 2014; Yosinski et al., 2014), note that this does not easily translate to SPNs. First, even rather simple models might yield extremely deep networks, when translated into SPNs. For example, when representing a hidden Markov model (HMM) as SPN Peharz et al. (2014b), the SPN’s depth grows linearly in the length of the HMM. Thus, representations learned by SPNs are not easily arranged in a meaningful layered hierarchy, due to their constrained
topology and how they are learned. Moreover, LearnSPN-like algorithms can introduce nodes with very different scope information at the same level of depth. This also occurs when compiling SPNs into a minimal layered structure (Vergari et al., 2016).
Therefore, we suggest that rather the scope length |sc(n)| of a node n should be associated with its level of abstraction. The filter activations in Figure 2 give confirming evidence for our conjecture. Thus, when the aim is to compress data into an abstract representation of at most d dimensions, one reasonable filter criterion for SPN/MPN representations would be to collect the d nodes with largest scopes. Clearly, the smaller we choose d, the smaller will be the theoretically achievable quality of the reconstructed data. In our experiments, we leverage SPN representations and decoding schemes by adopting full embeddings, comprising all nodes activations (all colored values in Figure 1c), and inner embeddings, dropping out the leaf information (only orange values in Figure 1c), according to the observation that the number of leaves is overabundant w.r.t. inner nodes in SPNs built by LearnSPN (Vergari et al., 2016). Note that, in both cases, the embedding size d is adaptively induced by the data, when building the SPN, without the need to fix or tune it beforehand.
4 DECODING REPRESENTATIONS
Now we tackle the task to revert SPN representations back to the input space, i.e. to find an inverse transformation g : EX → X such that xi ≈ x̂i = g(f(xi)). Being able to decode representations extends the ways one can exploit SPNs for RL to new learning scenarios for predictive tasks. For example, if one were to learn a classifier from features X to labels Y, he could train the classifier to predict the label embeddings EY rather than Y directly. Then, the predicted embeddings could be turned into the actual outputs by applying g for decoding. By disentangling dependencies over Y in the new space EY, one can obtain better predictors. Following this principle, label embeddings have been greatly employed in RL for structured output prediction. One common approach is to compress labels into a lower dimensional space, then a regressor is trained to predict such embeddings and the predictions are decoded (decompressed) by an inverse linear transformation Bhatia et al. (2015); Akata et al. (2013). The advantage of the decoding scheme we propose is that g does not need additional training to be learned, rather it is provided by an already learned SPN turned into MPN.
Let a perfect encoder-decoder be a pair (f, g) such that, for each x ∼ X, its reconstruction is the exact same sample, i.e. g(f(x)) = x. In our analysis, we focus on MPNs and characterize when they can be used as perfect encoder-decoders. In practice, autoencoders, for which f and g are learned from data, are usually not trained to be perfect encoder-decoders, as they often might learn trivial representation, such as the identity function. This seems not to be an issue for MPNs, since the learning phase is decoupled from the decoding one. We will also empirically confirm it in Section 5.
Given an MPNM , the encoder function fM is given by collecting activations as illutrated in Section 3. Concerning the decoder, we propose a procedure for gM that mimics the MPEassignment algorithm as presented in Section 2. Recall that MPEAssingment finds a solution in the input space (top-down phase) after probabilities, i.e. node activations, are evaluated (bottom-up phase). Consider Eq. 1 in the case in which Q = ∅ and sample xi is fully observed. If all the activations from the bottom-up phase are collected into an embedding ei, its components will exactly determine the top-down descending
phase, i.e. which branch to take when encountering a max node. As a consequence, the set of leaves in the traversed tree path completely depend on ei. This is also true if ei components are not determined from a bottom-up phase but come from the “outside”, e.g. they are predicted. In order to completely define gM , each leaf node encoding φn reached in the top-down phase has to provide itself a decoder function gφn , operating over its scope. Similarly to MPEAssignment, a fully decoded embedding is then constructed by collecting the reconstructions at the leaves according to each gφn decoder.
In practice, we are interested in decoding embeddings that have been predicted by some learned model, i.e. the decoding phase is not applied to the embeddings obtained by directly evaluating a network. Nevertheless, it is important to determine under which circumstances these models behave as perfect encoder-decoders when transforming each instance to a new representation and back. Proposition 1. If for an MPN M over X there exist a perfect encoder-decoder for each leaf distribution φn and it holds for each max node n ∈ M that there is only one child node c ∈ ch(n) for which Mn(x) = wncMc(x), given x ∼ X, then M is a perfect encoder-decoder.
Proof. It is easy to demonstrate this by inductive reasoning. If M comprised only a leaf node, then it would be a perfect encoder-decoder by definition. If it were composed by a product node over child encoder-decoder MPNs, then each input could be reconstructed perfectly by the composition of the reconstruction of the child MPNs. Lastly, if it were composed by a max node over child perfect encoder-decoder MPNs Mc, c = 1 . . . k, then it would also be a perfect encoder-decoder since for each possible input, only one child component Mc∗ would output a value s.t. Mn = wnc∗Mc∗ .
From Proposition 1 it follows immediately that deterministic MPNs can be perfect encoder-decoders. Proposition 2. An MPN M constructed from a selective SPN (Peharz et al., 2014a) S is a perfect encoder-decoder, provided that the leaves have perfect encoder-decoder functions.
Thus, to complete our decoding procedure, we still have to cope with the leaf decoder functions. We define the decoded state for a leaf n as the configuration over its scope that minimizes some distance D over the leaf activation value and its encoded representation: x̃|sc(n) = argminu∼sc(n)D(φn(u)||fMn(x)). In our experiments we will employ simple L1 distance |φn(u) − fMn(x)|. Unfortunately, decoding is ambiguous for most interesting leaf distributions, such as Gaussians. However, this approach works well for discrete data used in our experiments, as long as the state probabilities are mutually distinct. In future work, we will explore techniques to disambiguate decoding the leaves, e.g. by duplicating and splitting Gaussians. In Section 5, we empirically evaluate how good are the decoding schemes depicted here, since it is worth investigating how close to perfect encoder-decoders MPNs learned on real datasets can be.
In order to apply the proposed decoding procedure, a full embedding comprising all the node activations is required. In some real cases (e.g. data compression), only an incomplete embedding, comprising only activations from a subset of the network nodes, is available. For certain incomplete embeddings a full decoding, however, is still possible.
A decodable incomplete embedding e is an embedding such that for each missing activationMn(x) 6∈ e corresponding to a node n ∈M, all the activations ec =Mc(x) ∀c ∈ ch(n) are in e. For such an incomplete embedding, it is sufficient to evaluate the MPN by propagating the embedding activations bottom-up, evaluating parent nodes after their children. The missing embedding components are then reconstructed and the decoding phase for the now full embedding can proceed as before. If even this child information is missing, such a reconstruction is not possible in general. We argue that in such a case, the missing node activations can be reasonable imputed by their most probable value. When encountering a node nj , whose corresponding embedding value ej is not available, ej is estimated as maxu∼sc(nj)Mnj (u) by employing MPEassignment on the sub-networks rooted at nj . Since the MPE activations can be precomputed for all nodes, the complexity of the whole procedure is still linear in the size of M . The pseudocode for the complete decoding procedure is listed in Appendix A.
In our experiments we evaluate the effectiveness and robustness of the decoding procedure both for complete (full) and incomplete predicted embeddings. In particular, for structured output prediction, we employ inner embeddings (cf. Section 3), where leaf values are imputed using MPEassignment as stated above. Moreover, we investigate its resilience when imputing missing at random embedding components either by just replacing them by their MPE value or by additionally evaluating the MPN bottom-up after the missing leaf activations have been imputed.
5 EXPERIMENTS
The research questions we are validating are: i) how good are learned MPNs at reconstructing their input when full/inner embeddings are decoded? ii) how meaningful are the representations learned by SPNs and MPNs and how useful is to predict these embeddings instead of the raw targets and then decoding them? iii) how resilient to missing components are the proposed decoding schemes?
Structured output prediction tasks like Multi-label Classification (MLC) offer a good experimental ground to answer the above questions. In MLC one is interested in predicting the target labels associated to a sample x ∼ X and represented as binary arrays: y ∼ Y. Since there is no unique way to assess a classifier performance in MLC, we measure the JACCARD, HAMMING and EXACT MATCH scores, as metrics highly employed in the MLC literature and whose maximization equals to focus on different sets of probabilistic dependencies (Dembczyński et al., 2012).
For all experiments we use 10 standard benchmark datasets for MLC. To fairly compare all the algorithms in our experiments, we employ the binarized versions of all datasets already processed by Di Mauro et al. (2016) and divided in 5 folds. Detailed dataset statistics are reported in Appendix B.
We learn both the structure and weights of our SPN, and hence MPN, models on X andY separately for each fold by employing LearnSPN-b (Vergari et al., 2015), a variant of LearnSPN (see Appendix C)1. Structural statistics, e.g. the number of inner nodes, for all the models are reported in Appendix D. Please refer to Tables 3 and 4 to determine the extracted embedding sizes.
5.1 RECONSTRUCTION PERFORMANCES
We want to determine how close to perfect encoder/decoders are MPNs learned from real data and equipped with our decoding schemes. In particular, we evaluate their decoding performances when the leaf activations are available (full embeddings), and the decoder employed is the L1 distance, or when they are missing (inner embeddings) and therefore their MPE state is used.
First we turn each learned SPN into an MPN. Then, each model is asked to reconstruct both the training and test samples. Detailed results are reported in Tables 5 and 6, Appendix E.2. It can be observed that the L1 leaf decoder proves to be a very reasonable approximation for binary RVs, scoring very high reconstructions for all the three measures. For the models over Y, the MPE approximation scores surprisingly good reconstructions scoring > 80% EXACT MATCH on half datasets. In general, if the network is small enough, e.g., MPNs learned on the Flags dataset or on Y alone, it behaves as a perfect encoder-decoder for full embeddings. This demonstrates the efficacy of the proposed decoding schemes and shows how the presence of tied max node children activations impacts non-deterministic MPNs learned from data. We investigate if these potentially perfect reconstructions lead to banal representations in the following experiments.
5.2 STRUCTURED OUTPUT PREDICTION PERFORMANCES
We now focus on leveraging the representations learned by SPNs and MPNs in an unsupervised way for structured output prediction. In a fully supervised scenario one wants to build a classifier on the input RVs X to predict the output RVs Y directly (X→ Y). Instead, we can first encode both the input RVs X and/or the target RVs Y into different embedding spaces, EX, EY, and build a predictive model on top of them. In order to do so, we explore different settings: we learn a classifier on the input embeddings instead of the raw features (EX → Y); alternatively, one can first train a regressor on the original input X to predict label embeddings (X → EY), then decoding such predictions back to the original label space; finally, the same regressor can be trained on the input embeddings instead (EX → EY) and its predictions decoded as above. As a proxy measure to assess the meaningfulness of the learned representations, we are considering their prediction performances. Given a predictive model, its improvement in performance in one of the above settings over the raw input/output case, X→ Y, determines how good the representations employed are. To highlight the ability of these representations to disentangle the dependencies underlying the RVs, we always train a simple linear model in all the settings. In particular, we employ an L2-regularized logistic regressor, LR, (resp. a ridge regressor, RR) to predict each RV
1All the code employed for the experiments and visualizations will be made available
in Y (resp. component in EY) independently. Therefore, the most natural baseline to measure the aforementioned representation meaningfulness is to employ the same L2-logistic regressor to the X→ Y setting. We now introduce other models as either encoders or encoder/decoders to plug into our unsupervised settings. The aim is to compare SPN/MPN representations against theirs w.r.t. the aforementioned baseline. Therefore we select them as generative models for which inference can be exactly and tractably computed. For the EX → Y setting, we compare to RBMs (Smolensky, 1986) as highly expressive generative models, for which, while the joint likelihood is intractable for RBMs, exact conditionals of the latent RVs can be computed exactly and have been proven to be very predictive features (Larochelle & Bengio, 2008; Marlin et al., 2010; Larochelle et al., 2010). To evaluate different embedding sizes, we consider RBMs having 500, 1000 and 5000 hidden units (h). A natural competitor for all settings are MADEs (Germain et al., 2015), because they are deep autoencoders which are also tractable probabilistic models. We employ MADEs comprising 3 layers and 500 and 1000 (resp. 200 and 500) hidden units per layer for the EX → Y (resp. X→ EY) setting. Additionally, we add to the comparison MANIAC (Wicker et al., 2016) a non-probabilistic autoencoder model tailored to MLC. In MANIAC, stacked autoencoders are trained to reconstruct Y by compressing each label into a new representation, which, in turn, is used to train a base model exactly as in our X→ EY setting. We employ architectures up to 4 hidden layers with different compression factors. Finally, we employ a max-margin CRF Finley & Joachims (2008), CRFSSVM, in the X→ Y setting that considers a dependency structure on the label space in the form of a Chow-Liu tree. In this way we are able to frame the performances of all the models in the unsupervised setting against a fully supervised and discriminative method on the same datasets.
In Appendix E.1 we report all the choices made to learn and tune the involved models. For SPNs, and hence MPNs, we do not need to define a handcrafted structure a priori like for all the competitors above. Consequently, for RBMs, MADEs, MANIAC it is needed to learn and cross-validate several models with different capacities to obtain properly sized embeddings. On the other hand, the size of embeddings extracted from SPNs/MPNs is adaptively determined by data, as stated in Section 3. The learned embedding sizes are reported in Tables 3 and 4 in Appendix D.
5.2.1 RESULTS AND DISCUSSION
Detailed average fold metrics and their average ranks for all datasets are reported in Tables 9, 7, 8 in Appendix E.3.1. In Table 1, instead, we report the aggregated scores over all datasets d ∈ D for each method f in the form of the average relative improvement w.r.t. the LR baseline: 1 |D| ∑ d∈D scoref (d)−scoreLR(d) scoreLR(d)
· 100. The best models for each setting and score are in bold, the higher their improvement, the better.
In summary, SPN and MPN embeddings proved to be highly competitive and even superior to all other models in the three settings and for all the scores. Even the fully supervised and discriminative CRFSSVM performance are comparable to the best SPN/MPN JACCARD (resp. HAMMING) score in the EX → EY (resp. X → EY) setting, while reporting a largely worse EXACT MATCH improvement than our models in the EX → EY setting. In particular, the setting EX → Y has proven to be hard for many models. This likely indicates that the dependencies on the X might not contribute much to the Y prediction (Dembczyński et al., 2012). Representations from SPNs, even with smaller embeddings than RBMs and MADEs (see Table 3), yield the largest improvements. In the X → EY setting, disentangling the relationships among the Y gives all models a performance boost. This is not the case for MADEs on some datasets, probably due to their reconstruction power being traded off to their generalization power as generative models. MPNs, on the other hand, consistently exploit the label representation space and do not provide overfitted reconstructions. This answers our question about the meaningfulness of MPN representations suggesting that their tendency to be perfect encoder/decoders does not damage their representation performances.
Concerning two decoding schemes we proposed, operating on incomplete (inner) embeddings not only performs comparably to the full case, but also scores the best results on some datasets for JACCARD and EXACT MATCH scores. This aspect can be seen in the EX → EY setting as well.
Additionally, to better understand the role of our decoding procedures in the X→ EY and EX → EY settings, we run a new set of experiments in which the decoding phase for EY is performed by a nearest neighbor (k = 5) model on the basis of the training labelled embeddings. In these settings, therefore, we can add to the comparison even RBM-encoded representations. Even in this scenario, MPN embeddings are the best or very competitive. As expected, the non-linear kNN predictor performs better than our full decoding on several datasets for the JACCARD and EXACT MATCH scores, but less well for the inner variant. This is likely due to the smaller inner embedding sizes and highlights the goodness of the proposed decoding approach in presence of missing values and, more in general, for maximizing the HAMMING score.
All in all, with these structured output prediction tasks we gathered empirical confirmation of the meaningfulness and practical usefulness of SPN and MPN embeddings. The reported large improvements over the three scores cannot be due to SPN/MPN larger embedding sizes. In fact, their sizes are always comparable or smaller than RBM, MADE, and MANIAC ones since the latter max capacities have been chosen after SPNs have been learned (Tables 3 and 4, Appendix D). It is also not possible to state that these representation higher predictive performances are correlated to the SPN ability to better model the data distributions, at least we look at the model likelihoods. Indeed, MADE log-likelihoods have proven to be higher that SPN ones on many datasets and comparable on the rest. We argue that the reason behind these results lies in the hierarchical part-based representations SPNs provide. Each embedding component is responsible for capturing only the significant feature portions according to its corresponding node scope, as shown in Section 3. The meaningfulness of these components as features has to be found in the structure learning performed by LearnSPN-b (Section 2): while its hierarchical co-clustering chooses to split the data into sub-populations in an unsupervised way to determine a reasonable distribution estimation, it highlights meaningful ways to discriminate among them.
5.3 RESILIENCE TO MISSING COMPONENTS
Lastly, we evaluate the resilience of the decoding procedure proposed when label embedding components are missing at random in the X → EY setting. We want to compare the two imputation schemes presented in Section 4: either employing MPEAssignment to retrieve the most probable activation or evaluating the MPN bottom-up to compute the missing predicted components.
For all datasets, for each label embedding that has been predicted, we remove at random a percentage of components varying from 0 (full embedding) to 90%, by increments of 10%. If leaves activations are missing, their MPE activation is considered. After the full embedding has been reconstructed, the decoding phase proceeds as before. Figure 3 shows how the two strategies perform differently for the EXACT MATCH score. The re-evaluation scheme is much more resilient one among the two, being able to maintain comparable scores to the full embedding case up to 30% missing components, then decaying less faster than the MPE based one. The proposed decoding scheme is therefore proved to be not only surprisingly effective but also quite robust. Similar, but less prominent, behaviors are reported for the JACCARD and HAMMING scores in the Appendix.
6 CONCLUSION
In this work we investigated SPNs and MPNs under a RL lens. We suggested an interpretation of MPNs as generative autoencoders by providing a decoding procedure that leverages approximate MPE inference. We characterize when these networks can lead to perfect reconstructions of their inputs, linking this property to determinism. When empirically evaluated in an extensive comparison for MLC, SPN and MPN representations ranked as one of the most predictive features and MPN reconstructions proved to be surprisingly effective. Encouraged by these results, we plan to explore new learning schemes directly exploiting these models learned representatons, and not optimizing their likelihood scores only. For instance, a differentiable procedure for MPE inference would allow SPNs and MPNs to be trained directly to reconstruct or denoise their input, bridging the gap even more between these networks, autoencoders and other ANNs and opening the path to hybridize them.
0.0 0.2 0.4 0.6 0.8 0.10 0.15 0.20 0.25
Arts
0.0 0.2 0.4 0.6 0.8 0.45 0.50 0.55 0.60 MPE eval
Business
0.0 0.2 0.4 0.6 0.8 0.10 0.15 0.20 0.25 0.30
Emotions
0.0 0.2 0.4 0.6 0.8 0.10 0.12 0.14 0.16 0.18
Flags
0.0 0.2 0.4 0.6 0.8 0.20 0.30 0.40 0.50
Health
0.0 0.2 0.4 0.6 0.8 0.16 0.18 0.20 0.22
Human
0.0 0.2 0.4 0.6 0.8
0.22
0.24
0.26 Plants
0.0 0.2 0.4 0.6 0.8 0.10 0.20 0.30 0.40 0.50 0.60
Scene
0.0 0.2 0.4 0.6 0.8 0.04 0.06 0.08 0.10 0.12 0.14
Yeast
Figure 3: Average test EXACT MATCH scores (y axis) obtained by imputing different percentages of missing random embedding components (x axis) for the X → EY setting on all datasets by employing MPE inference (orange crosses) or the bottom-up evaluation imputation schemes (blue squares). Results for Cal dataset are not reported since they are all zeros (see Table 9).
A DECODING ALGORITHM
Algorithm 1 lists the pseudocode for our decoding procedure as illustrated in Section 4.
Algorithm 1 decodeEmbedding(M , e, a)
1: Input: an MPN M over X, an embedding e ∈ Rd and a map a : Me ⊆M→ {1, . . . , d} 2: Output: a sample x̃ ∼ X decoded from e, according to M 3: x̃← 0|X| 4: Qx root(M) . top-down traversal of M by using a queueQ 5: while not empty(Q) do 6: nx Q . process current node 7: if n ∈Mmax then . max node 8: cmax ← argmaxc∈ch(n) wncvc such that vc ← ea(c) if c ∈Me else maxu∼sc(c)Mc(u) 9: Qx cmax 10: else if n ∈M⊗ then . product node 11: ∀c ∈ ch(n) : Qx c 12: else . leaf node 13: if n ∈Me then 14: x̃sc(n) ← argminu∼(sc(n))D(φn(u)||ea(n)) 15: else . MPEAssignment (inner embedding) 16: x̃|sc(n) ← argmaxu∼sc(n)Mn(u) 17: return x̃
B DATASETS
The 10 datasets employed come from the freely accessible MULAN2, MEKA3, and LABIC4 repositories. They are real world standard benchmarks for MLC from text, image, sound and biological domains. Subsets of them have been also used in Dembczyński et al. (2012); Antonucci et al. (2013); Kong et al. (2013). They have been binarized as in (Di Mauro et al., 2016) by implementing the Label-Attribute Interdependence Maximization (LAIM) (Cano et al., 2016) discretization method5.
Table 2 reports the information about the adopted datasets, where N , M and L represent the number of attributes, instances, and possible labels respectively. They are divided into five standard folds. Furthermore, for each dataset D = {xi,yi}Mi=1 the following statistics are also reported: label cardinality: card(D) = 1M ∑M i=1 ∑L j=1 y i j , label density: dens(D) = card(D)L and distinct labels: dist(D) = |{y|∃(xi,y) ∈ D}|.
C LEARNING SPNS
To learn the structure and weights of our SPNs (and hence MPNs), we employ LearnSPN-b Vergari et al. (2015), a variant of LearnSPN. LearnSPN-b splits the data matrix slices always into two, while performing row clustering or checking for RVs independence. With the purpose of slowing down the greedy hierarchical clustering processes, it has proven to obtain simpler and deeper networks without limiting their expressiveness as density estimators. Based on the datasets statistics reported above in Appendix B, we define the same ranges for LearnSPN-b hyperparameters both when we learn our SPNs for the X and the Y. We set the G-test independence test threshold to 5, we limit the minimum number of instances in a slice to split to 10 and we performed a grid search for the best leaf distribution Laplace smoothing value in {0.1, 0.2, 0.5, 1.0, 2.0}. We perform all computations in the log space to avoid numerical issues.
For the SPN learned on the binarized version of MNIST in Section 3 we set the G-test independence test threshold to 20 and the instance threshold to 50 in order to reduce the network size. We then applied the same grid search as above for the leaf Laplace smoothing coefficient.
D SPN MODEL STATISTICS
Statistics for the reference SPN models learned with LearnSPN-b on the X RVs only are reported in Table 3. Their average (and standard deviations) values over the dataset folds provide information about the network topology and quality: how many nodes are in there (edges + 1), how are they divided into leaves and sum and products and their max depth (as the longest path from the root). The same statistics are reported for the SPNs over RVs Y, then turned in MPNs, in Table 4.
The length of the embeddings extracted from such models is the number of inner nodes from Table 3 for the inner embeddings over X. For the embeddings over RVs Y, their length in the full setting shall be considered as the number of all nodes from Table 4.
E MORE EXPERIMENT DETAILS AND RESULTS
E.1 TRAINING DETAILS
E.1.1 LEARNING LINEAR PREDICTORS
We learn to predict each target feature independently from the others, both when we employ the L2-regularized logistic regressor (LR) to predict RV Y directly and when we use a ridge regressor (RR) to predict the label embeddings.
To select the best value for the regularization parameter we will perform a grid search for LR in the space {10−4, 10−3, 10−2, 10−1, 1} and for RR in the space {10−4, 10−3, 10−2, 10−1, 1, 10, 102}6 for each experiment.
E.1.2 LEARNING RBMS
Concerning RBMs, we train them on the X alone (or on the Y alone for the kNN experiments) by using the Persistent Constrastive Divergence (PCD) Marlin et al. (2010) algorithm, leveraging the implementation available in scikit-learn. For the weight learning hyperparameters we run a grid search for the learning rate in {0.1, 0.01}, the batch size in {20, 100} and let the number of epochs range in {10, 20, 30} since no early stopping criterion was available. We then select the best models according to their pseudo-log likelihoods. To generate embeddings from RBMs, we evaluate the conditional probabilities of the hidden units given each sample. To make the comparison fairer we transform these values in the log domain in the same way we do for our SPN and MPN representations.
E.1.3 LEARNING MADES
For MADEs, following the experimentation reported in (Germain et al., 2015), we employ adadelta to schedule the learning rate during training and fix its decay rate at 0.95; we set the max number of worsening iterations on the validation set to 30 as for RBMs and we employed a batch size of 100 samples. We initialize the weights by employing an SVD-based init scheme.
6We leverage the python implementations for LR and RR from the scikit-learn package (http:// scikit-learn.org/). Note that in scikit-learn the grid parameter for LR has to be interpreted as an inverse regularization coefficient.
Other hyperparameters are optimized by a log-likelihood-wise grid search. The gradient dumping coefficient is searched in {10−5, 10−7, 10−9}, and we employ once the shuffling of mask and orders. Both ReLus and softplus functions are explored as the non-linearities employed for each hidden neuron. We employ a MADE openly available implementation, ported to python37.
We learn architectures of three hidden layers comprising 500 and 1000 (resp. 200 and 500) hidden neurons each for the X (resp. Y). For each reference model, we extract EX embeddings by evaluating all the hidden layer activations (d = 1500 and d = 3000); for the EY case, however, only the last hidden layer embeddings are actually exploited for the prediction (d = 200 and d = 500).
E.1.4 LEARNING MANIAC MODELS
Following the experiments in Wicker et al. (2016), we perform a grid search for the following hyperparameters: the number of layers is chosen in {2, 3, 4} and the compression factor β ∈ {0.7, 0.8, 0.9}. We employ the Java implementation freely available in MEKA. For the RF version of MANIAC we build a random forest comprising 100 trees as it has been used in Wicker et al. (2016) (see Appendix E.3.2 for the results of such a model).
We were not able to properly learn MANIAC for one dataset, Cal, for all measures, as a numerical error in MEKA prevented the model evaluation, thereby we removed it in the result Table.
We were also not able to train MANIAC on the EX → Y and hence EX → EY settings because the learned representations were not available through MEKA.
E.2 RECONSTRUCTION ERRORS
In this Section we provide the detailed results for the reconstructions of the input for our SPNs turned into MPNs for each train and test portion of each dataset, averaged by fold. Table 5 (resp. Table 6) reports the results for architectures trained on the X (resp. Y) and asked to reconstruct their inputs w.r.t these RVs.
E.3 OTHER RESULTS FOR MLC
E.3.1 JACCARD HAMMING AND EXACT MATCH MEASURES
In this Section we report the additional results for the JACCARD and HAMMING measures in Table 7 and Table 8 respectively. Figures 4 and 5 report the resilience of the decoding scheme for missing at random embedding components for the JACCARD and HAMMING measures, respectively. We employ a euclidean 5-nearest neighbor classifier to perform the decoding step on all our models. These results are reported in the table last rows.
7https://github.com/arranger1044/MADE.
E.3.2 MORE MANIAC RESULTS
In addition to the ridge regressor (RR) employed as the base model in our previous experiments, we also evaluate a much complex regressor as a random forest (RF) in conjunction with MANIAC, as
suggested in (Wicker et al., 2016). The rationale behind this is that a linear model, such as RR, could be at disadvantage on a compressed representation space, like those learned by MANIAC. Results for the JACCARD, HAMMING and EXACT MATCH scores are reported in Table 10 along with our previous results of MPN embeddings employing RR, for the X→ EY. The performance of a linear models on our embeddings is favorably comparable to that of a non-linear one on MANIAC embeddings, proving the efficacy of MPN as feature extractors.
E X →
E Y
5 -N N MADEhX=500,hY=200 24.21 44.51 0.00 18.21 15.92 40.30 22.09 23.44 63.06 9.14 4.7 MADEhX=500,hY=500 25.11 48.76 0.00 18.38 10.75 44.87 23.08 25.69 58.57 10.21 3.8 MADEhX=1000,hY=200 24.88 44.58 0.00 20.23 13.87 41.65 22.05 25.28 63.23 9.39 4.1 MADEhX=1000,hY=500 25.53 49.79 0.00 16.35 14.43 44.85 24.53 24.66 59.45 9.51 3.8 SPNinner → MPNfull 35.98 57.79 0.00 25.46 7.24 50.13 28.75 34.25 63.60 14.81 1.6 SPNinner → MPNinner 31.93 56.03 0.00 23.44 6.25 47.58 25.94 30.06 61.36 13.16 2.7
0.0 0.2 0.4 0.6 0.8 0.15 0.20 0.25 0.30 0.35
Arts
0.0 0.2 0.4 0.6 0.8 0.60 0.65 0.70 0.75 MPE eval
Business
0.0 0.2 0.4 0.6 0.8 0.20 0.21 0.21 0.22
Cal500
0.0 0.2 0.4 0.6 0.8 0.20 0.30 0.40 0.50 0.60
Emotions
0.0 0.2 0.4 0.6 0.8 0.52 0.54 0.56 0.58
Flags
0.0 0.2 0.4 0.6 0.8 0.30 0.40 0.50 0.60
Health
0.0 0.2 0.4 0.6 0.8 0.22
0.24
0.26 Human
0.0 0.2 0.4 0.6 0.8 0.26 0.27 0.28 0.29 0.30
Plants
0.0 0.2 0.4 0.6 0.8 0.20 0.30 0.40 0.50 0.60 0.70
Scene
0.0 0.2 0.4 0.6 0.8 0.35 0.40 0.45 0.50
Yeast
Figure 4: Average test JACCARD scores (y axis) obtained by imputing different percentages of missing random embedding components (x axis) for the X→ EY setting on all datasets by employing MPE inference (orange crosses) or the bottom-up evaluation imputation schemes (blue squares).
Arts
Business
Cal500
Yeast

The authors propose and evaluate using SPN's to generate embeddings of input and output variables, and using MPN to decode output embeddings to output variables. The advantage of predicting label embeddings is to decouple dependencies in the predicted space. The authors show experimentally that using SPN based embeddings is better than those produced by RBM's.
This paper is fairly dense and a bit hard to read. After the discussion, the main contributions of the authors are:
1. They propose the scheme of learning SPN's over Y and then using MPN's to decode the output, or just SPNs to embed X.
2. They propose how to decode MPN's with partial data.
3. They perform some analysis of when their scheme will lead to perfect encoding/decodings.
4. They run many, many experiments comparing various ways of using their proposed method to make predictions on multi-label classification datasets.
My main concerns with this paper are as follows:
- The point of this paper is about using generative models for representation learning. In their experiments, the main task is discriminative; e.g. predict multiple Y from X. The only discriminative baseline is a L2 regularized logistic regression, which does not have any structure on the output; it'd be nice to see how a discriminative structured prediction method would do, such as CRF or belief propagation.
- The many experiments suggest that their encoder/decoder scheme is working better than the alternatives; can you please give more details on the relative computation complexity of each method?
- One thing I'm still having trouble understanding is *why* this method works better than MADE and the other alternatives. Is it learning a better model of the distribution of Y? Is it better at separating out correlations in the output into individual nodes?  Does it have larger representations?
- I think the experiments are overkill and if anything, they way they are presented detract from the paper. There's already far too many numbers and graphs presented to be easy to understand.  If I have to dig through hundreds of numbers to figure out if your claim is correct, the paper is not clear enough. And, I said this before in my comments, please do not refer to Q1, Q2, etc. -- these shortcuts let you make the paper more dense with fewer words but at the cost of readability.
I *think* I convinced myself that your method works...I would love to see a table that shows, for each condition: (A) a baseline X->Y, (B) one *average* result across datasets for your method, and (C) one *average* result from a reasonable best competitor method. Please show for both the exact match and hamming losses, as that will demonstrate the gap between independent linear prediction and structured prediction. That would still be plenty of numbers but would make it much easier for the reader to verify your claims and you can put everything else in the Appendix.  E.g. something like:
Input | Predicted Output | Decoder | Hamming | Exact Match
----
X | P(Y) | CRF | xx.xx | xx.xx   (this is your baseline)
SPN E_X | P(Y) | n/a | xx.xx | xx.xx
X | SPN E_Y | MPN | xx.xx | xx.xx  (given X, predict E_Y, then decode it with an MPN)
Does a presentation like that make sense? It's just really hard and time-consuming for me as a reviewer to verify your results, the way you've laid them out currently.


HARD MONOTONIC ATTENTION
We present a supervised sequence to sequence transduction model with a hard attention mechanism which combines the more traditional statistical alignment methods with the power of recurrent neural networks. We evaluate the model on the task of morphological inflection generation and show that it provides state of the art results in various setups compared to the previous neural and non-neural approaches. Eventually we present an analysis of the learned representations for both hard and soft attention models, shedding light on the features such models extract in order to solve the task.
1 INTRODUCTION
Neural sequence to sequence transduction became a prominent approach to natural language processing tasks like morphological inflection generation (Faruqui et al., 2016) and automatic summarization (Rush et al., 2015) among others. A common way to improve the vanilla encoder-decoder framework for sequence to sequence tasks is the (soft) attention mechanism (Bahdanau et al., 2014) which enables the decoder to attend at specific elements in the encoded sequence, overcoming the issues in encoding very long sequences to a single vector.
It was also shown that the attention mechanism effectively learns an alignment between the input and the output sequences which is naturally found in the data, and practically learns this alignment to attend at the relevant elements of the input. However, in many NLP tasks like automatic transliteration or morphological inflection generation, the data is roughly monotonically aligned – meaning the ability of the soft attention model to attend at the entire input sequence may be sub-optimal for such tasks, while also requiring a relatively large amount of training examples which is not always available (especially for morphologically rich, low resource languages).
There have been several works on neural sequence transduction with monotonic assumptions. One approach is to train an alignment-aware RNN-based transducer which is composed of two independent RNN’s – one over the input sequence and one over the output sequence. The output distribution is computed by feeding a pair of aligned RNN states through an MLP, where the alignment is defined by null symbols in the output (Graves, 2012) or by a parameterized transition probability (Yu et al., 2016). In both cases training is performed by marginalizing over all possible alignments using a forward-backward procedure. This approach lacks an attention mechanism, as a dependency between the input and output RNN’s would make the inference intractable. Other related approaches employ modifications to the soft-attention mechanism, like attending on a fixed sized window over the input sequence (Jaitly et al., 2015) or ”smoothing” and ”sharpening” the soft-attention weight distribution in different manners (Chorowski et al., 2015). These works are motivated by the need to attend over the very long input sequences found in speech recognition. We suggest that for shorter input sequences like the characters of a word in natural language, a simple, hard attention mechanism over the elements of a bi-directional encoder may be sufficient.
More traditional approaches to monotonic sequence transduction in the NLP literature were handengineered finite state transducers (FST) (Koskenniemi, 1983; Kaplan & Kay, 1994) which relied on expert knowledge, or weighted finite state transducers (Mohri et al., 1997; Eisner, 2002) which combined expert knowledge with data-driven parameter tuning. While the FST approaches may
work well even on small datasets due to their engineered structure, it may be cumbersome to use them while conditioning on the entire output history as it requires a very large set of states, resulting in a model conditioning only on the last predicted output symbol (Rastogi et al., 2016).
We propose a model which handles the above issues by directly modeling a monotonic alignment between the input and output sequences which is used to perform hard attention. The model consists of an encoder-decoder neural network with a dedicated control mechanism: in each step, the decoder is fed with a single attended input state and either writes a symbol to the output sequence or advances the attention pointer to the next input state from the bi-directionally encoded sequence, as described visually in Figure 1.
This modeling suits the natural monotonic alignment between the input and output very well, as the network learns to attend at the relevant inputs before writing the output which they are aligned to. A bi-directional encoder together with the hard attention mechanism enables to condition on the entire input sequence, as each element in the input is represented using a concatenation of a forward LSTM and a backward LSTM over the input sequence. Since each element representation is aware of the entire context, non-monotone relations are also captured, which is important in tasks where segments in the output sequence are a result of long range dependencies in the input sequence. The recurrent nature of the decoder, together with a dedicated feedback connection that passes the last prediction to the next decoder step explicitly, enables the model to also condition on the entire output history at each prediction step. The hard attention mechanism allows the network to jointly align and transduce while using a focused representation at each step, rather then the weighted sum of representations used in the soft attention model. A simple training procedure using independently learned alignments enables training the network with correct alignments from the first gradientbased update, using a convenient cross-entropy loss.
To evaluate our model, we perform extensive experiments on three previously studied datasets for the morphological inflection generation task, which involves generating a target word (e.g. “härtestem”, the German word for “hardest”), given a source word (e.g. “hart”, the German word for “hard”) and the morpho-syntactic attributes of the target (POS=adjective, gender=masculine, type=superlative etc.). Several studies showed that inflection generation is beneficial for phrase-based machine translation (Chahuneau et al., 2013) and more recently for neural machine translation (Garcı́a-Martı́nez et al., 2016). We show that while our model is on par or better than the previous neural and nonneural state-of-the-art models on the task, it is also performing significantly better for very small training sets, being the first neural model to surpass the performance of a weighted FST model with latent variables specifically tailored for the task (Dreyer et al., 2008). Finally, we analyze and compare our model and the soft attention model, showing how they function very similarly with respect to the alignments and representations they learn, in spite of our model being much simpler.
2 THE HARD ATTENTION MODEL
2.1 MOTIVATION
We would like to transduce the input sequence, x1:n ∈ Σ∗x into the output sequence, y1:m ∈ Σ∗y , where Σx and Σy are the input and output vocabularies, respectively. Imagine a machine with readonly, random access to the encoding of the input sequence, and a single pointer that determines the current read location. We can then model the sequence transduction as a series of write operations and pointer movement operations. In the case where the alignment between the sequences is monotonic, the pointer movement can be controlled by a single “move one step forward” operation (step) which we add to the output vocabulary. We implement this behavior using an encoder-decoder neural network, with a control mechanism which determines in each step of the decoder whether it is time to predict an output symbol or promote the attention pointer the next element of the encoded input.
2.2 MODEL DEFINITION
In prediction time, we seek the output sequence y1:m ∈ Σ∗y , for which:
y1:m = arg max y′
p(y′|x1:n, f) (1)
Where: x ∈ Σ∗x is the input sequence and: f = {f1, ..., fm} is a set of features influencing the transduction task (for example, in the inflection generation task these would be the desired morphosyntactic features of the output sequence). Since we want our model to force a monotonic alignment between the input and the output, we instead look for a sequence of actions: s1:q ∈ Σ∗s , where: Σs = Σy ∪{step}. This sequence is the step/write action sequence required to go from x1:n to y1:m according to the monotonic alignment between them. In this case we define:
s1:q = arg max s′ p(s′|x1:n, f) = arg max s′ ∏ s′i∈s′ p(s′i|s′0...s′i−1, x1:n, f) (2)
We can then estimate this using a neural network:
s1:q = arg max s′ NN(x1:n, f,Θ) (3)
Where the network’s parameters Θ are learned using a set of training examples. We will now describe the network architecture.
2.3 NETWORK ARCHITECTURE
Notation We use bold letters for vectors and matrices. We treat LSTM as a parameterized function LSTMθ(x1...xn) mapping a sequence of input vectors x1...xn to a an output vector hn.
Encoder For every element in the input sequence: x1:n = x1...xn, we take the corresponding embedding: ex1 ...exn , where: exi ∈ RE . These embeddings are parameters of the model which will be learned during training. We then feed the embeddings into a bi-directional LSTM encoder (Graves & Schmidhuber, 2005) which results in a sequence of vectors: x1:n = x1...xn, where each vector: xi = [LSTMforward(ex1 , ex2 , ...exi),LSTMbackward(exn , exn−1 ...exi)] ∈ R2H is a concatenation of the forward LSTM and the backward LSTM outputs when fed with exi .
Decoder Once the input sequence is encoded, we feed the decoder RNN, LSTMdec, with three inputs at each step:
1. The current attended input, xa ∈ R2H , initialized with the first element of the encoded sequence, x1.
2. A set of feature embeddings that influence the generation process, concatenated to a single vector: f = [f1...fm] ∈ RF ·m.
3. yi−1 ∈ RE , which is the embedding of the predicted output symbol in the previous decoder step.
Those three inputs are concatenated into a single vector zi = [xa, f ,yi−1] ∈ R2H+F ·m+E , which is fed into the decoder, providing the decoder output vector: LSTMdec(z1...zi) ∈ RH . Finally, to
model the distribution over the possible actions, we project the decoder output to a vector of |Σŷ| elements, followed by a softmax layer:
p(si = cj) = softmax j(R · LSTMdec(z1...zi) + b) (4)
Control Mechanism When the most probable action is step, the attention is promoted so xa contains the next encoded input representation to be used in the next step of the decoder. This process is demonstrated visually in Figure 1.
2.4 TRAINING THE MODEL
For every example: (x1:n, y1:m, f) in the training data, we should produce a sequence of step and write actions s1:q to be predicted by the decoder, which is dependent on the alignment between the input and the output – the network must attend at all the input elements aligned to an output element before writing it. While recent work in sequence transduction advocate jointly training the alignment and the decoding (Bahdanau et al., 2014; Yu et al., 2016), we instead show that in our case it is worthwhile to decouple these stages and learn the hard alignment before hand, using it to guide the training of the encoder-decoder network and enabling the use of correct alignments for the attention mechanism from the beginning of the network training process. For that purpose, we first run a character level alignment process on the training data. We use the character alignment model of Sudoh et al. (2013) which is based on a Chinese Restaurant Process which weights single alignments (character-to-character) in proportion to how many times such an alignment has been seen elsewhere out of all possible alignments. Specifically, we use the implementation provided by the organizers of the SIGMORPHON2016 shared task.1 Once we have the character level alignment per input-output sequence pair in the training set, we deterministically infer the sequence of actions s1:q that results in the desired output by attending at all the input elements aligned to an output element (using the step action) before writing it. We then train the network to predict this sequence of actions by using a conventional cross entropy loss function per example:
L(x1:n, y1:m, f,Θ) = − ∑
sj∈s1:q
log softmax j(R · LSTMdec(z1...zi) + b) (5)
3 EXPERIMENTS
We perform extensive experiments with three previously studied morphological inflection generation datasets to evaluate our hard attention model in various settings. In all experiments we report the results of the best performing neural and non-neural baselines which were previously published on those datasets to our knowledge. The implementation details for the models are available in the supplementary material section of this paper. The source code for the models is available on github.2
CELEX In order to examine if our model fits the task, we first evaluate it on a very small dataset to see if it avoids the tendency to overfit on few training examples. For this purpose we report exact match accuracy on the German inflection generation dataset compiled by Dreyer et al. (2008) from the CELEX database (Baayen et al., 1993). The dataset includes only 500 training examples for each of the four inflection types: 13SIA→13SKE, 2PIE→13PKE, 2PKE→z, and rP→pA which we refer to as 13SIA, 2PIE, 2PKE and rP, respectively.3 We compare our model to three competitive baselines that reported results on this dataset: the Morphological Encoder-Decoder (MED) of Kann & Schütze (2016b) which is based on the soft-attention model of Bahdanau et al. (2014), the neural-weighted FST of Rastogi et al. (2016) which uses stacked bi-directional LSTM’s to weigh its arcs (WFST), and the model of Dreyer et al. (2008) which uses a weighted FST with latent-variables structured particularly for morphological string transduction tasks (LAT). Following previous reports on this dataset, we use the same data splits as Dreyer et al. (2008), dividing the data for each inflection type into five folds, each consisting of 500 training, 1000 development and 1000 test examples. We train a separate model for each fold and report exact match accuracy, averaged over the five folds.
1https://github.com/ryancotterell/sigmorphon2016 2https://github.com/roeeaharoni/morphological-reinflection 3The acronyms stand for: 13SIA=1st/3rd person, singular, indefinite, past;13SKE=1st/3rd person, subjunctive, present; 2PIE=2nd person, plural, indefinite, present;13PKE=1st/3rd person, plural, subjunctive, present; 2PKE=2nd person, plural, subjunctive, present; z=infinitive; rP=imperative, plural; pA=past participle.
Wiktionary To neutralize the negative effect of very small training sets on the performance of the different learning approaches, we also evaluate our model on the dataset created by Durrett & DeNero (2013), which contains up to 360k training examples per language. It was built by extracting Finnish, German and Spanish inflection tables from Wiktionary, used in order to evaluate their system based on string alignments and a semi-CRF sequence classifier with linguistically inspired features. We also used the expansion made by Nicolai et al. (2015) to include French and Dutch inflections as well. Their system also performs an align-and-transduce approach, extracting rules from the aligned training set and applying them in inference time with a proprietary character sequence classifier. In addition to those systems we also compare to the results of the recent neural approaches of Faruqui et al. (2016), which did not use an attention mechanism, and Yu et al. (2016), which coupled the alignment and transduction tasks, requiring a beam search decoding procedure.
SIGMORPHON As different languages show different morphological phenomena, we also experiment with how our model copes with this variety using the morphological inflection dataset from the SIGMORPHON2016 shared task (Cotterell et al., 2016). Here the training data consists of ten languages, with five morphological system types (detailed in Table 3): Russian (RU), German (DE), Spanish (ES), Georgian (GE), Finnish (FI), Turkish (TU), Arabic (AR), Navajo (NA), Hungarian (HU) and Maltese (MA) with roughly 12,800 training and 1600 development examples per language. We compare our model to two soft attention baselines on this dataset: MED (Kann & Schütze, 2016a), which was the best participating system in the shared task, and our implementation of the global (soft) attention model presented by Luong et al. (2015).
4 RESULTS
On the low resource setting (CELEX), our model significantly outperforms both the recent neural models of Kann & Schütze (2016b) and Rastogi et al. (2016) and the morphologically aware latent variable model of Dreyer et al. (2008), as detailed in Table 1. It is also, to our knowledge, the first model that surpassed in overall accuracy the latent variable model on this dataset. We explain our advantage over the soft attention model by the ability of the hard attention control mechanism to harness the monotonic alignments found in the data, while also conditioning on the entire output history which wasn’t available in the FST models. Figure 2 plots the train-set and dev-set accuracies of the soft and hard attention models as a function of the training epoch. While both models perform similarly on the train-set (with the soft attention model fitting it slightly faster), the hard attention model performs significantly better on the dev-set. This shows the soft attention model’s tendency to overfit on the small dataset, as it has significantly more parameters and modeling power and is not enforcing the monotonic assumption of the hard attention model.
On the large training set experiments (Wiktionary), our model is the best performing model on German verbs, Finnish nouns/adjectives and Dutch verbs, resulting in the highest reported average accuracy across all the inflection types when compared to the four previous neural and non-neural state of the art baselines, as detailed in Table 2. This shows the robustness of our model also with large amounts of training examples, and the advantage the hard attention mechanism provides over the encoder-decoder approach of Faruqui et al. (2016) which does not employ an attention mech-
anism. Our model is also significantly more accurate than the model of Yu et al. (2016), showing the advantage in using independently learned alignments to guide the network’s attention from the beginning of the training process.
As can be seen in Table 3, on the SIGMORPHON 2016 dataset our model performs better than both soft-attention baselines for the suffixing+stem-change languages (Russian, German and Spanish) and is slightly less accurate than our implementation of the soft attention model on the rest of the languages, which is now the best performing model on this dataset to our knowledge.
We explain this by looking at the languages from a linguistic typology point of view, as detailed in Cotterell et al. (2016). Since Russian, German and Spanish employ a suffixing morphology with internal stem changes, they are more suitable for monotonic alignment as the transformations they need to model are the addition of suffixes and changing characters in the stem. The rest of the languages in the dataset employ more context sensitive morphological phenomena like vowel harmony and consonant harmony, which require to model long range dependencies in the input sequence which better suits the soft attention mechanism. While our implementation of the soft attention model and MED are very similar model-wise, we hypothesize that our soft attention results are better due to the fact that we trained the model for 100 epochs and picked the best performing model on the development set, while the MED system was trained for a fixed amount of 20 epochs (although trained on both train and development data).
5 ANALYSIS
5.1 COMPARISON OF LEARNED ALIGNMENTS
In order to see if the alignments our model predict fit the monotonic alignment structure found in the data, and are they more suitable for the task when compared to the alignments found by the soft attention model, we examined alignment predictions of the two models from the CELEX dataset, depicted in Figure 3. First, we notice the alignments found by the soft attention model are also monotonic, encouraging our modeling approach for the task. We also notice how our model learns to handle morphological phenomena like deletion, as can be seen on the right part of Figure 3, showing
the alignments for the inflection: legte→lege. This inflection requires the model to delete the fourth character of the input sequence. We can see the model learned to delete by reading and writing each character until it reaches the t character, which is deleted by performing two consecutive step operations. Another notable morphological transformation is the one-to-many alignment, found on the example in the left: flog→fliege, where the model needs to transform a character in the input, o, to two characters in the output, ie. We can see the model learns to do this by performing two consecutive write operations after the step operation of the relevant character to be replaced. We also notice that in this case, the soft attention model performs a slightly different alignment by aligning the character i to o and the character g to the sequence eg, which is not the expected alignment in this case from a linguistic point of view.
5.2 LEARNED REPRESENTATIONS ANALYSIS
When witnessing the success of the hard and soft attention models in sequence to sequence transduction, the following questions arise: how does the models manage to learn monotonic alignments? perhaps the network learns to encode the sequential position as part of its encoding of an input element? In an attempt to answer those questions, we performed the following analysis. We took 500 continuous character representations in context from each model, where every representation is a vector in R200 which is the output of the bi-LSTM encoder of the model. Every vector of this form carries information of a specific character with its context. We perform dimension reduction to reduce those two sets of vectors, each set in: R500×200 into: R500×2 using SVD. We then plot the 2D character-in-context representations and color them in two ways. First, we color the 2D representations by the characters they represent, where we have different color for each character in the alphabet (Figure 4). In the second plot we color the representations by the location of the character they represent in the input sequence: here blue implies the character is closer to the beginning of the sequence and red implies it is closer to the end (Figure 5).
We can see that while both models tend to cluster similar character representations together (Figure 4), the hard attention model tends to have more dense character clusters. This is explained by looking at the location information in Figure 5: while both models encode the positional information to some extent, this information is much more pronounced in the soft-attention mechanism, where the X dimension correlates very strongly with the position information. It seems that the soft-attention mechanism encourages the encoder to encode positional information in its representation. In contrast, our hard-attention model has other means of obtaining the position information in the decoder using the step actions, and indeed does not encode it as strongly in the continuous representations. This behavior allows it to perform well even with fewer examples, as the location information is represented in the network training phase implicitly using the step actions.
6 RELATED WORK
Previous works on neural sequence transduction include the RNN Transducer (Graves, 2012) which uses two independent RNN’s over monotonically aligned sequences to compute a probability over the possible output symbols in each step, including a null symbol. The model by Yu et al. (2016) improves this approach by replacing the null symbol with a learned transition probability. Both models are trained using a forward-backward approach, marginalizing over all possible alignments. Our model differs from the above by learning the alignments independently, which enables a dependency between the encoder and decoder - the hard attention mechanism. This provided improved results while also greatly simplifying the model training, enabling learning through simple crossentropy loss. Jaitly et al. (2015) proposed the Neural Transducer model for online speech recognition, which is also trained on external alignments, similarly to our approach. They divide the input into blocks of a constant size and perform soft attention separately on each block when predicting the output symbols. Lu et al. (2016) used a combination of an RNN encoder together with a CRF layer to model the dependencies in the output sequence. A line of work on attention based speech recognition Chorowski et al. (2015); Bahdanau et al. (2016) proposed two relevant improvements to the vanilla attention mechanism: The first adds location awareness by using the previous attention weights when computing the next ones, and the second prevents the model from attending on too many or too few inputs using ”sharpening” and ”smoothing” techniques on the soft-attention weight distributions.
For the morphological inflection task, previous approaches usually make use of manually constructed Finite State Transducers (Koskenniemi, 1983; Kaplan & Kay, 1994), which require expert knowledge, or machine learning methods (Yarowsky & Wicentowski, 2000; Dreyer & Eisner, 2011; Durrett & DeNero, 2013; Hulden et al., 2014; Ahlberg et al., 2015; Nicolai et al., 2015) with specific assumptions about the set of possible processes that are needed to create the output sequence, requiring feature engineering. More recently, Faruqui et al. (2016) used encoder-decoder neural networks for the task, encoding the input sequence into a vector and decoding it one character at a time into the output sequence. Kann & Schütze (2016a;b) explored the soft attention model proposed for machine translation (Bahdanau et al., 2014) which gave the best results in the SIGMORPHON 2016 shared task (Cotterell et al., 2016). Another notable contribution is the work on weighting finite state transducers with neural context (Rastogi et al., 2016). There, the arcs of an FST are scored by optimizing a global loss function over all the possible paths in the state graph while modeling contextual features with bi-directional LSTM’s.
7 CONCLUSION
We presented the hard attention model for sequence to sequence transduction of monotonically aligned sequences and evaluated it on the well-studied task of morphological inflection generation. The model employs an explicit alignment model learned independently in training time which is used to teach a neural network to perform both alignment and transduction when decoding with a hard attention mechanism. We showed that our model performs better or on par with more complex soft attention and neural transduction models on various morphological inflection datasets, forming a new state of the art on the CELEX dataset and the Wiktionary dataset and outperforming the best system in the SIGMORPHON2016 inflection generation task. Future work may include experimenting with different external alignment methods, or applying the model to other tasks which require a monotonic align-and-transduce approach like abstractive summarization or transliteration.

The paper proposes an approach to sequence transduction for the case when a monotonic alignment between the input and the output is plausible. It is assumed that the alignment can be provided as a part of training data, with Chinese Restaurant process being used in the actual experiments.
The idea makes sense, although its applicability is limited to the domains where a monotonic alignment is available. But as discussed during the pre-review period, there has been a lot of strongly overlapping related work, such as probabilistic models with hard-alignment (Sequence Transduction With Recurrent Neural Network, Graves et al, 2012) and also attempts to use external alignments in end-to-end models (A Neural Transducer, Jaitly et al, 2015). That said, I do not think the approach is sufficiently novel.
I also have a concern regarding the evaluation. I do not think it is fair to compare the proposed model that depends on external alignment with the vanilla soft-attention model that learns alignments from scratch. In a control experiment soft-attention could be trained to match the external alignment. Such a pretraining could reduce overfitting on the small dataset, the one on which the proposed approach brings the most improvement. On a larger dataset, especially SIGMORPHON, the improvements are not very big and are only obtained for a certain class of languages.
To sum up, two main issues are (a) lack of novelty (b) the comparison of a model trained with external alignment and one without it.