Conditional Random Field Autoencoders for Unsupervised Structured Prediction
We introduce a framework for unsupervised learning of structured predictors with overlapping, global features. Each input's latent representation is predicted conditional on the observed data using a feature-rich conditional random field (CRF). Then a reconstruction of the input is (re)generated, conditional on the latent structure, using a generative model which factorizes similarly to the CRF. The autoencoder formulation enables efficient exact inference without resorting to unrealistic independence assumptions or restricting the kinds of features that can be used. We illustrate connections to traditional autoencoders, posterior regularization, and multi-view learning. We then show competitive results with instantiations of the framework for two canonical tasks in natural language processing: part-of-speech induction and bitext word alignment, and show that training the proposed model can be substantially more efficient than a comparable feature-rich baseline.
REPLY TO AUTHOR RESPONSEThanks for the response.  You write: "It is possible we missed a related paper in the neural network literature. We would be grateful if you could provide citations of neural architectures which used hand-crafted features for unsupervised learning of structured outputs."  This may have misunderstood my point.  I was really just making the same point as reviewer_26 (you responded to him or her).  But as it happens, I know of one branch of work loosely related to your question.  Stoyanov et al. (AISTATS 2011, NAACL 2012) showed how to derive the structure of a sum-product feed-forward network by unrolling loopy sum-product belief propagation on a given graphical model.  Each potential function in the graphical model can be parameterized in terms of hand-crafted features of its incident variables (e.g., it may be log-linear).  These potentials convert into parametric weights of the network, reflecting the hand-crafted features.  In addition, the structure of the network reflects the structure of the graphical model.  The network is then trained by backprop to minimize prediction error at designated output variables.  Is this "unsupervised"?  Well, like the original graphical model, the resulting feed-forward network learns to predict the latent variables of the graphical model from the input variables in a way that is helpful for predicting the output variables.  This could be regarded as "unsupervised" learning of a structure over the latent variables given the (input, output) variables.  If the graphical model were given an auto-encoder structure where output = input, then their method would result in an unsupervised system very much like yours.SUMMARYUnsupervised learning of latent structure y given input x.  This is a bottleneck method using an autoencoder p(x | y) p(y | x).  Specifically, p(y | x) is a domain-specific CRF, while p(x | y) appears to be a simple independent model \prod_i p(x_i | y_i).  The  y are compared with human annotations.It is efficient to compute the objective (2) and its gradient.  The objective is regularized log-loss of the reconstruction, marginalizing over y.  More generally, the autoencoder is p(\hat{x} | y,phi) p(y | \hat{x},x,phi) where \hat{x} is some ad hoc simplified version of x and phi is side information.  COMMENTSThe results are quite good.  In particular, the method seems to produce fairly stable improvements across multiple languages (8 for POS tagging, 3 for alignment as evaluated by MT).  However, the paper and supplementary material appear to have been written hastily.  It is often hard to figure out the details of what was done experimentally. The method is formally attractive and might turn out to be an important building block in learning certain kinds of latent structure.  (Particularly if reconstructing the input x is only one task in a multi-task learning setting, so that y should also be useful to predict other, supervised properties of the input x.  The authors don't discuss this.)However, the authors don't do a good job of explaining why the method will do a good job of unsupervised learning.  Their objective is to match human annotations, not just to learn representations that are useful for some other task.  The usual difficulty in this setting is that the latent variables end up learning the "wrong" properties of the data -- not the properties that the humans annotated.  The authors say something about this problem at L067, but they don't really explain why their method should do any better.  Nor do they do any analysis of the experimental results to understand what was learned and why.  There are single-number comparisons with other methods, but no breakdowns or targeted experiments aimed at understanding what is going on.  It's just left as a happy mystery.  (There are no learning curves, either.)  It would also be helpful to explore whether the improved results are achieved because they reduce model error or reduce search error.  For example, suppose method A (previous) and method B (this paper) find latent distributions p_A and p_B over the latent variables y.  If each method is initialized separately at p_A and p_B, then does method A's objective prefer p_A and method B's objective prefer p_B, which implies that the objective is better?  Or do they both prefer p_A or both prefer p_B, which suggests that the different stems from a search bias?The authors suggest (L262) that their autoencoder architecture is more appropriate in some respects than neural autoencoders, although this assertion is not defended by discussion, empirical comparison, or error analysis.  I think the authors are mainly motivated by the fact that the p(y | x) model can build on a long line of efficient, feature-engineered work on supervised structured prediction work in NLP.  But there are many architectures that could use such features, including neural architectures.DETAILED COMMENTS TO AUTHORThe p(\hat{x} | y) model is described only obliquely.  I have a good guess what it is, but please spell it out!What exactly are you evaluating?  At L172 you mention the distribution p(y | x,\hat{x},phi), under which, presumably, the probability of a given latent y is proportional to p(\hat{x} | y,phi) p(y | x,phi).  Does this mean that you are evaluating the 1-best y from this distribution?  One or many samples from this distrubtion?  Something else?It's not possible to really understand the features.  For example, supplement L103 says that auto+full features include functions of the y_i, but supplement Table 2 doesn't show what those features are.  It is also not clear whether "x_i, x_{i-1}" in the "full" column of that table is talking about conjoined features or is a list of features.  And it's not clear whether the full model also includes all of the h&k features or not.L147: Even before you introduced phi, you already allowed side information for the encoding phase, namely anything in x that was not part of \hat{x}.  So perhaps you want to say that phi is side information that is (also) available in the reconstruction phase?Possibly relevant is http://www.cs.cmu.edu/~nasmith/papers/gimpel+smith.naacl12b.pdf .Suggestion on how to handle the supplementary material: Currently this is formatted as a separate paper, which is confusing.  Please set it up as appendices to the main paper, following the bibliography, without duplicate material.  This will allow you to have a single bibliography, a single set of figure and equation numbers, and crossrefs with the main paper.  After running through pdflatex, use pdftk to divide the document into the main paper and the supplement.