Structure Regularization for Structured Prediction
While there are many studies on weight regularization, the study on structure regularization is rare. Many existing systems on structured prediction focus on increasing the level of structural dependencies within the model. However, this trend could have been misdirected, because our study suggests that complex structures are actually harmful to generalization ability in structured prediction. To control structure-based overfitting, we propose a structure regularization framework via structure decomposition, which decomposes training samples into mini-samples with simpler structures, deriving a model with better generalization power. We show both theoretically and empirically that structure regularization can effectively control overfitting risk and lead to better accuracy. As a by-product, the proposed method can also substantially accelerate the training speed. The method and the theoretical results can apply to general graphical models with arbitrary structures. Experiments on well-known tasks demonstrate that our method can easily beat the benchmark systems on those highly-competitive tasks, achieving record-breaking accuracies yet with substantially faster training speed.
Summary: The authors propose “tag structure regularization”, a novel regularization scheme whereby at training time, a structured prediction model is penalized based on losses on sub-structures of the training example. The key is that the model is forced to make predictions for each sub-structure independently, so it cannot rely on long-reaching information from other parts of the graph. They provide analysis based on stability arguments that the regularization strength decreases bounds on generalization error and increases the convergence rate of SGD. On several sequential prediction tasks they show that varying the regularization strength can produce state-of-the-art accuracies on several tasks and at the same time speed up training.Major comments: (+ Pros, - Cons)+ On the one hand, the central idea in this paper is very interesting. The authors argue that by breaking up training samples into sub-samples during learning, we can increase the generalization of structured models. Intuitively, this makes a certain kind of sense: the structured model is regularized to rely more on local information than incoming messages from the rest of the graph. This might prevent errors from propagating, and results in a “simpler” model. Another way to think about it is that if a model can make accurate predictions without relying on passing messages, the “tag structure regularization” will choose that simpler model, while more standard approaches do not have such preference.- On the other hand, the actual theoretical analysis in this paper seems like it isn’t making the correct assumptions or taking the right approach to analysis. Yes, increasing alpha does reduce the bound -- but at the tightest setting alpha = n, the bound is still far looser than other structured prediction generalization bounds. E.g. the original MMMN paper (Taskar et al.) had a logarithmic complexity in both the multi class label size and the number of variables (l in that paper, n here). More recently London et al. used PAC Bayes analysis to show that for *templated* models, increasing the size of the example actually **decreases** generalization error: in the limit, one could learn an entire templated model from a single example. So, from that perspective, using n as the measure of structure complexity makes no sense, since in most applications (including those in this paper) feature templates are used (bi-grams, etc.). So while I believe that their analysis is technically correct, the authors must reconcile their analysis with previous work in order for this paper to make sense.- n^4 seems like an awfully large term for a generalization bound: following the supplemental, it seems like it stems from the fact that the bounds rely on decomposing loss linearly in Lemma 6, and then due to having to multiple the norms of 2 examples (which could be O(n)) in addition to that, as well as decomposing the regularization term linearly (O(n/alpha)). To me, that suggests that this is really not the right approach to take here.- The aforementioned previous work uses a more subtle measure of graph complexity in their bounds, based on concentration inequalities, that measures the maximum dependence of one variable on the others in the graph. It seems like a better approach to analysis would be to relate the novel regularization to the resulting complexity of the learned model in terms of a complexity measure like that, where one can assume some sort of templating. Instead of just assuming complexity = n.+ I hate to say this (because this idea is so trendy), but the tag structure regularization reminds me a lot of dropout: essentially, for each example, you generate new examples by removing edges in the graph. In the case of sequences, this creates disconnected components, but one could imagine more generally just removing edges. So in that sense I think this paper does help shed light on other ideas in the field.+ All that being said about the theory, the experimental results are very strong, and the idea is simple enough to be easy to experiment with and verify.