On the Accuracy of Self-Normalized Log-Linear Models
Calculation of the log-normalizer is a major computational obstacle in applications of log-linear models with large output spaces. The problem of fast normalizer computation has therefore attracted significant attention in the theoretical and applied machine learning literature. In this paper, we analyze a recently proposed technique known as "self-normalization", which introduces a regularization term in training to penalize log normalizers for deviating from zero. This makes it possible to use unnormalized model scores as approximate probabilities. Empirical evidence suggests that self-normalization is extremely effective, but a theoretical understanding of why it should work, and how generally it can be applied, is largely lacking. We prove upper bounds on the loss in accuracy due to self-normalization, describe classes of input distributions that self-normalize easily, and construct explicit examples of high-variance input distributions. Our theoretical results make predictions about the difficulty of fitting self-normalized models to several classes of distributions, and we conclude with empirical validation of these predictions.
This paper provides a theoretical analysis of self-normalized log-linear models, which are log-linear models trained with a penalty to keep the partition function close to one, so that scores can be directly interpreted as probabilities. These models are being used in the NLP community when the output space is large. The authors provide 1) a simple geometric characterization of exact and approximate self-normalization 2) a natural framework to characterize the tradeoff between fitting the data and keeping the log-partition function close to zero 3) some results showing that distributions that are either close to uniform or close to degenerate can be self-normalized (without losing too much in terms of likelihood). The paper is well written, it provides intuitive explanations for the results, examples and figures. I would have liked to see more discussion for the results at the end of section 4 (looks like it was finished in a hurry). As far as I know, the results are novel. The theoretical results shed some light on the role of self-normalization. The framework developed for the analysis is of (independent) interest and could be used by other researchers, e.g. to obtain deeper characterizations of the tradeoffs involved. This work could be of interest to people working on models where dealing with the intractable log-normalizer is the key computational bottleneck (on energy based models, noise constrastive estimation, etc). One limitation is that the results presented don't directly lead to new algorithms, and it's not clear to me if they are useful to inform the work of a practitioner (e.g, how to set parameters or design models). I am also not sure about the significance and the possible implications of Theorem 4. How does the model fit (data likelihood) enter the picture? Can you say something about small perturbations of \eta_0?More details should be given about the experimental evaluations. Currently, there is not enough information to replicate the results (what is the X, Y, features used etc.)in proposition 1, should it be ||\eta||_2 <= B ?