Theoretical guarantees for the EM algorithm when applied to mis-specified Gaussian mixture models
Recent years have witnessed substantial progress in understanding the behavior of EM for mixture models that are correctly specified. Given that model misspecification is common in practice, it is important to understand EM in this more general setting. We provide non-asymptotic guarantees for the population and sample-based EM algorithms when used to estimate parameters of certain misspecified Gaussian mixture models. Due to mis-specification, the EM iterates no longer converge to the true model and instead converge to the projection of the true model onto the fitted model class. We provide two classes of theoretical guarantees: (a) a characterization of the bias introduced due to the mis-specification; and (b) guarantees of geometric convergence of the population EM to the model projection given a suitable initialization. This geometric convergence rate for population EM implies that the EM algorithm based on n samples converges to an estimate with 1/ √ n accuracy. We validate our theoretical findings in different cases via several numerical examples.
# After authors response  I have read the authors’ responses about the references and some intuitions about  the results and the model and the model in section 4, and I thank the authors for their explanations.  I made some detailed comments about the figures and their interpretation, that go  beyond a simple size problem. I acknowledge the will of the authors to revisit both  their display and exploitation. I also acknowledge the authors’ intention to work on  the exposition and redaction in some places, previously signaled. I thank the authors for receiving these comments, I think that the paper would highly benefit from an  improvement on theses matters.  # Summary  In this paper, the authors are interested in the effect of model specifications on the convergence of the EM algorithm. Specifically, they focus on three simple cases where the formulas are tractable, and assess the bias, as well as the convergence properties of the population and sample EM.  The three cases considered are instances of Gaussian mixtures: 1. The true model has three classes, with two classes having close means, and    the fitting model has only two classes. Variance and weights of the fitted    model are fixed according to the true values. 2. The true model has three classes, including one centered in zero and with a    small associated weight. The fitting model is the same as above. Variances and    weights of the fitting models are fixed according to the true values. 3. Both the true and fitted models have two classes. The weight and variances of the    fitted model are fixed, different from the true one.  After showing some theoretical results in each situations, the authors validate heuristically their findings with associated simulation studies.  # General comments  The problem of fitting misspecified statistical models is an important one, and this paper makes a first step toward some theoretical bounds in some specific cases. Although limited and relatively simple, the three cases considered are of practical interest, and give way to consistent developments (although I did not check the proofs in the supplementary material carefully).  As a personal opinion, it felt that the paper was missing some context on the state-of-the art. Although some references are given (mainly, Balakrishnan et al. 2017), the results are not compared to the literature, which made it difficult to get an idea of the sharpness of the bounds.  In a similar way, I found no reference to the classical model selection theory,  which deal with similar problems (trying to find the 'best' number of components to keep in the fitted model). Although the focus is a bit different, it could be interesting to make some parallels. For background on model selection, see e.g.: - Massart, P. (2007). Concentration Inequalities and Model Selection. - Baraud, Y. et al (2009). Gaussian model selection with an unknown variance. AoS.  Finally, it felt like the submission could benefit from a proof-reading for punctuation, typos, English style and grammar issues. (See below for some more specific remarks.)  # Specific remarks  ## Section 2  * Table 1: maybe adding the corresponding section for each model could help   the reader (3.1, 3.2 and 4, if I'm correct). (This is just a suggestion.)  * It could be useful to add a reference for formulas for section 2.1 (EM for    a two-component Gaussian mixture).    ## Section 3  * I found the introductory paragraph of section 3 (lines 115 - 124) confusing.   It could be worth to refactor it a bit. Specifically:   * l.118: "we do not assume any knowledge about the number of component in the     true model." Maybe I missed something, but I was under the impression that     the true model was assumed to have three or two classes. It might also be     in contradiction with the first sentence of the paragraph ("... the number     of components in the true model is *larger* than that in the fitted     model.") Maybe a clarification could help.   * l.119-121: it was not clear to me whether these scenarios were assumptions,     or statements of the general nature of misspecification.   * l.122-123: "the number of components" is repeated, maybe rephrase.     * I felt that a comparison with classical bounds of the EM (that is, when the   model is correctly specified) was maybe missing. It could be interesting to   recall these bounds, and point out the differences induced by this   misspecification.  * Similarly, it seems that more comments could be made on the links between the   results of sections 3.1 and 3.2. For instance, in seems that the bounds in   Th. 1 and Cor. 1 do not depend on the disparity parameter $\rho$, while the   ones in Th. 2 and Cor. 2 do depend on $\omega$. Could the authors provide an   intuition of why this is the case ?  * l.158: should it be $\gamma = e^{-c''\eta^2}$ to match expression l.151 ?   * l.177: By multiplying the constant of Prop. 2 by 1/\eta^{1/4}, should the   constant $C_\omega = c\omega^{1/8} / \sqrt{1 - \omega}$ ? be (without the   $\sigma$) ?  ## Section 4  * It could be interesting to comment on the specific form of model (16). In   particular, why choosing an expression where both the distance between the   two Gaussians and there variance vary conjointly, through parameter $\theta*$ ?  * It is not clear from the text whether $\sigma^2$ in the fitted model (17) is   fixed or not, and, if so, to which value.   ## Simulation studies (section 5)  It was not always clear to me how the simulations illustrated the results of the previous sections.  Also, the figures are quite small, and I had to magnify Figures 2 and 3 with a factor 3 on my screen to read them properly. I'm myopic, so maybe not representative, but it could ease the reading to leave more space to them, and/or remove some of the panels (see comments below).  * I did not understand how the results presented in the third panel of each   figure (histogram) was related to the previous theoretical results. Maybe I   missed it, but it does not seem to be exploited in the text (beside its   description l.248-250). Maybe a clarification could help the reader.  * Figure 1: Maybe it could help the reader to give the value of $\bar{\theta}$.   From the graph, it seems that it is almost equal to $\theta^*$ (4 or 3), so   that there is no bias in this case. On the second panel figure 1b for   instance, the sample EM seems to converge to $\theta^* = 3$, while $\rho=0.5$   is large, so the bias should be larger, as hinted by the text (see l.258).   Maybe I read the figure wrong, but then a clarification could help.   Could a plot of bound $C_\rho$ help ?  * Figure 3a: It looks like the fitted model is unimodal (panel 1), while   from the text (see Table 1), it should have two components. Again, I might   be missing something, but then a clarification could help.  ## Discussion  Please consider proof reading this discussions, which is difficult to read in its current state because of the many typos (see below).  # Possible typos  Below are a few possible typos that I noted along the way:  * l.52: lower case We ? * l.65: show appears two time, maybe rephrase ? * l.82: should it be "... performance of *the* EM algorithm … " (add 'the') ?   Also I think lines 89 ('the location'), 91 ('the EM'), 115 ('the EM'), 133   ('of the EM'), 168 ('the EM'), 176 ('the results'), 177 ('the bias'), 178   ('the EM'a, 270 ('the behavior of the EM')). * l.91: maybe rephrase: "... may not belong to the fit model, …" * l.91: maybe rephrase: "... the best possible estimator that can be achieved   using EM can be …" ('can' appears two times). * l.168: missing period at the end of the sentence.  * l.270: "general" appears two times. * l.271: "…for the same. illustrate..." maybe a piece is missing ? * l.273: "... is focused … " ? * l.274: "... to ask if we fit a model with scale as well." maybe rephrase ? Also: * "2 or 3 component(s) mixture": this formula appears in several places,   with 2 and 3 sometimes spelled out, and with or without an -s at "component".   It might be good to unify. * Why is there a lonely section 2.1 ?