Learning Additive Exponential Family Graphical Models via ℓ 2,1 -norm Regularized M-Estimation
We investigate a subclass of exponential family graphical models of which the sufficient statistics are defined by arbitrary additive forms. We propose two ℓ 2,1 -norm regularized maximum likelihood estimators to learn the model parameters from i.i.d. samples. The first one is a joint MLE estimator which estimates all the parameters simultaneously. The second one is a node-wise conditional MLE estimator which estimates the parameters for each node individually. For both estimators, statistical analysis shows that under mild conditions the extra flexibility gained by the additive exponential family models comes at almost no cost of statistical efficiency. A Monte-Carlo approximation method is developed to efficiently optimize the proposed estimators. The advantages of our estimators over Gaussian graphical models and Nonparanormal estimators are demonstrated on synthetic and real data sets.
This paper proposes parameter estimators for exponential family graphical models in which the univariate and pairwise sufficient statistics are representable as linear combinations of pre-defined basis functions.  Two regularized maximum likelihood estimators are presented, the first for joint estimation of all parameters, the second for conditional estimation of the parameters associated with each node.  A Monte-Carlo approximation to certain expectations is used to simplify the algorithms. Finite-sample statistical guarantees are established on the estimation error, which agree with existing results on Gaussian and nonparanormal graphical models. Experimental results show greater flexibility in modeling nonlinear interactions between variables. I think this last aspect is the main strength of the paper compared to existing approaches, in addition to the theoretical results.
Most important: 1. In Theorem 1, the error rate is as expected since max{q,r} ||theta*||_{2,0} is approximately the number of parameters in the model.  I think the factor of max{q,r} should not be omitted from Section 1.1, last paragraph so as not to be misleading.  I am surprised however that the same factor of max{q,r} is not present in the error rate in Theorem 2, being replaced instead by (q+r) inside the logarithm. This result should be checked since if I understand correctly, the number of parameters is still something like q or r times ||theta*_s||_{2,0}. If Theorem 2 is correct as currently stated then a more thorough justification in words is needed.  Major comments:  2. I think it may be inaccurate to call the model "semi-parametric" because the set of basis functions is finite and fixed.  Moreover the dimensions of the bases used in the experiments are small (< 10).  3. Related to the point above, I believe the proposed approach shifts the burden onto choosing an appropriate set of basis functions for the data at hand. For the simulated data in Section 4.1, the bases are chosen to be a perfect fit or at least a good fit to the true form of the interactions.  A case where the basis is not well-matched would be of interest.  In general, the choice of basis is an issue deserving of more attention.  4. The definition of groups is not clear in the Notations subsection and remains so until below (7) where the (2,1)-norm is defined. Why are node groups penalized the same as edge groups?  5. In Proposition 1, definition of sigmahat, should it be sigmahat^2 instead? The RHS looks like a variance.  6. I am not seeing how the number of Monte Carlo samples m was chosen in the experiments.  7. In Section 4.2, the use of stock category as ground truth for links seems a bit questionable since stocks in different categories could also be strongly dependent, while not all stocks in the same category may be dependent.  Minor issues:  8. Section 1, paragraph 1: "undirected graph consists" --> "undirected graph consisting"  9. In Section 1.2, the second half of the paragraph could be better organized. Do [9,27] belong to the nonparanormal approach?  It appears that [7,20,8] are fully non-parametric methods.  This could be emphasized with a brief discussion of the advantages/disadvantages of non-parametric models vs. semi-parametric ones and the proposed model.  10. Below (10) in the definition of ||theta||_{2,1}, I think the summation should be over t in N(s).  11. Just above Proposition 1, is "A.1" an appendix in the supplementary material?  12. In Theorems 1 and 2, the constants c_0 and ctilde_0 are undefined. In Theorem 2, there may be missing tildes e.g. on delta.  13. In Section 4.1, is performance reported on the training sample, after the tuning sample has been used to select lambda_n?