Optimization Methods for Sparse Pseudo-Likelihood Graphical Model Selection
Sparse high dimensional graphical model selection is a popular topic in contemporary machine learning. To this end, various useful approaches have been proposed in the context of 1 -penalized estimation in the Gaussian framework. Though many of these inverse covariance estimation approaches are demonstrably scalable and have leveraged recent advances in convex optimization, they still depend on the Gaussian functional form. To address this gap, a convex pseudo-likelihood based partial correlation graph estimation method (CONCORD) has been recently proposed. This method uses coordinate-wise minimization of a regression based pseudo-likelihood, and has been shown to have robust model selection properties in comparison with the Gaussian approach. In direct contrast to the parallel work in the Gaussian setting however, this new convex pseudo-likelihood framework has not leveraged the extensive array of methods that have been proposed in the machine learning literature for convex optimization. In this paper, we address this crucial gap by proposing two proximal gradient methods (CONCORD-ISTA and CONCORD-FISTA) for performing 1 -regularized inverse covariance matrix estimation in the pseudo-likelihood framework. We present timing comparisons with coordinate-wise minimization and demonstrate that our approach yields tremendous payoffs for 1 -penalized partial correlation graph estimation outside the Gaussian setting, thus yielding the fastest and most scalable approach for such problems. We undertake a theoretical analysis of our approach and rigorously demonstrate convergence, and also derive rates thereof.
This paper addresses the use of pseudo-likelihood for solving (sparse) Graphical Model selection problems. The use of log likelihood for Gaussian Graphical Models has been the subject of considerable research, but this is limited to the Gaussian assumption. This has not been the case for pseudo-likelihood –based methods, and non-Gaussian graphical models. Since the Gaussian assumption is limiting, there are potentially large benefits in this line of work. At a high-level the formulation is based on a recent approach called CONCORD, which optimizes Qcon, a convex pseudo-likelihood objective. CONCORD uses coordinate-wise descent. The paper proposes proximal gradient techniques to minimize Qcon  and provides a theoretical analysis. Proximal gradient has become a common useful tool in machine learning. The methods are tested in learning non-Gaussian graphical models.The paper derives convergence rates for CONCORD-ISTA/-FISTA which is of significance, it also compares CONCORD-ISTA/FISTA and coordinate-wise minimization. Results suggest that CONCORD-ISTA is much faster than coordinate-wise, specially in high-dimensions. Since their computational complexity is similar O(p^2), a primary reason for this is the possibility of parallelizing the computation in the proposed method, while this is not the case for the coordinate-wise method. The derived converge rates are not fully consistent across pretty much all of the results. It would be great to have further discussion on this.