In this paper, a Gaussian conditional random field model for structured binary classification (GCRFBC) is proposed. The model is applicable to classification problems with undirected graphs, intractable for standard classification CRFs. The model representation of GCRFBC is extended by latent variables which yield some appealing properties. Thanks to the GCRF latent structure, the model becomes tractable, efficient, and open to improvements previously applied to GCRF regression. Two different forms of the algorithm are presented: GCRFBCb (GCRGBC -Bayesian) and GCRFBCnb (GCRFBC -non-Bayesian). The extended method of local variational approximation of sigmoid function is used for solving empirical Bayes in GCRFBCb variant, whereas MAP value of latent variables is the basis for learning and inference in the GCRFBCnb variant. The inference in GCRFBCb is solved by Newton-Cotes formulas for one-dimensional integration. Both models are evaluated on synthetic data and real-world data. It was shown that both models achieve better prediction performance than relevant baselines. Advantages and disadvantages of the proposed models are discussed.
The work involves modifiying gaussian conditional random fields to work for classification problems instead of regression problems. The main idea is to apply a bernoulli distribution on top of the regression values to convert them to work with binary classification problems. Two variations are discussed along with the inference and learning methodology. The inference can be done using numerical approximation and learning using variational methods and is still untracktable. Comparisons with other modeling strategies is done using experiments.
The paper is incremental and doesn't really provide improvements to learning parameters (or at least there is no theory showing this in the paper). The experiments do not seem satisfactory as discussed below.
a) Applying a bernoulli distribution on the output of the GCRF seems trivial. It is not very clear when the GCRFBCb model would be better than the GCRFBCnb. The learning procedure is untracktable and hard to follow on why this might provide better results.
b) The datasets (music classification and gene classification) don't seem to be good datasets for structured predictions i.e. the interaction needed between the nodes is not clear. Since they are multilabel problems, one could have just modeled the system with N independent nodes or design a multinomial distribution instead of only for binary classification.
c) There should be more thorough fine-tuning of other models, for e.g. in the ski lifts experiment, the CRF does much worse than logistic regression in the results. This is most likely because the parameters were not initialized properly using normal tricks like using logistic regression. Typically for truly structured problems, CRFs do better than their logistic regression counter parts. It is also not clear how the other models (CRF and SSVM) pairwise potentials were modeled.
It would really help to make this paper stronger by showing the new modeling technique does better than CRFs (that are tuned properly) on better structured datasets. It would be good to have a discussion on when this model would do worse than the other structured models and why.