Decomposing Parameter Estimation Problems
We propose a technique for decomposing the parameter learning problem in Bayesian networks into independent learning problems. Our technique applies to incomplete datasets and exploits variables that are either hidden or observed in the given dataset. We show empirically that the proposed technique can lead to orders-of-magnitude savings in learning time. We explain, analytically and empirically, the reasons behind our reported savings, and compare the proposed technique to related ones that are sometimes used by inference algorithms.
This paper proposes a method for finding a max likelihood estimate of the parameters of a Bayesian Network (BN) with hidden variables by decomposing the global optimization problem into a set of independent sub-problems.  The main motivation for this decomposition is that in a data set the variables are either always hidden, or always observed - i.e. variables aren’t missing at random. By assuming the learning problem has this property, the authors propose a partitioning of the BN’s parameters in the same way that one decomposes the BN parameter estimation problem in the fully observed case.  They then demonstrate the benefit of this decomposition empirically by comparing their method to off-the-shelf implementations of the EM algorithm - perhaps the default approach for BN parameter estimation in missing data problems.Overall, I found this paper to be very well written and easy to follow, with clear and concise definitions and plenty of well placed examples.  My two main criticisms of this paper are with its novelty and significance.  The two main ideas appear to be: 1) that a BN with fixed structure can be decomposed into smaller, easier to learn sub-problems; and 2) that the dataset can be compressed by maintaining counts for each of these subproblems. The idea of exploiting a network’s structure to expedite inference/learning has been studied in various hidden-variable architectures, such as RBMs and HMMs. The proposed decomposition falls out naturally, for example, in an RBM architecture.  In contrast, the proposed method provides no-benefit in the case of an HMM architecture as there is no decomposition.  Along these lines, I felt that the claims made in the experimental suggestion were a bit exaggerated.  It seems a bit unfair to take on off-the-shelf EM method, which has no knowledge of the fixed pattern of missingness, and comparing to a method designed explicitly to exploit such a missingness pattern. The reason these general purpose methods can’t take advantage of the “data projection” operator is because they do not assume that the missingness pattern is fixed!  Rather than just reporting the speed-up, it would have been better to show how the speed-up degrades as the decomposition becomes less prominent. Increasing the percentage of observed variables does not tell us if the number of sub-networks decreases or increases, or anything about the size of the sub-networks relative to the entire network. In addition, how does the method perform as the size of the data set is changed. Finally, I think the authors should have addressed the role of regularization. What type of priors on the hidden parameters are supported by the proposed method and which are not.Detailed Comments:- Line 261: what is meant by classical EM?  EM runs inference on each training instance, while D-EM runs on a sub-network using summarized/compressed data.- Line 267: Why does making all unobserved variables hidden make the problem harder?- Line 270: The caption of Figure 3 suggests that I should see three differently bars, one for each chain length.  However, I only see one set of bars.- Line 284: In Table 1, provide some summary information of the UAI networks, e.g. number of variables, tree-width, variable arity, etc…- Line 324: In Figure 4, the center and right charts were too small to reveal anything.  Perhaps a log-log plot would be better.