Factorized Asymptotic Bayesian Inference for Latent Feature Models
This paper extends factorized asymptotic Bayesian (FAB) inference for latent feature models (LFMs). FAB inference has not been applicable to models, including LFMs, without a specific condition on the Hessian matrix of a complete loglikelihood, which is required to derive a "factorized information criterion" (FIC). Our asymptotic analysis of the Hessian matrix of LFMs shows that FIC of LFMs has the same form as those of mixture models. FAB/LFMs have several desirable properties (e.g., automatic hidden states selection and parameter identifiability) and empirically perform better than state-of-the-art Indian Buffet processes in terms of model selection, prediction, and computational efficiency.
Section 3: 1) we relate FAB E-steps to a convex concave procedure (CCCP) [29]. Inspired by this analysis, we propose a shrinkage acceleration method which drastically reduces computational cost in practice, and 2) we show that FAB/LFMs have parameter identifiability. This analysis offers a natural guide to the merging post-processing of latent features. Rigorous proofs and assumptions with respect to the main results are given in the supplementary materials. Notation In this paper, we denote the (i, j)-th element, the i-th row vector, and the j-th column vector of A by a ij , a i , and a ·j , respectively.
Summary: The paper presents an inference 
      algorithm for the binary latent feature model (LFM) which expresses each 
      observation x_n as a K size binary vector z_n where z_{nk} = 1 means that 
      x_n has the k-th latent feature present. The goal of inference is to learn 
      the z's for all observations (along with the other model parameters). The 
      presented EM based algorithm uses a shrinkage mechasnism to learn the z's 
      as well as infer the number of latent features (K). This is similar in 
      spirit to nonparametric Bayesian latent feature models such as the Indian 
      Buffet Process (IBP). However, the presented algorithm doesn't actually 
      use prior distributions inducing model parsimony but rather uses a model 
      selection criteria - Factorized Asymptotic Bayesian Inference (FAB) - 
      recently introduced in the context of mixture models and HMMs. THe FAB 
      idea is based on expressing the log marginal likelihood in terms of the 
      expected complete log-likelihood plus a model selection term that 
      encourages shrinkage of the number of latent features (by specifying an 
      upper-bound on K and *learning* the correct K during inference). On some 
      large datasets, the FAB is shown to outperform other inference methods 
      including Gibbs sampling, variational inference, and MAP estimation 
      methods for the IBP (in terms of running time and inference quality). 
      Quality: The technical quality seems reasonably sound and 
      algorithmic details seem correct. The ability of dealing with large 
      datasets and also learning the number of latent features is very 
      appealing. Clarity: The paper is reasonably clear in the 
      methodology part. However, some of the experimental details are somewhat 
      unclear (discussed below). Originality: The proposed method is 
      based on recently proposed framework on Factorized Asymptotic Bayesian 
      (FAB) Inference applied for mixture models and HMMs. The application of 
      FAB to latent feature models however is novel. Significance: The 
      paper is addressing an important problem (efficient inference in latent 
      feature models while also inferring the number of latent features). 
      Weak Points: - There is no discussion about the 
      limitations of the proposed algorithm. Are there any cases when the 
      algorithm might perform worse than vanilla Gibbs sampling? - There 
      is no discussion about the possible difficulties in convergence (given it 
      is an EM like procedure). - The algorithm is limited to binary 
      latent feature models (can't be applied for factor analysis or 
      probabilistic PCA). - The experimental evaluation is not thorough 
      enough (and seems flawed at some points; see comments below). - In 
      the small-data regime (when the asymptotic argument doesn't hold), it is 
      unclear how the algorithm will behave (there should be some discussion or 
      experiments). - Some other recent works on efficient inference in 
      LFMs has not been discussed in the paper (see comments below). - 
      The MEIBP algorithm proposed in [22] is applicable only for non-negative 
      linear Gaussian models (W has to be positive, not real-valued). Therefore, 
      the artificial simulation and the block data experiments are simply 
      invalid for the MEIBP baseline. If you were to compare with MEIBP then the 
      symthetic dataset should have been generated such that the loading matrix 
      W is positive. Other Comments: - For block images 
      data, since the noise value is known (and given to the FAB algorithm), for 
      fairness the IBP Gibbs sampler should also be given the same value 
      (instead of the 0.75 std-dev heuristic). - For VB-IBP [2], the 
      infinite variational version could be used (the experiments used finite 
      variational version). The experimental settings for VB isn't described in 
      enough details (e.g., how many restarts were given?). -I am 
      surprised that the accelerated Gibbs sampling discovered about 70 latent 
      features on the 49 dimensional Sonar data!). I suspect it is because of 
      bad initialization and/or badly specified noise variance value. - 
      Line 70: The reference [21] isn't about Gibbs sampling but rather MAP 
      estimate for the IBP (just like reference [22]). Please modify the text 
      and correct the description. - There is recent work on efficient 
      inference using small-variance asymptotic in case of nonparametric LFMs 
      (see "MAD-Bayes: MAP-based Asymptotic Derivations from Bayes" from ICML 
      2013). It would be nice to discuss this work as well. Minor 
      comments: - Line 349: For real-data, the suggestion in [1,22] was 
      not to set std-dev equal to 0.75, but to set it equal to 0.25 *times* the 
      std-dev of examples across all the dimensions. 
      ********************************************* Comments after 
      the author-feedback: The feedback answered some of the questions. There 
      are a few things that need to be fixed/improved before the paper could be 
      accepted. In particular: - Experimental methodology (for the proposed 
      method and the baseline) need to be better explained, and there should be 
      justifications/explanations about why certain algorithms behaved in a 
      certain way (e.g., Gibbs sampler used a fixed alpha that might have led to 
      an overestimated K, or noise variance hyperparamters weren't properly 
      set). - Reference [21] does MAP inference, not sampling. Please fix 
      this. - Include the reference on MAD-Bayes inference for the 
      IBP.