Scaling Factorial Hidden Markov Models: Stochastic Variational Inference without Messages
Factorial Hidden Markov Models (FHMMs) are powerful models for sequential data but they do not scale well with long sequences. We propose a scalable inference and learning algorithm for FHMMs that draws on ideas from the stochastic variational inference, neural network and copula literatures. Unlike existing approaches, the proposed algorithm requires no message passing procedure among latent variables and can be distributed to a network of computers to speed up learning. Our experiments corroborate that the proposed algorithm does not introduce further approximation bias compared to the proven structured mean-field algorithm, and achieves better performance with long sequences and large FHMMs.
This paper develops more scalable variational methods for factorial hidden Markov models by combining ideas from structured mean-field optimization, recognition networks and stochastic gradient descent.   In the proposed approximating posterior based on a Gaussian bivariate copula, each timestep t within chain m has two variational parameters: marginal probability \theta_tm and correlation \rho_tm. These two parameters, together with parameters of neighboring timesteps, can represent any possible pairwise distribution for neighboring binary states. For a single binary chain of length T, there are in total 2T required parameters. The choice of the marginal and correlation values, as opposed to other representations, is done to make each value predictable from a separate recognition network. The proposed recognition network approach reduces the total number of parameters per chain from 2T to 2H, where H is the total number of parameters in a feed-forward network that predicts the scalar correlation or marginal probabilty value for timestep t given a fixed-width window of data around timestep t. The length of the fixed-width window and the internal structure of the recognition network are in general application dependent and likely must be chosen carefully.  The whole approach results in a variational objective function parameterized by global parameters like emission and transition probabilities, and the recognition network weight vectors which control the local state sequence posterior. The number of parameters per sequence is much less than the structured mean field approach, leading to gains in training and evaluation speed. This function can be optimized via stochastic gradient descent, where we repeatedly sample small windows of small, fixed length from the long sequence and take gradient steps. Automatic differentiation packages are used here.  Experiments compare the proposed approach against a full-batch SMF baseline on several datasets including Bach chorales and household power consumption.
Overall I think this work is interesting and the proposed union of recognition networks and variational inference is of growing interest in the community. This could make a good poster presentation at NIPS, but I could also understand arguments for borderline rejection. My primary concerns are the method requires lots of tuning to find the right recognition network structure with little guidance on this front, and that the experiments are a bit weak in that they only compare the proposed method to a very basic method (full-batch structured mean field) without evaluating recent alternatives like Foti et al [4] which might be more scalable by using small windows of the long sequence without the added complexity of recognition networks.  Technical feedback ================== Overall the proposed method requires several important decisions on the part of the end user: recognition network structure, fixed-window length, how to handle the edge case timesteps at the beginning and end of the sequence where the network's window does not fit, etc. These are casually mentioned, but no experiment really provides guidance about how to deal with these in a principled way, or even compares tradeoffs in runtime vs accuracy under different possibilities. I think this lack of careful consideration will make this method difficult to apply in practice, and I'd like to see more discussion in a revision.   The question of local optima is definitely relevant here, and little guidance is provided on this front. Without showing results from many independent initializations, it is difficult to know whether the proposed approach might be more or less vulnerable to local optima than the standard SMF approach.  Is code available? Perhaps sharing code would mitigate some concerns, especially when for dealing with the edge cases.  I'd like to see some scalability experiments that directly compare two methods for a sequence of length T and a trained model with a single binary chain: (1) using a recognition network of width W to predict the entire state sequence, including edge cases, and (2) using the standard forward/backward algorithm to estimate the state sequence. I think it would be useful to directly quantify how much easier this kind of test-time inference gets under the proposed approach... is it 2x? 10x better? Hard to say right now.  Experiment feedback =================== Looking at Table 1 for the Bach dataset, the baseline structured mean field (SMF) approach produces log likelihood on training (LLtrain) values that continue to improve with more hidden chains all the way through 8 chains, as we would expect because the model capacity is improving and can fit the data better. However, for the proposed algorithm, the LLtrain with 7 chains (-5.75) is noticeably better than for 8 chains (-5.836), which I would not expect. I would guess this is a local optima issue. I'd like to hear more about this in rebuttal. Should we be concerned that for large model capacity the performance of the proposed approach might suffer? This doesn't really reassure me that the performance is always indistinguishable from SMF.  Overall, I'd much rather see Table 1 turned into a line plot of performance vs # chains. Tables are not ideal for helping your audience understand relative performance trends.  When evaluating scalability in Sec. 5, the comparison between the proposed approach (which can stochastically use subsequences *and* take advantage of the cheap recognition network) and the SMF-EM algorithm, which must process a whole sequence before each global parameter update, might not be the fairest possible comparison. I'd be curious how the Foti et al method [4], which can process small subsequences from the long sequence but does *not* use the recognition network, stacks up in this comparison. This may be too much to ask, but otherwise this experiment isn't especially convincing that each piece in the proposed pipeline lead to scalability wins. It's hard to know whether the effort for the recognition network is really "worth it", or whether just doing stochastic updates after a few short windows would give similar performance.  Household power dataset questions: how long (wallclock time) did training take here? How many SGD iterations were done? How much data does one iteration use?  Instead of just showing the predictions of the method on the power dataset, I'd suggest a within-method comparison, where you choose some useful tweak to your model (5 vs 10 chains, or rolling window size of 5 vs 10, etc) and report how the resulting performance vs speed tradeoffs.  Typos, etc ========== line 160 typo: "ramdom variables in different chains"  Sec 5.2 suggestion: Say "this experiment consists of two parts" instead of "two components".