Bidirectional Recurrent Neural Networks as Generative Models
Bidirectional recurrent neural networks (RNN) are trained to predict both in the positive and negative time directions simultaneously. They have not been used commonly in unsupervised tasks, because a probabilistic interpretation of the model has been difficult. Recently, two different frameworks, GSN and NADE, provide a connection between reconstruction and probabilistic modeling, which makes the interpretation possible. As far as we know, neither GSN or NADE have been studied in the context of time series before. As an example of an unsupervised task, we study the problem of filling in gaps in high-dimensional time series with complex dynamics. Although unidirectional RNNs have recently been trained successfully to model such time series, inference in the negative time direction is non-trivial. We propose two probabilistic interpretations of bidirectional RNNs that can be used to reconstruct missing gaps efficiently. Our experiments on text data show that both proposed methods are much more accurate than unidirectional reconstructions, although a bit less accurate than a computationally complex bidirectional Bayesian inference on the unidirectional RNN. We also provide results on music data for which the Bayesian inference is computationally infeasible, demonstrating the scalability of the proposed methods.
Summary:The paper proposes a modification of the traditional bidirectional RNN, which predicts output at time t from forward hidden state t-1 and backward hidden state t+1, without getting information from x_t. This allows training the network to map directly from inputs X to outputs Y. The authors test the capability of this architecture to compute the likelihood of completed gaps in sequences, using GSN and NADE - the results are compared to "Bayesian MCMC" inference in standard RNNs. The paper shows that for longer gaps, GSN using the specified BRNN architecture is computationally cheaper and competitive in quality with RNNs and NADE is inferior in quality to GSN. Quality and Clarity The models and the inference, which was used to compute the NLL scores, are both clearly described. I did not see major technical or experimental setup issues.- The "one-way inference" baseline seems to outperform the BRNN for the first couple of characters (Fig 2, left side). Why is it missing in the right plot?- Given the good performance of "one-way inference" on the first couple of characters (Fig 2), wouldn't it make sense to also compare to the setup where we have 2 RNNs, one trained to predict characters going forward, one trained to predict going backward, and then we pick the first two characters from the forward RNN, last two characters from the backward RNN and perhaps the middle by voting? How would this baseline have done? This naive approach may well break down for longer sequences, but if it dominates it would suggest the experimental setup being "too easy" -- a gap of 5 characters may not be sufficient to show significant BRNN benefits.- Why is GSN better than NADE on wikipedia in Fig 1 yet appearing noticeably worse for wikipedia in Fig 2?- For Bayesian MCMC on RNNs (eq 10), when the gap is large making it hard to explore every single variable setting, is it possible to approximately estimate the posterior using direct MCMC sampling in the RNN rather than explicitly enumerating all possible missing states?Originality The modification of BRNN that connects each output y_t to forward hidden state x_{t-1} and to backward hidden state x_{t+1} appears novel to the best of my knowledge. It is a rather straightforward extension of the standard RNN. I am also not aware of existing experiments on evaluating the performance of such a model for likelihood estimation of sequence gaps task, although the integration of the BRNN with GSN and NADE is rather straightforward. Significance I am not convinced that the task of estimating the likelihood of sequence completions has much practical interest. While it is a convenient way to evaluate model quality, getting MAP estimates (e.g. using beam search) may be sufficient for most practical uses?While the BRNN is faster than the Bayesian MCMC, the quality seems a bit inferior. I am curious if there is a way to achieve getting the estimates using MCMC and RNNs without enumerating all states? That would be a reasonable alternative.