LEARNING PROCEDURAL ABSTRACTIONS AND EVAL- UATING DISCRETE LATENT TEMPORAL STRUCTURE
Clustering methods and latent variable models are often used as tools for pattern mining and discovery of latent structure in time-series data. In this work, we consider the problem of learning procedural abstractions from possibly highdimensional observational sequences, such as video demonstrations. Given a dataset of time-series, the goal is to identify the latent sequence of steps common to them and label each time-series with the temporal extent of these procedural steps. We introduce a hierarchical Bayesian model called PRISM that models the realization of a common procedure across multiple time-series, and can recover procedural abstractions without supervision. We also bring to light two characteristics ignored by traditional evaluation criteria when evaluating latent temporal labelings (temporal clusterings) -segment structure, and repeated structure -and develop new metrics tailored to their evaluation. We demonstrate that our metrics improve interpretability and ease of analysis for evaluation on benchmark time-series datasets. Results on benchmark and video datasets indicate that PRISM outperforms standard sequence models as well as state-of-the-art techniques in identifying procedural abstractions.
This is a hybrid paper, making contributions on two related fronts:
1. the paper proposes a performance metric for sequence labeling, capturing salient qualities missed by other metrics, and
2. the paper also proposes a new sequence labeling method based on inference in a hierarchical Bayesian model, focused on simultaneously labeling multiple sequences that have the same underlying procedure but with varying segment lengths.
This paper is not a great topic fit for ICLR: it's primarily about a hand-designed performance metric for sequence labeling and a hierarchical Bayesian model with Gaussian observations and fit with Gibbs sampling in a full-batch setting. The ICLR 2019 reviewer guidelines suggest "Ask yourself: will a substantial fraction of ICLR attendees be interested in reading this paper?" and based on my understanding of the ICLR audience I suspect not. Based on looking at past ICLR proceedings, this paper's topic and collection of techniques is not in the ICLR mainstream (though it's not totally unrelated). The authors could convince me that I'm mistaken by pointing out closely related ICLR papers (e.g. with a similar mix of techniques in their methods, or similarly proposing a hand-designed performance metric); as far as I can tell, none of the papers cited in the references are from ICLR, but rather from e.g. NIPS, AISTATS, and IEEE TPAMI, which I believe would be better fits for this kind of work.
One way to make this work more relevant to the ICLR audience would be to add feature learning (especially based on neural network architectures). That might also entail additional technical contributions, like how to fit models like these in the minibatch setting (where the current Gibbs sampling method might not apply).
On the proposed performance metric, the discussion of existing metrics as they apply to the example in Fig 3 was really helpful. (I assume, but didn't check, that the authors' characterization of the published performance metrics is accurate, e.g. "no traditional clustering criteria can distinguish C_2 from C_3".) The proposed metric seems to help.
But it's a bit complicated, with several free design decisions involved (e.g. choosing the scoring function \mathcal{H} in Sec 3.1, the choice of conditional entropy H in Sec 3.2, the choice of \beta in Sec 3.3, the choice of the specific algebraic forms of RSS, LASS, SSS, and TSS). Certainly the proposed metrics incorporate the kind of information that the authors argue can be important, but the design details of how that information is summarized into a single number aren't really explored or weighed against alternative designs choices. 
If a primary aim of this paper is to propose a new performance metric, and presumably to have it catch on with the rest of the field, then the contribution would be much greater if the design space was clearly articulated, alternatives were considered, and multiple proposals were validated. Validation could be done with human labelers ranking the intuitive 'goodness' of labeling results (and then compared to rankings derived from the proposed performance metrics), and with comparing how the metrics correlate with performance on various downstream tasks.
Another idea is to take advantage of a better segmentation performance metric and use it to automatically tune the hyperparameters of the sequence labeling methods considered in the experiments section. (IIUC hyperparameters were set by hand in the experiments.). That would make for more interesting experiments that give a more comprehensive summary of how these techniques can compare.
However, as it stands, while the performance metric itself may have merit, in this paper it is not sufficiently well validated or compared to alternatives.
On the hierarchical Bayesian model, the current model design andinference algorithm are okay but don't constitute major technical contributions. I was surprised by some model details: for example, in "Modeling the procedure" of Sec 4.1, it would be much more satisfying to generate the (p_1, ..., p_s) sequence from an HMM instead of sampling the elements of the sequence independently, dropping any chance to learn transition structure as part of the Bayesian inference procedure. More importantly, it wasn't made clear if 'self-transitions' where p_s = p_{s+1} were ruled out, though such transitions might confuse the model's semantics. As another example, in "Modeling the realizations in each time-series" of Sec 4.1, the procedure based on iid sampling and sorting seems unnatural, and might make inference more complex. Why not just sample the durations directly (rather than indirectly defining them via sorting independently-generated indices)? If there's a good reason, it should probably be discussed (e.g. maybe parameterizing the durations directly would make it easier to express prior distributions over *absolute* segment lengths, but harder to express distributions over *relative* segment lengths?). Finally, the restriction to conditionally iid Gaussian observations was disappointing.
The experimental results were solid on the task for which the model's extra assumptions paid off, but that's a niche comparison.
One suggestion on the baseline front: you can tie multiple HMMs to have the same procedure (i.e. the same state sequences not counting repeats) by fixing the number of states to be s (the length of the procedure sequence) and fixing the transition matrices to have an upper-bidiagonal support structure. A similar construction can be used for HSMMs. I think a natural Gibbs sampling procedure would emerge. This approach is probably written down in the HMM literature (it seems every conceivable HMM variant has been studied!) but I don't have a reference for it.
Overall, this paper needs more work.
Minor suggestions:
- maybe refer to "segment structure" (e.g. in Sec 3), as "changepoint structure" (and consider looking into changepoint performance metrics if you haven't already)
- if you used code from other authors in your baselines, it would be good to cite that code (e.g. GitHub links)