KERNEL CHANGE-POINT DETECTION WITH AUXIL- IARY DEEP GENERATIVE MODELS
Detecting the emergence of abrupt property changes in time series is a challenging problem. Kernel two-sample test has been studied for this task which makes fewer assumptions on the distributions than traditional parametric approaches. However, selecting kernels is non-trivial in practice. Although kernel selection for two-sample test has been studied, the insufficient samples in change point detection problem hinders the success of those developed kernel selection algorithms. In this paper, we propose KL-CPD, a novel kernel learning framework for time series CPD that optimizes a lower bound of test power via an auxiliary generative model. With deep kernel parameterization, KL-CPD endows kernel twosample test with the data-driven kernel to detect different types of change-points in real-world applications. The proposed approach significantly outperformed other state-of-the-art methods in our comparative evaluation of benchmark datasets and simulation studies.
The manuscript entitled "Kernel Change-Point Detection with Auxiliary Deep Generative Models" describes a novel approach to optimising the choice of kernel towards increased testing power in this challenging machine learning problem.  The proposed method is shown to offer improvements over alternatives on a set of real data problems and the minimax objective identified is well motivated, however, I am not entirely convinced that (a) the performance improvements arise for the hypothesised reasons, and (b) that the test setting is of wide applicability.
A fundamental distinction between parametric and non-parametric tests for CPD in timeseries data is that the adoption of parametric assumptions allows for an easier introduction of strict but meaningful relationships in the temporal structure---e.g. a first order autoregressive model introduces a simple Markov structure---whereas non-parametric kernel tests typically imagine samples to be iid (before and after the change-point).  For this reason, the non-parametric tests may lack robustness to certain realistic types of temporal distributional changes: e.g. in the parameter of an autoregressive timeseries.  On the other hand, it may be prohibitively difficult to design parametric models to well characterise high dimensional data, whereas non-parametric models can typically do well in high dimension when the available data volumes are large.  In the present application it seems that the setting imagined is for low dimensional data of limited size in which there is likely to be non-iid temporal structure (i.e., outside the easy relative advantage of non-parametric methods).  For this reason it seems to me the key advantage offered by the proposed approach with its use of a distributional autoregressive process for the surrogate model may well be to introduce robustness against Type 1 errors due to otherwise unrepresented temporal structure in the base distribution (P).  In summarising the performance results by AUC it is unclear whether it is indeed the desired improvement in test power that offers the advantages or whether it is in fact a decrease in Type 1 errors.
Another side of my concern here is that I disagree with the statement: "As no prior knowledge of Q ... intuitiviely, we have to make G as close to P as possible" interpretted as a way to maximise test power; as a way to minimise Type 1 errors, yes.
Across change-point detection methods it is also important to distinguish key aspects of the problem formulation.  One particular specification here is that we have already some labelled instances of data known to come from the P distribution, and perhaps also some fewer instances of data labelled from Q.  This is distinct from fully automated change point detection methods for time series such as automatic scene selection in video data.  Another dissimilarity to that archetypal scenario is that here we suppose the P and Q distributions may have subtle differences that we're interested in; and it would also seem that we assume there is only one change-point to detect.  Or at least the algorithm does not seem to be designed to be applied in a recursive sense as it would be for scene selection.
Finally there is no discussion here of computational complexity and cost?