Precision and Recall for Time Series
Classical anomaly detection is principally concerned with point-based anomalies, those anomalies that occur at a single point in time. Yet, many real-world anomalies are range-based, meaning they occur over a period of time. Motivated by this observation, we present a new mathematical model to evaluate the accuracy of time series classification algorithms. Our model expands the well-known Precision and Recall metrics to measure ranges, while simultaneously enabling customization support for domain-specific preferences.
Summary: The authors present a parameterized model that generalizes precision and recall to "range-based" anomaly detection. A range-based anomaly is defined as "an anomaly that occurs over a consecutive sequence of time points, where no non-anomalous data points exist between the beginning and the end of the anomaly". The model is carefully and precisely defined. Then it is evaluated by comparing it against the metric proposed in the Numenta Anomaly Benchmark. The authors conclude that their metric is superior to the NAB metric.   Assessment: I like this paper, because it reflects clear thinking about problem formulation that will be valuable to people working in this area.   Clarity: The paper is well-written. I had a few questions. At line 269, what does "recessing" mean? In general, the paper would be improved if the authors provided better justification for biases beyond the front-end bias. I didn't see how their examples related to mid- and back-end bias.  I can't imagine a case in which one would want a back-end bias. You need to persuade the reader that a problem exists before providing a solution.  I think you have an error at line 22. In the context of anomaly detection, recall is the fraction of true anomalies that were detected and precision is the fraction of detections that were true anomalies. This is how you define things later, but the definitions here seem focused on detecting non-anomalies.   Technical Soundness:  The main claim of the paper is that their model can be used to "evaluate, rank, and compare results of AD algorithms". However, their model does not give a single measure but instead amounts to a domain-specific language for expressing various notions of precision and recall. While any specific instantiation of the model can be applied to evaluate, rank, and compare AD algorithms, it isn't clear how to do this in general. Indeed, is that a realistic goal?  Every anomaly detection application has its unique costs and interaction protocol.  For example, if we consider fraud detection, we are typically interested in the precision @ K: how many of the K top-ranked alarms are true alarms?  This is because our scarce resource is the fraud analyst, and he/she can only examine K candidates per day.  The measure described in this paper could give us a good definition of precision, but I would want to then rank by precision @ K.  I might also want to rank by the financial cost or other features of the anomaly. These are application-specific measures that I don't think can be expressed by the current model (except for the simple measure of the size (duration) of the anomaly).   If we consider equipment diagnosis, the natural measure is precision @ 100% recall (or some high percentile such as 95%). Again, the definitions in this paper would correctly define precision and recall, but I would then compare algorithms based on a measure computed from those.   It is hard to assess the comparison with the NAB scoring model. Are F1, F2, and F0.5 the right measures? They make sense for information retrieval, but in my experience, they rarely make sense for anomaly detection applications.   A stronger paper would have performed a series of case studies where, in each case, the application-specific criterion would have been expressed in the model. I would suggest replacing section 6.4 with this kind of analysis. To me, it is obvious that the metrics could be computed efficiently, but I understand that other readers might want some reassurance on this point. (Minor note: You could express the asymptotic complexity as O(max{N_r,N_p}).)  I thought the experiment reported in Table 2 was very nice.   Originality: I believe the work is original.  Significance: The paper reflects clear thinking about the problem which will be valuable to anyone who works in this area.