text:
\section{Introduction}

\bibliographystyle{IEEEtran}

\section{Results}\label{sec:expres}

\subsection{Datasets, metrics, and experimentation setup}\label{sec:dataset}

\FG{shall we directly start with: "We compare HypAD against the current best univariate anomaly detector TadGAN~[ref] on a benchmark of 11 time series~[ref], and extend the comparison to 2 multivariate sensor datasets: one comprising a water treatment plant and one comprising daily activities in elderly residences." In this way, it's clear upfront what is comparable to what, fairly, and we do not need a a-posteriori justification.}
To analyse the performances of HyAD w.r.t.\ the state-of-the-art we employ several anomaly benchmarks on 11 univariate time series,\BM{1 cyber-physical system dataset}\FG{these terms are not telling me much. I'd prefer a more description "sensor data from a water treatment plant". Is the terminology "cyber-physical system" established for SWaT? If so, I won't complain :))}, and 1 medical dataset containing multivariate sensoristic data on elderly home residences \cite{cook2012casas}.
Table \ref{tab:datasets} illustrates a summarised analysis of the main characteristics of the datasets. For each dataset, we illustrate the number of signals which represent the number of different inputs within the same category. In other words, the number of signals is the number of different time series coming from several files. Notice that we only have one signal per medical condition (e.g. Nocturia, MoreTimeInChair) in the multivariate scenario\footnote{We abbreviate the signals in CASAS: Fall (F), MoreTimeInChair (MTC), Weakness (W), SlowerWalking (SW), and Nocturia (N).}. The number of anomalies depicts the sum of the number of instances with a corresponding anomalous label. As in \cite{geiger2020tadgan}, we also observe whether the anomalies in the dataset are point or collective anomalies\footnote{A point anomaly is a single instance isolated in time that is anomalous. Whereas, a collective anomaly is constituted by multiple subsequent anomalies in time.}. \FG{any comment after the observation? may one say that point anomalies are more difficult? They occur in Yahoo, where we are outperformed. I just raise this comment here because an observation without a conclusion may be used against us.} Finally, to elucidate the hardness of the anomaly detection in each dataset, we label each anomaly point as out-of-distribution if it falls $4\sigma$ away from the mean of a particular signal \FG{is there a reference for this type of observation? It feels like this would only be valid under a Gaussian assumption?}. \FG{Among the univariate time series, -- maybe better to add it upfront, instead of commenting that this only applies to univariate, else one may straightaway complain about this statement of what is more complex} According to this labelling strategy, NASA (SMAP), and Yahoo (A1, A2) represent the hardest scenarios since the percentage of out-of-distribution instances is elevated, in some cases surpassing that of the non anomalous instances (i.e. A1). Note that this approach is meaningful for the univariate datasets, but cannot be applied in the multivariate scenario.

\paragraph{Univariate datasets} \textit{NASA} includes two spacecraft telemetry datasets\footnote{MSL and SMAP can be downloaded from \url{https://s3-us-west-2.amazonaws.com/telemanom/data.zip}.} based on the Mars Science Laboratory (MSL) and the Soil Moisture Active Passive (SMAP) signals. The former consists of scientific and housekeeping engineering data taken from the Rover Environmental Monitoring Station aboard the Mars Science Laboratory. The latter includes measurements of soil moisture and freeze/thaw state from space for all non-liquid water surfaces globally within the top layer of the Earth.

We analyse \textit{Yahoo} datasets based on real production traffic to Yahoo computing systems. Additionally, we consider three synthetic datasets coming from the same source\footnote{Upon request, the dataset can be downloaded from \url{https://webscope.sandbox.yahoo.com/catalog.php?datatype=s&did=70}}. The dataset tests the detection accuracy of various anomaly-types including outliers and change-points. The synthetic dataset consists of time-series with varying trend, noise and seasonality. The real dataset consists of time-series representing the metrics of various Yahoo services.

\textit{Numenta Anomaly Benchmark (NAB)} is a well-established collection of univariate time-series from real-world application domains\footnote{We invite the reader to consult \url{https://github.com/numenta/NAB/tree/master/data} for the entire directory of datasets.}. To be consistent with \cite{geiger2020tadgan}, we analyse Art, AdEx, AWS, Traf, and Tweets from the original collection.

\paragraph{Multivariate dataset} We consider \BM{for analysis the SWaT~\cite{10.1007/978-3-319-71368-7_8,Mathur2016SWaTAW} and CASAS~\cite{cook2012casas,dahmen2021indirectly} datasets.

\noindent \textit{SWaT} is collected from a cyber-physical system testbed that is a scaled down replica of an industrial water treatment plant. The data was collected every second for a total of 11 days, where for the first few days the system was operated normally, while for the remaining days certain cyber-physical attacks were launched. Following~\cite{10.1145/3447548.3467137}, we sampled sensor data every every five seconds.}

\noindent \textit{CASAS} is a collection of two weeks of sensor data from retirement homes. Each sensor reading has a label attached according to the activity recognised by human annotators. In this way, we have a collection of time series of activities performed by elderly people. Furthermore, the anomalies are labelled according to several medical conditions\footnote{Because most of the people involved in the test suffer from chronic diseases such as severe depression and Alzheimer's, criteria like falling, night toilet visits, weakness in performing activities are used to label the time series.} established prior to the experimentation. Although sensor readings give fine-grained information, we are interested in creating daily patient profiles. Hence, we collapse them into a single engendered activity with a start and end time\footnote{The start time corresponds to the first sensor reading for that activity, whereas the end time is the last reading.} for each consecutive sensor signal. We then create a time matrix structure $\mathcal{M}^{|d|,1440}$ where $|d|$ is the number of days the patient is monitored and 1440 represent the total number of minutes in a day. $\mathcal{M}[i,j]$ represents the label performed in minute $j$ of day $i$. Lastly, we densify each value $\mathcal{M}[i,j]$ with contextual and duration information corresponding to the label therein.

Finally, we create train and test splits of this data, caring to safeguard sequentiality of observations, and gathering the few anomalies into the test set for evaluation only. For this reason we name this Unsupervised-CASAS, dubbed U-CASAS, differing from the original CASAS \cite{cook2012casas,dahmen2021indirectly} since the latter interleaves anomalous sensor readings with normal instances, thus breaking the sequentiality of anomalies.
For each anomalous day encountered in the test set, we pad two days prior to it and two after. If the padding overlaps with another anomalous day, then we concatenate\footnote{The concatenation procedure merges common sequences. If the sequence contains more than one anomalous day within the two-day padding window, the padded sequence gets collapsed into a single one.} them and perform the padding procedure again. We delete the days assigned to the test set from the overall dataset and assign the rest as training. Moreover, because we employ a time-related strategy, we create time windows of 30 actions to detect anomalies.

\paragraph{Metrics}
Being all the enlisted datasets highly unbalanced, accuracy is misleading. Therefore, as done in \cite{suris2021hyperfuture},we use the F1 score to account for this challenge. Based on \cite{hundman2018detecting}, we penalise high false positive rates and encourage the detection of true positive in a timely fashion. Since anomalies are rare events and come in collective sequences in real-world applications, we proceed as follows:
\begin{enumerate}
\item We record a true positive (TP) if any predicted window overlaps a true anomalous window.
\item We record a false negative (FN) if a true anomalous window does not overlap any predicted window.
\item We record a false positive (FP) if a predicted window does not overlap any true anomalous region.
\end{enumerate}
For U-CASAS, we also measure the g-measure which is the geometric mean of the product of recall (R) and precision (P), being a robust metric when classes are imbalanced \cite{dahmen2021indirectly}.

\input{tab/performances_univariate}
\input{tab/performances_casas}

\paragraph{Baselines}
 We include the following strategies as our baselines in this paper:
\begin{itemize}
    \item \textit{AE} \cite{baldi2012autoencoders} - We use a six-layer fully-connected autoencoder.
    \item \textit{ConvAE} \cite{maggipinto2018convolutional} - We have three layers of convolutional encoding interleaved with max pooling. The decoder has a specular composition as the encoder where the de-convolution is aided by two-dimensional up-sampling layers.
    \item \textit{LstmAE} \cite{sagheer2019unsupervised} - We use a deep stacked LSTM autoencoder with four layers. The first LSTM hidden and output vectors get passed to the second LSTM layer. The latent representation of the encoder gets then reconstructed in reverse order from the decoder.
    \item \textit{TadGAN*} \cite{geiger2020tadgan} - We use a one-layer bidirectional LSTM for the generator $E$, and a two-layer bidirectional LSTM for $G$. For the critic $D_z$ we use a fully connected layer, and two dense layers for $D_x$.
\end{itemize}

\noindent\textit{Implementation details.} For the first three baselines, we set the number of epochs to 30, the batch size to 32, and the learning rate to $10^{-3}$. For TadGAN*, we set the epochs to 30, the batch size to 64, the learning rate to $5\times10^{-4}$, and the iteration for the critic to 5. We use Adam as the optimisation function to train all the baselines.

For our proposed method HypAD, we took inspiration from an online available PyTorch implementation\footnote{\url{https://github.com/arunppsg/TadGAN}}. We leave the architecture of TadGAN unvaried, but we incorporate the hyperbolic transformation\footnote{\url{https://github.com/cvlab-columbia/hyperfuture}} as in \cite{suris2021hyperfuture}. The hyperparameters are the same in the original paper, but we use the Riemannian Adam\footnote{\url{https://github.com/geoopt/geoopt}} as optimisation function.

\subsection{Comparison to the state of the art}
\textbf{HypAD outperforms all the baseline methods by
having the highest average F1-scores of 0.631 and 0.428 across the univariate and multivariate datasets, respectively.} Both Tables \ref{tab:performances_univariate} and \ref{tab:performances_casas} rank all the methods based on their averaged F1 scores (the last column) across all datasets. HypAD outperforms the second (TadGAN*, 0.600) and the third (ConvAE, 0.212) by 5.17\% and 197.64\%, respectively in the univariate scenario. Contrarily, HypAD surpasses TadGAN* by 32.51\% and ConvAE by 486.3\% according to the averaged F1-score, in the multivariate scenario.

Table \ref{tab:performances_univariate} depicts the comparison of our proposed method, HypAD, with the baselines strategies described above. For reference purposes, we enlist also the performances of TadGAN as reported\footnote{We implemented TadGAN relying on PyTorch which does not match the reported performances in the original paper.} in \cite{geiger2020tadgan}. Here, we report the F1-score, and the mean and standard deviation over all datasets. Notice that only TadGAN competes with HypAD. In this scenario, we have time windows of 100 values. Because we apply the poincaré distance directly on the windows and not on the reconstructed signal as with TadGAN, HypAD experiences a slight drop in performances. Nevertheless, with an average F1-score of $0.631$, HypAD is the best-performing method in the univariate case.

Table \ref{tab:performances_casas} illustrates the performances in the multivariate scenario \BM{on the U-CASAS dataset}. We cannot  include Isudra\cite{dahmen2021indirectly} because the underlying architecture uses a small amount of labels to conduct the selection of parameters for the execution. Additionally, Isudra is trained in a supervised fashion differing from all the other methods reported here. However, for completeness purposes, we also report the g-measure and its average across all datasets. Here, the dominance of HypAD is more evident since the transformation of multivariate time window in hyperspace has the benefit of learning latent relationships between the input features. Notice that HypAD is the best-performing method according to both g-measure and F1-score.

\BM{Table \ref{tab:performances_swat_fg} compares the proposed HypAD to other methods on the SWaT dataset. Following previous works~\cite{10.1145/3447548.3467137}, we also report the precision and recall in addition to the F1- score. As shown in the table, HypAD gives the highest precision (0.996) among all other methods. This means thate our model detects an anomaly only when it is certain of its understanding of the input signal and therefore refrains from giving wrong anomalies. However, HypAD still lags behind the other recent approaches in terms of recall and F1-score. We argue that..  }

\input{tab/swat_permormances}

\input{tab/ablation}
\subsection{Ablation Studies}
\BM{In Table~\ref{tab:ablation}, we demonstrate the effectiveness of HypAD on three different dataset types: Univariate, U-CASES and SWaT. The first row shows the performance of the model in euclidean space. This corresponds to the model TadGAN* in Table~\ref{tab:performances_univariate},~\ref{tab:performances_casas} and~\ref{tab:performances_swat_fg}. Next we move the model to the hyperbolic space as discussed in Section~\ref{sec:hypRe}. This brings an improvement of 0.7\% and 18.6\% in the F1 score for the univariate and U-CASES datasets, respectively. However for the SWaT  dataset the performance drops, which we discuss qualitatively in the following section. Finally, we integrate the hyperbolic uncertainty to the anomaly detection. As shown in the last two rows of the table, this brings an improvement of 4.5\% (0.604 vs 0.631) for the univariate cases, 7.8\% (0.397 vs 0.428) for the U-CASES and 33\% for the SWaT. Clearly, \textbf{Uncertainty is fundamental in ameliorating anomaly detection.} For all the datasets, the best performance is achieved using the proposed hyperbolic uncertainty.}

\noindent \textbf{Qualitative Ablation on SWaT:} \BM{In Figure \ref{fig:ablation1}, we demonstrate qualitative ablation on SWaT dataset that consists of only one long-term signal. The three plots correspond to the three rows in the last column of Table~\ref{tab:ablation}. In all plots, the blue and green points represent the predicted anomaly scores and ground-truth anomalies, respectively. The red line denotes the anomaly detection threshold i.e. the blue points that lie above the red line are predicted anomalies. If a blue point lies above the red line but does not coincide with any green point (ground-truth), it is a false positive.

As visible in the Figure, the model in euclidean space (first plot) gives a lot of false positives but when moving to hyperbolic space (second plot) the model becomes more precise reducing false positives substantially. However note that this hyperbolic model also brings a part of positive anomalies (in the middle of the signal) below the threshold, reducing true positives. This explains the drop in the F-1 score for the SWaT dataset (cf. Table \ref{tab:ablation}). Interestingly, the integration of uncertainty recovers these points back in addition to others, increasing the true positives but not false positives, giving the best F-1 score of 0.753.}

\input{fig/ablation1.tex}

\section{Results}\label{sec:expres}

We compare HypAD against the current best univariate anomaly detector TadGAN~\cite{geiger2020tadgan} on a benchmark of 11 time series, and extend the comparison to 2 multivariate sensor datasets: one comprising a water treatment plant and one comprising daily activities in elderly residences.
First, we introduce the benchmarks (Sec.~\ref{sec:dataset}), then we compare against baselines and the state-of-the-art (Sec.~\ref{sec:sota}), finally we conduct ablative studies on the importance of uncertainty for performance and the reduction of false alarms (Sec.~\ref{sec:abl}).

\subsection{Datasets, metrics, and experimentation setup}\label{sec:dataset}

Table \ref{tab:datasets} illustrates a summarised analysis of the main characteristics of the datasets. For each dataset, we illustrate the number of signals which represent the number of different inputs within the same category. In other words, the number of signals is the number of different time series coming from several files. Notice that we only have one signal per medical condition (e.g. Nocturia, MoreTimeInChair) in the multivariate scenario\footnote{We abbreviate the signals in CASAS: Fall (F), MoreTimeInChair (MTC), Weakness (W), SlowerWalking (SW), and Nocturia (N).}. The number of anomalies depicts the sum of the number of instances with a corresponding anomalous label. As in \cite{geiger2020tadgan}, we also observe whether the anomalies in the dataset are point or collective anomalies\footnote{A point anomaly is a single instance isolated in time that is anomalous. Whereas, a collective anomaly is constituted by multiple subsequent anomalies in time.}. \FG{any comment after the observation? may one say that point anomalies are more difficult? They occur in Yahoo, where we are outperformed. I just raise this comment here because an observation without a conclusion may be used against us.} Finally, to elucidate the hardness of the anomaly detection in each dataset, we label each anomaly point as out-of-distribution if it falls $4\sigma$ away from the mean of a particular signal. \FG{is there a reference for this type of observation? It feels like this would only be valid under a Gaussian assumption?}. \FG{Among the univariate time series, -- maybe better to add it upfront, instead of commenting that this only applies to univariate, else one may straightaway complain about this statement of what is more complex} According to this labelling strategy, NASA (SMAP), and Yahoo (A1, A2) represent the hardest scenarios since the percentage of out-of-distribution instances is elevated, in some cases surpassing that of the non anomalous instances (i.e. A1). Note that this approach is meaningful for the univariate datasets, but cannot be applied in the multivariate scenario.

\paragraph{Univariate datasets} \textit{NASA} includes two spacecraft telemetry datasets\footnote{MSL and SMAP can be downloaded from \url{https://s3-us-west-2.amazonaws.com/telemanom/data.zip}.} based on the Mars Science Laboratory (MSL) and the Soil Moisture Active Passive (SMAP) signals. The former consists of scientific and housekeeping engineering data taken from the Rover Environmental Monitoring Station aboard the Mars Science Laboratory. The latter includes measurements of soil moisture and freeze/thaw state from space for all non-liquid water surfaces globally within the top layer of the Earth.

We analyse \textit{Yahoo} datasets based on real production traffic to Yahoo computing systems. Additionally, we consider three synthetic datasets coming from the same source\footnote{Upon request, the dataset can be downloaded from \url{https://webscope.sandbox.yahoo.com/catalog.php?datatype=s&did=70}}. The dataset tests the detection accuracy of various anomaly-types including outliers and change-points. The synthetic dataset consists of time-series with varying trend, noise and seasonality. The real dataset consists of time-series representing the metrics of various Yahoo services.

\textit{Numenta Anomaly Benchmark (NAB)} is a well-established collection of univariate time-series from real-world application domains\footnote{We invite the reader to consult \url{https://github.com/numenta/NAB/tree/master/data} for the entire directory of datasets.}. To be consistent with \cite{geiger2020tadgan}, we analyse Art, AdEx, AWS, Traf, and Tweets from the original collection.

\paragraph{Multivariate dataset} We consider for analysis the SWaT~\cite{10.1007/978-3-319-71368-7_8,Mathur2016SWaTAW} and CASAS~\cite{cook2012casas,dahmen2021indirectly} datasets.

\noindent \textit{SWaT} is collected from a cyber-physical system testbed that is a scaled down replica of an industrial water treatment plant. The data was collected every second for a total of 11 days, where for the first few days the system was operated normally, while for the remaining days certain cyber-physical attacks were launched. Following~\cite{10.1145/3447548.3467137}, we sampled sensor data every every 5 seconds.

\noindent \textit{CASAS} is a collection of two weeks of sensor data from retirement homes. Each sensor reading has a label attached according to the activity recognised by human annotators. In this way, we have a collection of time series of activities performed by elderly people. Furthermore, the anomalies are labelled according to several medical conditions\footnote{Because most of the people involved in the test suffer from chronic diseases such as severe depression and Alzheimer's, criteria like falling, night toilet visits, weakness in performing activities are used to label the time series.} established prior to the experimentation. Although sensor readings give fine-grained information, we are interested in creating daily patient profiles. Hence, we collapse them into a single engendered activity with a start and end time\footnote{The start time corresponds to the first sensor reading for that activity, whereas the end time is the last reading.} for each consecutive sensor signal. We then create a time matrix structure $\mathcal{M}^{|d|,1440}$ where $|d|$ is the number of days the patient is monitored and 1440 represent the total number of minutes in a day. $\mathcal{M}[i,j]$ represents the label performed in minute $j$ of day $i$. Lastly, we densify each value $\mathcal{M}[i,j]$ with contextual and duration information corresponding to the label therein.

Finally, we create train and test splits of this data, caring to safeguard sequentiality of observations, and gathering the few anomalies into the test set for evaluation only. For this reason we name this Unsupervised-CASAS, dubbed U-CASAS, differing from the original CASAS \cite{cook2012casas,dahmen2021indirectly} since the latter interleaves anomalous sensor readings with normal instances, thus breaking the sequentiality of anomalies.
For each anomalous day encountered in the test set, we pad two days prior to it and two after. If the padding overlaps with another anomalous day, then we concatenate\footnote{The concatenation procedure merges common sequences. If the sequence contains more than one anomalous day within the two-day padding window, the padded sequence gets collapsed into a single one.} them and perform the padding procedure again. We delete the days assigned to the test set from the overall dataset and assign the rest as training. Moreover, because we employ a time-related strategy, we create time windows of 30 actions to detect anomalies.

\paragraph{Metrics}
Being all the enlisted datasets highly unbalanced, accuracy is misleading. Therefore, as done in \cite{suris2021hyperfuture},we use the F1 score to account for this challenge. Based on \cite{hundman2018detecting}, for the univariate and the CASAS datasets, we penalize high false positive rates and encourage the detection of true positive in a timely fashion. Since anomalies are rare events and come in collective sequences in real-world applications, we proceed as follows:
\begin{enumerate}
\item We record a true positive (TP) if any predicted window overlaps a true anomalous window.
\item We record a false negative (FN) if a true anomalous window does not overlap any predicted window.
\item We record a false positive (FP) if a predicted window does not overlap any true anomalous region.
\end{enumerate}
For U-CASAS, we also measure the g-measure which is the geometric mean of the product of recall (R) and precision (P), being a robust metric when classes are imbalanced \cite{dahmen2021indirectly}.

\input{tab/performances_univariate}
\input{tab/performances_casas}

\paragraph{Baselines}
 We include the following strategies as our baselines in this paper:
\begin{itemize}
    \item \textit{AE} \cite{baldi2012autoencoders} - We use a six-layer fully-connected autoencoder.
    \item \textit{ConvAE} \cite{maggipinto2018convolutional} - We have three layers of convolutional encoding interleaved with max pooling. The decoder has a specular composition as the encoder where the de-convolution is aided by two-dimensional up-sampling layers.
    \item \textit{LstmAE} \cite{sagheer2019unsupervised} - We use a deep stacked LSTM autoencoder with four layers. The first LSTM hidden and output vectors get passed to the second LSTM layer. The latent representation of the encoder gets then reconstructed in reverse order from the decoder.
    \item \textit{TadGAN*} \cite{geiger2020tadgan} - We use a one-layer bidirectional LSTM for the generator $E$, and a two-layer bidirectional LSTM for $G$. For the critic $D_z$ we use a fully connected layer, and two dense layers for $D_x$.
\end{itemize}

\noindent\textit{Implementation details.} For the first three baselines, we set the number of epochs to 30, the batch size to 32, and the learning rate to $10^{-3}$. For TadGAN*, we set the epochs to 30, the batch size to 64, the learning rate to $5\times10^{-4}$, and the iteration for the critic to 5. We use Adam as the optimisation function to train all the baselines.

For our proposed method HypAD, we took inspiration from an online available PyTorch implementation\footnote{\url{https://github.com/arunppsg/TadGAN}}. We leave the architecture of TadGAN unvaried, but we incorporate the hyperbolic transformation\footnote{\url{https://github.com/cvlab-columbia/hyperfuture}} as in \cite{suris2021hyperfuture}. The hyperparameters are the same in the original paper, but we use the Riemannian Adam\footnote{\url{https://github.com/geoopt/geoopt}} as optimisation function.

\subsection{Comparison to the state of the art} \label{sec:sota}

\textbf{HypAD defines a new state-of-the-art performance for univariate anomaly detection by having the highest average F1-scores of 0.631.}
\FG{Please split this statement. Leave here only the univariate, else our earlier note is spoiled. I mean: it's best to discuss univariate, and then say that we extend this test to multivariate.}
Both Tables \ref{tab:performances_univariate} and \ref{tab:performances_casas} rank all the methods based on their averaged F1 scores (the last column) across all datasets. HypAD outperforms the second (TadGAN*, 0.600) and the third (ConvAE, 0.212) by 5.17\% and 197.64\%, respectively in the univariate scenario. Contrarily, HypAD surpasses TadGAN* by 32.51\% and ConvAE by 486.3\% according to the averaged F1-score, in the multivariate scenario.
\FG{in the spirit of my note, please check that all earlier paragraphs regard univariate just.}

Table \ref{tab:performances_univariate} depicts the comparison of our proposed method, HypAD, with the baselines strategies described above. For reference purposes, we enlist also the performances of TadGAN as reported\footnote{We implemented TadGAN relying on PyTorch which does not match the reported performances in the original paper.} in \cite{geiger2020tadgan}. Here, we report the F1-score, and the mean and standard deviation over all datasets. Notice that only TadGAN competes with HypAD. In this scenario, we have time windows of 100 values. Because we apply the poincaré distance directly on the windows and not on the reconstructed signal as with TadGAN, HypAD experiences a slight drop in performances. Nevertheless, with an average F1-score of $0.631$, HypAD is the best-performing method in the univariate case.

In Table~\ref{tab:performances_casas} we extend the evaluation of HypAD to the multivariate scenario of the U-CASAS dataset. We cannot  include Isudra\cite{dahmen2021indirectly} because the underlying architecture uses a small amount of labels to conduct the selection of parameters for the execution. Additionally, Isudra is trained in a supervised fashion differing from all the other methods reported here. However, for completeness purposes, we also report the g-measure and its average across all datasets. Here, the dominance of HypAD is more evident since the transformation of multivariate time window in hyperspace has the benefit of learning latent relationships between the input features. Notice that HypAD is the best-performing method according to both g-measure and F1-score.
\FG{I'd suggest smoothing this message, else it seems one should expect higher performance from multivariate, which is not the case.}

Finally, in Table~\ref{tab:performances_swat_fg}, we also extend comparison to the multivariate SWaT dataset. Here, following previous work~\cite{10.1145/3447548.3467137}, we also report the precision and recall in addition to the F1-score. HypAD outperforms the baseline TadGAN (0.753 Vs.\ 0.722) but both techniques are behind the current state-of-the-art on multivariate anomaly detection NSIBF~\cite{10.1145/3447548.3467137}.
Observe however that HypAD achieves its highest F1-performance at the highest precision (0.996) among all other methods. This confirms that HypAD really detects anomalies which it is certain about, i.e.\ when it \emph{understands} the time series, it is certain about what to expect, but cannot reconstruct the input signal due to an anomaly.

\input{tab/swat_permormances}

\input{tab/ablation}

\subsection{Ablation Studies}\label{sec:abl}

In Table~\ref{tab:ablation}, we demonstrate the effectiveness of HypAD on three different dataset types: Univariate, U-CASAS and SWaT. The first row shows the performance of the model in euclidean space. This corresponds to the model TadGAN* in Table~\ref{tab:performances_univariate},~\ref{tab:performances_casas} and~\ref{tab:performances_swat_fg}. Next we move the model to the hyperbolic space as discussed in Section~\ref{sec:hypRe}. This brings an improvement of 0.7\% and 18.6\% in the F1 score for the univariate and U-CASAS datasets, respectively. However for the SWaT  dataset the performance drops, which we discuss qualitatively in the following section. Finally, we integrate the hyperbolic uncertainty to the anomaly detection. As shown in the last two rows of the table, this brings an improvement of 4.5\% (0.604 vs 0.631) for the univariate cases, 7.8\% (0.397 vs 0.428) for the U-CASAS and 33\% for the SWaT. Clearly, \textbf{Uncertainty is fundamental in ameliorating anomaly detection.} For all the datasets, the best performance is achieved using the proposed hyperbolic uncertainty.

\noindent \textbf{Qualitative Ablation on SWaT:} In Figure \ref{fig:ablation1}, we demonstrate qualitative ablation on the SWaT dataset that consists of only one long-term signal. The three plots correspond to the three rows in the last column of Table~\ref{tab:ablation}. In all plots, the blue and green points represent the predicted anomaly scores and ground-truth anomalies, respectively. The red line denotes the anomaly detection threshold i.e. the blue points that lie above the red line are predicted anomalies. If a blue point lies above the red line but does not coincide with any green point (ground-truth), it is a false positive.

As visible in the Figure, the model in euclidean space (first plot) gives a lot of false positives but when moving to hyperbolic space (second plot) the model becomes more precise reducing false positives substantially. However note that this hyperbolic model also brings a part of positive anomalies (in the middle of the signal) below the threshold, reducing true positives. This explains the drop in the F-1 score for the SWaT dataset (cf. Table \ref{tab:ablation}). Interestingly, the integration of uncertainty recovers these points back in addition to others, increasing the true positives but not false positives, giving the best F-1 score of 0.753.

\input{fig/ablation1.tex}

\section{Results}\label{sec:expres}

We compare HypAD against the current best univariate anomaly detector TadGAN~\cite{geiger2020tadgan} on a benchmark of 11 time series, and extend the comparison to 2 multivariate sensor datasets: one comprising a water treatment plant and one comprising daily activities in elderly residences.
First, we introduce the benchmarks (Sec.~\ref{sec:dataset}), then we compare against baselines and the state-of-the-art (Sec.~\ref{sec:sota}), finally we conduct ablative studies on the importance of uncertainty for performance and the reduction of false alarms (Sec.~\ref{sec:abl}).

\subsection{Datasets, metrics and experimental setup}\label{sec:dataset}

Table~\ref{tab:datasets} summarizes the main characteristics of the datasets, which we coarsely divide into univariate, main test bed of our baseline TadGAN~\cite{geiger2020tadgan}, and multivariate, to which we extend comparison. In the table, we report the sources of signals (NASA, Yahoo, NAB, CASAS and SWaT) and the datasets within each source (SMAP, MSL, A1 etc.), cf.\ detailed description later in the section.

In the table, the \textit{number of signals} is the number of time series within the datasets. Note that univariate datasets are composed of multiple signals, while the multivariate only comprise single large time series.
The \textit{number of anomalies} counts the instances, within the time series, labelled as anomalous. These are further detailed as \textit{point anomalies}, single anomalous values at a specific point in time, or \textit{collective anomalies}, sets of contiguous times which are altogether anomalous. Yahoo is the sole dataset with point anomalies, and the only with \textit{synthetic} sequences A2, A3, A4.
We also report the \textit{percentage of total} anomalous points and, following \cite{geiger2020tadgan}, for the univariate datasets, the \textit{number of out-of-distribution} points, exceeding the means by more than $4\sigma$.

\paragraph{Univariate datasets} \textit{NASA} includes two spacecraft telemetry datasets\footnote{\url{https://s3-us-west-2.amazonaws.com/telemanom/data.zip}.} based on the Mars Science Laboratory (MSL) and the Soil Moisture Active Passive (SMAP) signals. The former consists of scientific and housekeeping engineering data taken from the Rover Environmental Monitoring Station aboard the Mars Science Laboratory. The latter includes measurements of soil moisture and freeze/thaw state from space for all non-liquid water surfaces globally within the top layer of the Earth.

We analyse \textit{Yahoo} datasets based on real production traffic to Yahoo computing systems. Additionally, we consider three synthetic datasets coming from the same source\footnote{\url{https://webscope.sandbox.yahoo.com/catalog.php?datatype=s&did=70}}. The dataset tests the detection accuracy of various anomaly-types including outliers and change-points. The synthetic dataset consists of time-series with varying trend, noise and seasonality. The real dataset consists of time-series representing the metrics of various Yahoo services.

\textit{Numenta Anomaly Benchmark (NAB)} is a well-established collection of univariate time-series from real-world application domains\footnote{We invite the reader to consult \url{https://github.com/numenta/NAB/tree/master/data} for the entire directory of datasets.}. To be consistent with \cite{geiger2020tadgan}, we analyse Art, AdEx, AWS, Traf, and Tweets from the original collection.

 \paragraph{Multivariate datasets.} We consider for analysis the SWaT~\cite{10.1007/978-3-319-71368-7_8,Mathur2016SWaTAW} and CASAS~\cite{cook2012casas,dahmen2021indirectly} datasets.
\textit{SWaT} is collected from a cyber-physical system testbed that is a scaled down replica of an industrial water treatment plant. The data was collected every second for a total of 11 days, where for the first few days the system was operated normally, while for the remaining days certain cyber-physical attacks were launched. Following~\cite{10.1145/3447548.3467137}, we sample sensor data every every 5 seconds.

\textit{CASAS} is a collection of two weeks of sensor data from retirement homes. Each sensor reading has a label attached to it, according to the activity of elderly people recognised by human annotators. The 5 time series are collection of activities, grouped by the medical conditions established prior to the experimentation: Falling (F), More Time in Chair (MTC), Weakness (W), Slower Walking (SW), and Nocturia (N) for nightly time visits.
Although sensor readings give fine-grained information, we are interested in creating daily patient profiles. Hence, we collapse them into a single engendered activity with a start and end time\footnote{The start time corresponds to the first sensor reading for that activity, whereas the end time is the last reading.} for each consecutive sensor signal. We then create a time matrix structure $\mathcal{M}^{|d|,1440}$ where $|d|$ is the number of days the patient is monitored and 1440 represent the total number of minutes in a day. $\mathcal{M}[i,j]$ represents the label performed in minute $j$ of day $i$. Lastly, we densify each value $\mathcal{M}[i,j]$ with contextual and duration information corresponding to the label therein.

Finally, we create train and test splits of this data, caring to safeguard sequentiality of observations, and gathering the few anomalies into the test set for evaluation only. For this reason we name this Unsupervised-CASAS, dubbed U-CASAS, differing from the original CASAS \cite{cook2012casas,dahmen2021indirectly} since the latter interleaves anomalous sensor readings with normal instances, thus breaking the sequentiality of anomalies.
For each anomalous day encountered in the test set, we pad two days prior to it and two after. If the padding overlaps with another anomalous day, then we concatenate\footnote{The concatenation procedure merges common sequences. If the sequence contains more than one anomalous day within the two-day padding window, the padded sequence gets collapsed into a single one.} them and perform the padding procedure again. We delete the days assigned to the test set from the overall dataset and assign the rest as training. Moreover, because we employ a time-related strategy, we create time windows of 30 actions to detect anomalies.

\paragraph{Metrics}
Being all the enlisted datasets highly unbalanced, accuracy is misleading. Therefore, as done in \cite{suris2021hyperfuture},we use the F1 score to account for this challenge. Based on \cite{hundman2018detecting}, for the univariate and the CASAS datasets, we penalize high false positive rates and encourage the detection of true positive in a timely fashion. Since anomalies are rare events and come in collective sequences in real-world applications, we proceed as follows:
\begin{enumerate}
\item We record a true positive (TP) if any predicted window overlaps a true anomalous window.
\item We record a false negative (FN) if a true anomalous window does not overlap any predicted window.
\item We record a false positive (FP) if a predicted window does not overlap any true anomalous region.
\end{enumerate}
For U-CASAS, we also measure the g-measure which is the geometric mean of the product of recall (R) and precision (P), being a robust metric when classes are imbalanced \cite{dahmen2021indirectly}.

\input{tab/performances_univariate}
\input{tab/performances_casas}

\paragraph{Baselines}
 We include the following strategies as our baselines in this paper:
\begin{itemize}
    \item \textit{AE} \cite{baldi2012autoencoders} - We use a six-layer fully-connected autoencoder.
    \item \textit{ConvAE} \cite{maggipinto2018convolutional} - We have three layers of convolutional encoding interleaved with max pooling. The decoder has a specular composition as the encoder where the de-convolution is aided by two-dimensional up-sampling layers.
    \item \textit{LstmAE} \cite{sagheer2019unsupervised} - We use a deep stacked LSTM autoencoder with four layers. The first LSTM hidden and output vectors get passed to the second LSTM layer. The latent representation of the encoder gets then reconstructed in reverse order from the decoder.
    \item \textit{TadGAN*} \cite{geiger2020tadgan} - We use a one-layer bidirectional LSTM for the generator $E$, and a two-layer bidirectional LSTM for $G$. For the critic $D_z$ we use a fully connected layer, and two dense layers for $D_x$.
\end{itemize}

\noindent\textit{Implementation details.} For the first three baselines, we set the number of epochs to 30, the batch size to 32, and the learning rate to $10^{-3}$. For TadGAN*, we set the epochs to 30, the batch size to 64, the learning rate to $5\times10^{-4}$, and the iteration for the critic to 5. We use Adam as the optimisation function to train all the baselines.

For our proposed method HypAD, we took inspiration from an online available PyTorch implementation\footnote{\url{https://github.com/arunppsg/TadGAN}}. We leave the architecture of TadGAN unvaried, but we incorporate the hyperbolic transformation\footnote{\url{https://github.com/cvlab-columbia/hyperfuture}} as in \cite{suris2021hyperfuture}. The hyperparameters are the same in the original paper, but we use the Riemannian Adam\footnote{\url{https://github.com/geoopt/geoopt}} as optimisation function.

\subsection{Comparison to the state of the art} \label{sec:sota}

\textbf{HypAD defines a new state-of-the-art performance for univariate anomaly detection by having the highest average F1-scores of 0.631.}
As shown in Table~\ref{tab:performances_univariate}, HypAD outperforms the current best technique (TadGAN*, 0.600) by 5.17\%, as well as all baselines.

Table \ref{tab:performances_univariate} depicts the comparison of our proposed method, HypAD, with the baselines strategies described above. For reference purposes, we enlist also the performances of TadGAN as reported\footnote{We implemented TadGAN relying on PyTorch which does not match the reported performances in the original paper.} in \cite{geiger2020tadgan}. Here, we report the F1-score, and the mean and standard deviation over all datasets. \BM{The table rank all the methods based on their averaged F1 scores (the last column) across all datasets.} Notice that only TadGAN competes with HypAD. In this scenario, we have time windows of 100 values and we apply the poincaré distance directly on the windows and not on the reconstructed signal as with TadGAN\st{, HypAD experiences a slight drop in performances}. Nevertheless, with an average F1-score of $0.631$, HypAD is the best-performing method in the univariate case.

In Table~\ref{tab:performances_casas} we extend the evaluation of HypAD to the multivariate scenario of the U-CASAS dataset. We cannot include Isudra\cite{dahmen2021indirectly} because the underlying architecture uses a small amount of labels to conduct the selection of parameters for the execution. Additionally, Isudra is trained in a supervised fashion differing from all the other methods reported here. \st{However,} for completeness purposes, we also report the g-measure and its average across all datasets. \st{Here, the dominance of HypAD is more evident since the transformation of multivariate time window in hyperspace has the benefit of learning latent relationships between the input features. Notice that HypAD is the best-performing method according to both g-measure and F1-score.} \BM{As shown, HypAD surpasses TadGAN* by 32.51\% and ConvAE by 486.3\% according to the averaged F1-score.}

Finally, in Table~\ref{tab:performances_swat_fg}, we also extend comparison to the multivariate SWaT dataset. Here, following previous work~\cite{10.1145/3447548.3467137}, we also report the precision and recall in addition to the F1-score. HypAD outperforms the baseline TadGAN* (0.753 Vs.\ 0.722) but both techniques are behind the current state-of-the-art on multivariate anomaly detection NSIBF~\cite{10.1145/3447548.3467137}.
Observe however that HypAD achieves its highest F1-performance at the highest precision (0.996) among all other methods. This confirms that HypAD really detects anomalies which it is certain about, i.e.\ when it \emph{understands} the time series and is certain about what to expect, but cannot reconstruct the input signal due to an anomaly.

\input{tab/swat_permormances}

\input{tab/ablation}

\subsection{Ablation Studies}\label{sec:abl}

In Table~\ref{tab:ablation}, we demonstrate the effectiveness of HypAD on three different dataset types: Univariate, U-CASAS and SWaT. The first row shows the performance of the model in euclidean space. This corresponds to the model TadGAN* in Table~\ref{tab:performances_univariate},~\ref{tab:performances_casas} and~\ref{tab:performances_swat_fg}. Next we move the model to the hyperbolic space as discussed in Section~\ref{sec:hypRe}. This brings an improvement of 0.7\% and 18.6\% in the F1 score for the univariate and U-CASAS datasets, respectively. However for the SWaT  dataset the performance drops, which we discuss qualitatively in the following section. Finally, we integrate the hyperbolic uncertainty to the anomaly detection. As shown in the last two rows of the table, this brings an improvement of 4.5\% (0.604 vs 0.631) for the univariate cases, 7.8\% (0.397 vs 0.428) for the U-CASAS and 33\% for the SWaT. Clearly, \textbf{Uncertainty is fundamental in ameliorating anomaly detection.} For all the datasets, the best performance is achieved using the proposed hyperbolic uncertainty.

\noindent \textbf{Qualitative Ablation on SWaT:} In Figure \ref{fig:ablation1}, we demonstrate qualitative ablation on the SWaT dataset that consists of only one long-term signal. The three plots correspond to the three rows in the last column of Table~\ref{tab:ablation}. In all plots, the blue and green points represent the predicted anomaly scores and ground-truth anomalies, respectively. The red line denotes the anomaly detection threshold i.e. the blue points that lie above the red line are predicted anomalies. If a blue point lies above the red line but does not coincide with any green point (ground-truth), it is a false positive.

As visible in the Figure, the model in euclidean space (first plot) gives a lot of false positives but when moving to hyperbolic space (second plot) the model becomes more precise reducing false positives substantially. However note that this hyperbolic model also brings a part of positive anomalies (in the middle of the signal) below the threshold, reducing true positives. This explains the drop in the F-1 score for the SWaT dataset (cf. Table \ref{tab:ablation}). Interestingly, the integration of uncertainty recovers these points back in addition to others, increasing the true positives but not false positives, giving the best F-1 score of 0.753.

\input{fig/ablation1.tex}

\section{Introduction}

\input{fig/teaser}

\label{sec:intro}
Anomaly detection stands for detecting outliers (anomalies) in data, i.e.\ points that deviate significantly from the distribution of the data. Outlier detection, however, is an under-specified and consequently ill-posed task due to its inherent unsupervised nature. Anomaly detection strategies such as distance-based \cite{fan2006nonparametric,ghoting2008fast}, density-based \cite{lof,papadimitriou2003loci}, and subspace-based methods \cite{keller2012hics,lazarevic2005feature} have been pioneers in the literature. Additionally, autoencoders \cite{chen2017outlier,sarvari2021unsupervised}, and adversarial networks \cite{geiger2020tadgan} have given a substantial contribution. However, the literature neglects assessing the trustfulness of the predicted outcomes, namely their uncertainty.

Uncertainty is a measure of confidence for model prediction by learning directly from the data \cite{lakshminarayanan2016simple,kendall2017uncertainties} or by having access to additional instances \cite{gal2016dropout}. Uncertainty estimation has been a long-standing challenge in machine learning. Some works based on supervised learning have exploited uncertainty approximation techniques to improve performances in scenarios such as, object detection \cite{jiang2018acquisition,kuppers2020multivariate,9156274,neumann2018relaxed}, pose estimation \cite{vakhitov2021uncertainty}, and, more complicated, scene understanding \cite{badrinarayanan2017segnet}. Recently, uncertainty has become a cardinal aspect in ameliorating the development of unsupervised/self-supervised learning approaches \cite{fiery2021,8289350,suris2021hyperfuture}. Yet it remains a largely unexplored aspect in this context.

In this work, we propose a novel model based on Hyperbolic uncertainty for Anomaly Detection, which we dub HypAD. We leverage the current state of the art anomaly detection technique~\cite{geiger2020tadgan}, which detects anomalous events as those that are more arduous to reconstruct. The proposed hyperbolic uncertainty is integrated into the reconstruction loss according to the following modelling principle: the reconstruction loss is exponentially higher if the reconstructed signal turns out to be wrong and it was estimated with high certainty.
HypAD is therefore encouraged to reconstruct the original signal flawlessly with high certainty -- which makes the reconstruction loss zero -- or by attempting the best reconstruction with some degree of uncertainty. The latter is a self-regulating principle which is enabled by the hyperbolic Poincaré distance measure, exponentially larger for points which are certain, as we detail in Sec~\ref{sec:hypUn}.

By analysing uncertainty in the context of anomaly detection, in this work we provide a novel definition of anomaly detection. A detectable anomaly is an instance where the model is certain about its outcome, but which it reconstructs wrongly. Hence, we intertwine the anomaly status of a particular instance to both its detectability and its corresponding uncertainty in the hyperspace.

The main contributions of this work are:
\begin{itemize}
\item We propose the first model for anomaly detection based on hyperbolic uncertainty;
\item We analyse the model hyperbolic uncertainty in relation to the reconstruction error and discuss the self-regulating principle for self-supervised learning;
\item We integrate the estimated uncertainty into a state-of-the-art anomaly detection algorithm, improving the state-of-the-art on established anomaly detection benchmarks and elderly daily routine trajectories.
\end{itemize}

\section{Method}
In this section, we first discuss best practises in anomaly detection (Section \ref{sec:background}); then we detail the proposed hyperbolic uncertainty and its use for detecting anomalies (Section \ref{sec:hypAD}); finally we discuss the motivation for it (Section \ref{sec:mot}).

\input{fig/hyperTadGAN.tex}

\subsection{Background}
\label{sec:background}

The current state-of-the-art in univariate anomaly detection is a reconstruction-based technique~\cite{geiger2020tadgan} which additionally leverages a GAN critic score. TadGAN encodes the input data $x$ to a latent space and then decodes the encoded data. This encoding-decoding operation requires two mapping functions $E : X \rightarrow Z$ and $G : Z \rightarrow  X$.
The reconstruction operation can be given as $x \rightarrow E(x) \rightarrow G(E(x)) \rightarrow \widetilde{x} \approx x$. TadGAN leverages adversarial learning to train the two mappings by using two adversarial critics $D_x$ and $D_z$. The goal of $D_x$ is to distinguish between the real and the generated time series, while $D_z$ measures the performance of the mapping into latent space. The model is trained using the combination of Wasserstein loss~\cite{pmlr-v70-arjovsky17a} and Cycle consistency loss~\cite{8237506}.
TadGAN computes the reconstruction error $RE(x)$ between x and $\widetilde{x}$ using
three types of reconstruction  functions: \textbf{i.}\ Point-wise difference: considers the difference of values at every time stamp; \textbf{ii.}\ Area difference: is applied on signals of fixed lengths and measures the similarity between local regions; \textbf{iii.}\ Dynamic time warping: additionally handles time gaps between the two signals to calculate the reconstruction error.

To calculate the anomaly score, TadGAN first normalises the reconstruction error and critic scores by subtracting the mean and dividing by standard deviation. The normalised scores, $Z_{RE}(x)$ and $Z_{D_x}(x)$, are then combined using either their convex combination:
\begin{equation}
    s_c(x) = \alpha Z_{RE}(x) + (1 - \alpha) Z_{D_x}(x)
    \label{eq:anomaly_score_sum}
\end{equation}
or their product:
\begin{equation}
    s_p(x) = Z_{RE}(x) \odot Z_{D_x}(x)
    \label{eq:anomaly_score_prod}
\end{equation}

\subsection{Hyperbolic uncertainty for Anomaly detection (HypAD)}
\label{sec:hypAD}

We propose a novel model for anomaly detection in time series based on hyperbolic uncertainty. HypAD is a reconstruction-based model and minimises the reconstruction loss, given by a measure of the hyperbolic distance between the input signal and its reconstruction. In hyperbolic space, errors are exponentially larger when predictions are certain. Therefore, HypAD tends to predict either certain correct reconstructions or uncertain possibly mistaken reconstructions.

This leads, as we discuss in Sec.~\ref{sec:mot}, to a novel definition of detectable anomaly: i.e. the case of a large reconstruction error with high certainty.

\subsubsection{Hyperbolic Reconstruction Error}
\label{sec:hypRe}

The proposed HypAD is illustrated in Figure~\ref{fig:hypAD}. It integrates the machinery of hyperbolic neural networks into the reconstruction-based architecture of TadGAN. In HypAD the input signal $x$ is first passed through an encoder, then followed by a decoder sub-network. The output of the decoder $G(E(x))$ as well as the original signal $x$ are mapped to the hyperspace, shown as the dotted red edge box with red background.

An $n$-dimensional hyperspace is a Riemannian geometry with a constant negative sectional curvature. As in~\cite{Khrulkov_2020_CVPR,suris2021hyperfuture}, we adopt the Poincaré ball model of hyperspaces, given by the manifold $\mathbb{D}_n = \{x \in R^n : \|x\| < 1\}$ endowed with the Riemmanian metric $g^\mathbb{D}(x) = \lambda_x^2g^E$,
where $\lambda_x =\frac{2}{1-\|x\|^2}$ is the \textit{conformal factor}
and $g^E = I_n$ is the Euclidean
metric tensor. For details, see~\cite{riemannian,smoothmanifold}.

In order to map $x$ and $G(E(x))$ to the Poincaré ball, we leverage an \textit{exponential map} centered at $\bf{0}$~\cite{suris2021hyperfuture}. This is followed by a hyperbolic feed-forward layer~\cite{NEURIPS2018_dbab2adc} to estimate the corresponding hyperbolic embeddings $h$ and $\widetilde{h}$, shown as solid green boxes in Figure~\ref{fig:hypAD}. Finally, the two hyperbolic embeddings are compared using the Poincaré distance, formulated as follows:
\begin{equation}
{RE}(x)  =  \cosh^{-1}\Bigg(1 + 2{\frac{\|h-\widetilde{h}\|^2}{\big(1-\|h\|^2\big)\big(1-\|\widetilde{h}\|^2\big)}\Bigg)}
\label{eq:poincare}
\end{equation}
where $\|h\|^2$ and $\|\widetilde{h}\|^2$ are the distances of the embeddings from the center of the Poincaré ball.
Note that the same reconstruction error function $RE(x)$, is used at training as well as inference time.

\subsubsection{Hyperbolic Uncertainty }
\label{sec:hypUn}
A key property of the Poincaré ball is that the distance between two points grows exponentially as we move away from the origin. This means that an erroneous reconstruction towards the circumference of the disk is penalised exponentially more than an erroneous reconstruction close to the centre. This leads to the useful tendency of HypAD to either predict a matched reconstruction ($\widetilde{h}$ and $h$ are close by) or an unmatched reconstruction towards the origin ($\widetilde{h}$ and $h$ are far-away, $\|h\|^2$ and $\|\widetilde{h}\|^2$ are small), in order to minimize Eq. \eqref{eq:poincare}.

Hence, the distance of the reconstruction to the origin provides a natural estimate of the model's uncertainty, referred to as hyperbolic uncertainty, $U(X)$, thus formulated:
\begin{equation}
{U}(X)  =  1 - \| \widetilde{h}\|^2
\label{eq:hypUn}
\end{equation}

The smaller the distance from the origin, the more uncertain is the model.

\subsubsection{Combining Hyperbolic Uncertainty with Reconstruction Error and Critic Score}
\label{sec:combine}

Hyperbolic uncertainty ${U}(x)$ is integrated into the anomaly score as follows:
\begin{equation}
    s_u(x) =  Z_{RE}(x) \odot Z_{D_x}(x) \odot (1-{U}(x))
    \label{eq:uncer}
\end{equation}

Equation \ref{eq:uncer} brings together the reconstruction error $Z_{RE}(x)$ (the larger the error, the more likely the anomaly) with the critic score $Z_{D_x}(x)$ (larger critic scores point to anomalies) and the model certainty: $1-{U}(x)$.

The simple multiplication formulation of the certainty of the model reduces the scores of anomalies when HypAD is not confident of the reconstructions. While being simple, this outperforms the current state-of-the-art, as we show in Section~\ref{sec:expres}.

\subsection{Motivation for HypAD}
\label{sec:mot}

HypAD takes motivation from a key idea: \textit{a detectable anomaly is one where the model is certain, and which it predicts wrongly}. In other words, if the model encounters a known pattern, which it knows how to reconstruct, then it will call it anomalous, if the reconstruction does not match the input signal.

The principled formulation of hyperbolic uncertainty is paramount towards this goal: HypAD predicts a reconstruction as uncertain if it is doubtful that it may be wrong.

\input{fig/cosine_barplot.tex}

\BM{Figure~\ref{fig:cosine_barplot} illustrates this key concept for all the datasets. The first, second and third rows correspond to the univariate, U-CASAS and SWaT datasets respectively.}
The bar plot depicts the average cosine distances between the input signals and their reconstructions, against specific intervals of certainty, along the x-axis.
The higher the cosine distances, the more distinct the reconstruction is from the provided signals\footnote{The Poincaré ball model is conformal to the Euclidean space and it preserves the same angles~\cite{shimizu2021hyperbolic}.}.
\BM{
Note that for the first two rows, the initial three plots (columns) correspond to the signals that report the best improvement in F1-score and the last plot corresponds to the signal with worst improvement. Since SWaT contains only a single long-term signal, we report only its
corresponding bar-plot.}

Observe \BM{in the second row}, how HypAD learns to correctly assign lower certainty to wronger estimates for the cases of Fall, Weakness, and Nocturia. For the last signal SlowerWalking, HypAD fails to learn a meaningful uncertainty and labels all reconstructions as certain. It is however notable that the representation is still interpretable and the case of failure discernible.

\BM{Similar trends can be seen for the case of Univariate and SWaT datasets.}

\input{tab/datasets}

\section{Conclusions}
We have proposed a novel model for anomaly detection based on hyperbolic uncertainty, HypAD.
The proposed hyperbolic uncertainty allows HypAD to self-adjusts its output, encouraging the model to either predict a correct reconstruction or a less certain wrong reconstruction. This benefits anomaly detection in two ways: it provides better reconstructions of the signal (the deviation from those being anomalous) and it yields a measure of certainty.
This is a novel viewpoint on anomaly detection: detectable anomaly instances are those which are certain but wrongly predicted.

\section{Results}\label{sec:expres}

We compare HypAD against the current best univariate anomaly detector TadGAN~\cite{geiger2020tadgan} on a benchmark of 11 time series, and extend the comparison to 2 multivariate sensor datasets: one comprising a water treatment plant and one comprising daily activities in elderly residences.
First, we introduce the benchmarks (Sec.~\ref{sec:dataset}), then we compare against baselines and the state-of-the-art (Sec.~\ref{sec:sota}), finally we conduct ablative studies on the importance of uncertainty for performance and the reduction of false alarms (Sec.~\ref{sec:abl}).

\subsection{Datasets, metrics and experimental setup}\label{sec:dataset}

Table~\ref{tab:datasets} summarizes the main characteristics of the datasets, which we coarsely divide into univariate, main test bed of our baseline TadGAN~\cite{geiger2020tadgan}, and multivariate, to which we extend comparison. In the table, we report the sources of signals (NASA, Yahoo, NAB, CASAS and SWaT) and the datasets within each source (SMAP, MSL, A1 etc.), cf.\ detailed description later in the section.

In the table, the \textit{number of signals} is the number of time series within the datasets. Note that univariate datasets are composed of multiple signals, while the multivariate only comprise single large time series.
The \textit{number of anomalies} counts the instances, within the time series, labelled as anomalous. These are further detailed as \textit{point anomalies}, single anomalous values at a specific point in time, or \textit{collective anomalies}, sets of contiguous times which are altogether anomalous. Yahoo is the sole dataset with point anomalies, and the only with \textit{synthetic} sequences A2, A3, A4.
We also report the \textit{percentage of total} anomalous points and, following \cite{geiger2020tadgan}, for the univariate datasets, the \textit{number of out-of-distribution} points, exceeding the means by more than $4\sigma$.

\paragraph{Univariate datasets} \textit{NASA} includes two spacecraft telemetry datasets\footnote{\url{https://s3-us-west-2.amazonaws.com/telemanom/data.zip}.} based on the Mars Science Laboratory (MSL) and the Soil Moisture Active Passive (SMAP) signals. The former consists of scientific and housekeeping engineering data taken from the Rover Environmental Monitoring Station aboard the Mars Science Laboratory. The latter includes measurements of soil moisture and freeze/thaw state from space for all non-liquid water surfaces globally within the top layer of the Earth.

We analyse \textit{Yahoo} datasets based on real production traffic to Yahoo computing systems. Additionally, we consider three synthetic datasets coming from the same source\footnote{\url{https://webscope.sandbox.yahoo.com/catalog.php?datatype=s&did=70}}. The dataset tests the detection accuracy of various anomaly-types including outliers and change-points. The synthetic dataset consists of time-series with varying trend, noise and seasonality. The real dataset consists of time-series representing the metrics of various Yahoo services.

\textit{Numenta Anomaly Benchmark (NAB)} is a well-established collection of univariate time-series from real-world application domains\footnote{We invite the reader to consult \url{https://github.com/numenta/NAB/tree/master/data} for the entire directory of datasets.}. To be consistent with \cite{geiger2020tadgan}, we analyse Art, AdEx, AWS, Traf, and Tweets from the original collection.

 \paragraph{Multivariate datasets} We consider for analysis the SWaT~\cite{10.1007/978-3-319-71368-7_8,Mathur2016SWaTAW} and CASAS~\cite{cook2012casas,dahmen2021indirectly} datasets.
\textit{SWaT} is collected from a cyber-physical system testbed that is a scaled down replica of an industrial water treatment plant. The data was collected every second for a total of 11 days, where for the first few days the system was operated normally, while for the remaining days certain cyber-physical attacks were launched. Following~\cite{10.1145/3447548.3467137}, we sample sensor data every every 5 seconds.

\textit{CASAS} is a collection of two weeks of sensor data from retirement homes. Each sensor reading has a label attached to it, according to the activity of elderly people recognised by human annotators. The 5 time series are a collection of activities, grouped by the medical conditions established prior to the experimentation: Falling (F), More Time in Chair (MTC), Weakness (W), Slower Walking (SW), and Nocturia (N) for nightly time visits.
Although sensor readings give fine-grained information, we are interested in creating daily patient profiles. Hence, we collapse them into a single engendered activity with a start and end time\footnote{The start time corresponds to the first sensor reading for that activity, whereas the end time is the last reading.} for each consecutive sensor signal. We then create a time matrix structure $\mathcal{M}^{|d|,1440}$ where $|d|$ is the number of days the patient is monitored and 1440 represent the total number of minutes in a day. $\mathcal{M}[i,j]$ represents the label performed in minute $j$ of day $i$. Lastly, we densify each value $\mathcal{M}[i,j]$ with contextual and duration information corresponding to the label therein.

Finally, we create train and test splits of this data, caring to safeguard sequentiality of observations, and gathering the few anomalies into the test set for evaluation only. For this reason we name this Unsupervised-CASAS, dubbed U-CASAS, differing from the original CASAS \cite{cook2012casas,dahmen2021indirectly} since the latter interleaves anomalous sensor readings with normal instances, thus breaking the sequentiality of anomalies.
For each anomalous day encountered in the test set, we pad two days prior to it and two after. If the padding overlaps with another anomalous day, then we concatenate\footnote{The concatenation procedure merges common sequences. If the sequence contains more than one anomalous day within the two-day padding window, the padded sequence gets collapsed into a single one.} them and perform the padding procedure again. We delete the days assigned to the test set from the overall dataset and assign the rest as training. Moreover, because we employ a time-related strategy, we create time windows of 30 actions to detect anomalies.

\paragraph{Metrics}
Being all the enlisted datasets highly unbalanced, accuracy is misleading. Therefore, as done in \cite{suris2021hyperfuture}, we use the F1 score to account for this challenge. Notice that we do not use the cumulative F1 score as proposed in \cite{garg2021evaluation} to evaluate the performances because not all datasets contain anomalous events; rather they contain anomalous data points. Based on \cite{hundman2018detecting}, for the univariate and the CASAS datasets, we penalize high false positive rates and encourage the detection of true positives in a timely fashion. Since anomalies are rare events and come in collective sequences in real-world applications, we proceed as follows:
\begin{enumerate}
\item We record a true positive (TP) if any predicted window overlaps a true anomalous window.
\item We record a false negative (FN) if a true anomalous window does not overlap any predicted window.
\item We record a false positive (FP) if a predicted window does not overlap any true anomalous region.
\end{enumerate}
For U-CASAS, we also measure the g-measure which is the geometric mean of the product of recall (R) and precision (P), being a robust metric when classes are imbalanced \cite{dahmen2021indirectly}.

\input{tab/performances_univariate}
\input{tab/performances_casas}

\paragraph{Baselines}
 We include the following strategies as our baselines in this paper:
\begin{itemize}
    \item \textit{AE} \cite{baldi2012autoencoders} - We use a six-layer fully-connected autoencoder.
    \item \textit{ConvAE} \cite{maggipinto2018convolutional} - We have three layers of convolutional encoding interleaved with max pooling. The decoder has a specular composition as the encoder where the de-convolution is aided by two-dimensional up-sampling layers.
    \item \textit{LstmAE} \cite{sagheer2019unsupervised} - We use a deep stacked LSTM autoencoder with four layers. The first LSTM hidden and output vectors get passed to the second LSTM layer. The latent representation of the encoder gets then reconstructed in reverse order from the decoder.
    \item \textit{TadGAN*} \cite{geiger2020tadgan} - We use a one-layer bidirectional LSTM for the generator $E$, and a two-layer bidirectional LSTM for $G$. For the critic $D_z$ we use a fully connected layer, and two dense layers for $D_x$.
\end{itemize}

\noindent\textit{Implementation details.} For the first three baselines, we set the number of epochs to 30, the batch size to 32, and the learning rate to $10^{-3}$. For TadGAN*, we set the epochs to 30, the batch size to 64, the learning rate to $5\times10^{-4}$, and the iteration for the critic to 5. We use Adam as the optimisation function to train all the baselines.

For our proposed method HypAD, we took inspiration from an online available PyTorch implementation\footnote{\url{https://github.com/arunppsg/TadGAN}}. We leave the architecture of TadGAN unvaried, but we incorporate the hyperbolic transformation\footnote{\url{https://github.com/cvlab-columbia/hyperfuture}} as in \cite{suris2021hyperfuture}. The hyperparameters are the same in the original paper, but we use the Riemannian Adam\footnote{\url{https://github.com/geoopt/geoopt}} as optimisation function.

\subsection{Comparison to the state of the art} \label{sec:sota}

\textbf{HypAD defines a new state-of-the-art performance for univariate anomaly detection by having the highest average F1-scores of 0.631.}
In Table~\ref{tab:performances_univariate}, HypAD outperforms the current best technique (TadGAN*, 0.600) by 5.17\%, as well as all baselines by a large margin.
Note that we also report the original paper~\cite{geiger2020tadgan} performance of TadGAN in the first row (0.629), which we could not reproduce with the available PyTorch code (cf.\ Sec.~\ref{sec:dataset}).
In the Table, the column \textit{F1} reports mean $\mu$ and standard deviation $\sigma$ over all datasets.  Looking at $\sigma=0.075$, HypAD also appears as the most consistent performer.
Considering the F1-score, the largest performance gain of HypAD Vs TadGAN are on NAB and NASA datasets, while it is outperformed more largely on the A2, A3 and A4 Yahoo datasets, the sole synthetic univariate ones.

In Table~\ref{tab:performances_casas}, we extend the evaluation of HypAD to the multivariate U-CASAS dataset. We cannot include Isudra~\cite{dahmen2021indirectly} because the underlying architecture uses a small amount of labels to conduct the selection of parameters for the execution. Additionally, Isudra is trained in a supervised fashion differing from all the other methods reported here. For completeness, we also report the g-measure~\cite{dahmen2021indirectly} and its average across all datasets.
As shown in the table, HypAD surpasses TadGAN* by 32.51\% in terms of average F1-score (0.428 Vs.\ 0.323).

Finally, in Table~\ref{tab:performances_swat_fg}, we also extend comparison to the multivariate SWaT dataset. Here, following previous work~\cite{10.1145/3447548.3467137}, we also report the precision and recall in addition to the F1-score. HypAD outperforms the baseline TadGAN* (0.753 Vs.\ 0.722) but both techniques are behind the current state-of-the-art on multivariate anomaly detection, NSIBF~\cite{10.1145/3447548.3467137}.
Observe however that HypAD achieves its highest F1-performance at the highest precision (0.996) among all other methods. This confirms that HypAD really detects anomalies which it is certain about, i.e.\ when it \emph{understands} the time series and it is certain about what to expect, but cannot reconstruct the input signal due to an anomaly.

\input{tab/swat_permormances}

\input{tab/ablation}

\subsection{Ablation Studies}\label{sec:abl}

In Table~\ref{tab:ablation}, we analyze the importance of the hyperbolic embedding and the use of uncertainty for anomaly detection on the univariate datasets, as well as the multivariate U-CASAS and SWaT.
The first row shows the performance of the model in Euclidean space (average F1-score across datasets). This corresponds to TadGAN* in Table~\ref{tab:performances_univariate},~\ref{tab:performances_casas} and~\ref{tab:performances_swat_fg}. In the second row, we report performance for the hyperbolic TadGAN*, without including uncertainty (cf.~Sec.~\ref{sec:hypRe}). This improves marginally over the univariate (0.604 Vs.\ 0.600) and more importantly over the U-CASAS datasets (0.397 Vs.\ 0.323), but it decreases performance in SWaT (0.566 Vs.\ 0.722), which we further analyze in the following subsection.

The complete proposed HypAD model, in the third row, improves consistently on both ablative variants. HypAD yields 0.631 on the univariate datasets, 0.428 on U-CASAS, and 0.753 on SWaT. Improvements \emph{wrt} not using uncertainty are large, respectively 4.5\%, 7.8\% and 33\%. We therefore conclude that \textbf{uncertainty is fundamental for ameliorating anomaly detection.}

\noindent \textbf{Qualitative Ablation on SWaT:} In Figure \ref{fig:ablation1}, we demonstrate qualitative ablation on the SWaT dataset, which consists of a single long signal. The three plots correspond to the three ablative variants of Table~\ref{tab:ablation}. In all plots, the blue and green points represent the predicted anomaly scores and the ground-truth anomalies, respectively. The red line denotes the anomaly detection threshold i.e.\ blue points above the red line are the predicted anomalies. False positives are blue points above the red line threshold outside the green (ground-truth) anomalous regions.

As it shows in the figure, the Euclidean model (first plot) yields many false positives; the hyperbolic model w/o uncertainty (second plot) reduces the number of false positives substantially, but it also misses anomalies, esp.\ in the middle signal part. This explains the drop in the F-1 score for the SWaT dataset (cf.\ Table~\ref{tab:ablation}). Integrating hyperbolic uncertainty (proposed HypAD, third plot) recovers detection of these anomalies, it increases true positives but maintains the number of false positives low, thus yielding the best F1 score of 0.753.

\input{fig/ablation1.tex}

\section{Introduction}
\label{sec:intro}

\input{fig/teaser}

Anomaly detection stands for detecting outliers (anomalies) in data, i.e.\ points that deviate significantly from the distribution of the data. Outlier detection, however, is an under-specified and consequently ill-posed task due to its inherent unsupervised nature. Anomaly detection strategies such as distance-based \cite{fan2006nonparametric,ghoting2008fast}, density-based \cite{lof,papadimitriou2003loci}, and subspace-based methods \cite{keller2012hics,lazarevic2005feature} have been pioneers in the literature. Additionally, autoencoders \cite{chen2017outlier,sarvari2021unsupervised}, and adversarial networks \cite{geiger2020tadgan} have given a substantial contribution. However, the literature neglects assessing the trustfulness of the predicted outcomes, namely their uncertainty.

Uncertainty is a measure of model confidence, which may be learnt from the data~\cite{lakshminarayanan2016simple,kendall2017uncertainties} or by the use of extra instances~\cite{gal2016dropout}. Uncertainty estimation has been a long-standing challenge in machine learning. Most recently, it has been successfully adopted to improve performance on object detection~\cite{jiang2018acquisition,kuppers2020multivariate,9156274,neumann2018relaxed}, pose estimation~\cite{vakhitov2021uncertainty}, unsupervised and self-supervised learning~\cite{fiery2021,8289350,suris2021hyperfuture}. Yet, uncertainty remains largely unexplored in the context of anomaly detection.

In this work, we propose a novel model based on Hyperbolic uncertainty for Anomaly Detection, which we dub HypAD. We leverage the current state-of-the-art technique for anomaly detection in univariate time series, TadGAN~\cite{geiger2020tadgan}. TadGAN detects anomalies by attempting to reconstruct the input signal, making use of an LSTM sequence encoding and two GAN critics, cf.\ Sec.~\ref{sec:background}. We introduce uncertainty into the anomaly detector: we map the input and the reconstructed signal into a hyperbolic space, where the signals additionally have an uncertainty score; and we train the novel embeddings end-to-end with a Poincaré distance loss, cf.\ Sec.~\ref{sec:hypAD}.

The proposed HypAD uses uncertainty to discern whether the reconstruction error is large because the signal is anomalous, or simply because the model cannot reconstruct it well. In the former case, HypAD is certain about the reconstruction (e.g.\ most signal is well-behaved and the model expects known patterns) but its reconstruction is wrong, as a part of the signal is anomalous. In the latter, HypAD downgrades its anomaly score because it is not certain about the signal reconstruction. This may be because of a complex pattern, which the model did not have enough capacity to learn. The larger uncertainty indicates that the larger reconstruction error may be due to anomaly or to a model failure in the reconstruction. (See discussion in Sec~\ref{sec:hypUn}.)

Thanks to uncertainty, HypAD outperforms the state-of-the-art univariate anomaly detector TadGAN~\cite{geiger2020tadgan} on the established univariate benchmarks of NASA, Yahoo, Numenta Anomaly Benchmark~\cite{lavin2015evaluating}, as well as on two multivariate datasets of daily activities in elderly home residences CASAS~\cite{cook2012casas} and industrial water treatment plant SWaT~\cite{Mathur2016SWaTAW}. As we show in experimental results in Sec.~\ref{sec:expres}, reducing anomaly scores in uncertain cases also yields fewer false alarms (the model achieves best F1 performance with larger precision).

The main contributions of this work are:
\begin{itemize}
\item We propose the first model for anomaly detection based on hyperbolic uncertainty;
\item We propose the novel key idea of \emph{detectable anomaly}: an instance is anomalous when the model is certain about it but wrong;
\item We integrate the estimated uncertainty into a state-of-the-art univariate anomaly detector and consistently outperform it on established univariate and multivariate datasets.
\end{itemize}

\section{Method}
In this section, we first discuss best practises in anomaly detection (Section \ref{sec:background}); then we detail the proposed hyperbolic uncertainty and its use for detecting anomalies (Section \ref{sec:hypAD}); finally we discuss the motivation for it (Section \ref{sec:mot}).

\input{fig/hyperTadGAN.tex}

\subsection{Background}
\label{sec:background}

The current state-of-the-art in univariate anomaly detection is a reconstruction-based technique~\cite{geiger2020tadgan} which additionally leverages a GAN critic score. TadGAN encodes the input data $x$ to a latent space and then decodes the encoded data. This encoding-decoding operation requires two mapping functions $E : X \rightarrow Z$ and $G : Z \rightarrow  X$.
The reconstruction operation can be given as $x \rightarrow E(x) \rightarrow G(E(x)) \rightarrow \widetilde{x} \approx x$. TadGAN leverages adversarial learning to train the two mappings by using two adversarial critics $D_x$ and $D_z$. The goal of $D_x$ is to distinguish between the real and the generated time series, while $D_z$ measures the performance of the mapping into latent space. The model is trained using the combination of Wasserstein loss~\cite{pmlr-v70-arjovsky17a} and Cycle consistency loss~\cite{8237506}.
TadGAN computes the reconstruction error $RE(x)$ between x and $\widetilde{x}$ using
three types of reconstruction  functions: \textbf{i.}\ Point-wise difference: considers the difference of values at every time stamp; \textbf{ii.}\ Area difference: is applied on signals of fixed lengths and measures the similarity between local regions; \textbf{iii.}\ Dynamic time warping: additionally handles time gaps between the two signals to calculate the reconstruction error.

To calculate the anomaly score, TadGAN first normalises the reconstruction error and critic scores by subtracting the mean and dividing by standard deviation. The normalised scores, $Z_{RE}(x)$ and $Z_{D_x}(x)$, are then combined using their product:
\begin{equation}
    s_p(x) = Z_{RE}(x) \odot Z_{D_x}(x)
    \label{eq:anomaly_score_prod}
\end{equation}

\subsection{Hyperbolic uncertainty for Anomaly detection (HypAD)}
\label{sec:hypAD}

We propose a novel model for anomaly detection in time series based on hyperbolic uncertainty. HypAD is a reconstruction-based model and minimises the reconstruction loss, given by a measure of the hyperbolic distance between the input signal and its reconstruction. In hyperbolic space, errors are exponentially larger when predictions are certain. Therefore, HypAD tends to predict either certain correct reconstructions or uncertain possibly mistaken reconstructions.

This leads, as we discuss in Sec.~\ref{sec:mot}, to a novel definition of detectable anomaly: i.e. the case of a large reconstruction error with high certainty.

\subsubsection{Hyperbolic Reconstruction Error}
\label{sec:hypRe}

The proposed HypAD is illustrated in Figure~\ref{fig:hypAD}. It integrates the machinery of hyperbolic neural networks into the reconstruction-based architecture of TadGAN. In HypAD the input signal $x$ is first passed through an encoder, then followed by a decoder sub-network. The output of the decoder $G(E(x))$ as well as the original signal $x$ are mapped to the hyperspace, shown as the red edge box with red background.

An $n$-dimensional hyperspace is a Riemannian geometry with a constant negative sectional curvature. As in~\cite{Khrulkov_2020_CVPR,suris2021hyperfuture}, we adopt the Poincaré ball model of hyperspaces, given by the manifold $\mathbb{D}_n = \{x \in R^n : \|x\| < 1\}$ endowed with the Riemmanian metric $g^\mathbb{D}(x) = \lambda_x^2g^E$,
where $\lambda_x =\frac{2}{1-\|x\|^2}$ is the \textit{conformal factor}
and $g^E = I_n$ is the Euclidean
metric tensor. For details, see~\cite{riemannian,smoothmanifold}.

In order to map $x$ and $G(E(x))$ to the Poincaré ball, we leverage an \textit{exponential map} centered at $\bf{0}$~\cite{suris2021hyperfuture}. This is followed by a hyperbolic feed-forward layer~\cite{NEURIPS2018_dbab2adc} to estimate the corresponding hyperbolic embeddings $h$ and $\widetilde{h}$, shown as solid green boxes in Figure~\ref{fig:hypAD}. Finally, the two hyperbolic embeddings are compared using the Poincaré distance, formulated as follows:
\begin{equation}
{RE}(x)  =  \cosh^{-1}\Bigg(1 + 2{\frac{\|h-\widetilde{h}\|^2}{\big(1-\|h\|^2\big)\big(1-\|\widetilde{h}\|^2\big)}\Bigg)}
\label{eq:poincare}
\end{equation}
where $\|h\|^2$ and $\|\widetilde{h}\|^2$ are the distances of the embeddings from the center of the Poincaré ball.
Note that the same reconstruction error function $RE(x)$, is used at training as well as inference time.

\subsubsection{Hyperbolic Uncertainty }
\label{sec:hypUn}
A key property of the Poincaré ball is that the distance between two points grows exponentially as we move away from the origin. This means that an erroneous reconstruction towards the circumference of the disk is penalised exponentially more than an erroneous reconstruction close to the centre. This leads to the useful tendency of HypAD to either predict a matched reconstruction ($\widetilde{h}$ and $h$ are close by) or an unmatched reconstruction towards the origin ($\widetilde{h}$ and $h$ are far-away, $\|h\|^2$ and $\|\widetilde{h}\|^2$ are small), in order to minimize Eq. \eqref{eq:poincare}.

Hence, the distance of the reconstruction to the origin provides a natural estimate of the model's uncertainty, referred to as hyperbolic uncertainty, $U(X)$, thus formulated:
\begin{equation}
{U}(X)  =  1 - \| \widetilde{h}\|^2
\label{eq:hypUn}
\end{equation}

The smaller the distance from the origin, the more uncertain is the model.

\subsubsection{Combining Hyperbolic Uncertainty with Reconstruction Error and Critic Score}
\label{sec:combine}

Hyperbolic uncertainty ${U}(x)$ is integrated into the anomaly score as follows:
\begin{equation}
    s_u(x) =  Z_{RE}(x) \odot Z_{D_x}(x) \odot (1-{U}(x))
    \label{eq:uncer}
\end{equation}

Equation \ref{eq:uncer} brings together the reconstruction error $Z_{RE}(x)$ (the larger the error, the more likely the anomaly) with the critic score $Z_{D_x}(x)$ (larger critic scores point to anomalies) and the model certainty: $1-{U}(x)$.

The simple multiplication formulation of the certainty of the model reduces the scores of anomalies when HypAD is not confident of the reconstructions. While being simple, this outperforms the current state-of-the-art, as we show in Section~\ref{sec:expres}.

\subsection{Motivation for HypAD}
\label{sec:mot}

HypAD takes motivation from a key idea: \textit{a detectable anomaly is one where the model is certain, but it predicts wrongly}. In other words, if the model encounters a known pattern, which it knows how to reconstruct, then it will call it anomalous, if the reconstruction does not match the input signal.

The principled formulation of hyperbolic uncertainty is paramount towards this goal: HypAD predicts a reconstruction as uncertain if it is doubtful that it may be wrong.

\input{fig/cosine_barplot.tex}

Fig.~\ref{fig:cosine_barplot} illustrates this key concept for all the datasets. The first, second and third rows correspond to the univariate, U-CASAS and SWaT datasets respectively.
The bar plot depicts the average cosine distances between the input signals and their reconstructions, against specific intervals of uncertainty, along the x-axis.
The higher the cosine distances, the more distinct the reconstruction is from the provided signals\footnote{The Poincaré ball model is conformal to the Euclidean space and it preserves the same angles~\cite{shimizu2021hyperbolic}.}.
Note that for the first two rows, the initial three plots (columns) correspond to the signals that report the best improvement in F1-score and the last plot corresponds to the signal with worst improvement. A single bar-plot is reported for SWaT because this consists of a single long-term signal.

Observe in the second row, how HypAD learns to correctly assign higher uncertainty to wronger estimates for the cases of Fall, Weakness, and Nocturia. For the last signal SlowerWalking, HypAD fails to learn a meaningful uncertainty and labels all reconstructions as certain. It is however notable that the representation is still interpretable and the case of failure discernible. Trends are similar for the cases of Univariate and SWaT datasets.

\input{tab/datasets}

\section{Related works}
\label{sec:related}

To the best of our knowledge, this is the first work to have combined anomaly detection with uncertainty estimation, and the first work to have further proposed hyperbolic uncertainty for it. Previous work relates to ours from three main perspectives, which we review here: uncertainty estimation techniques, anomaly detection in time series and hyperbolic neural networks.

\subsection{Uncertainty Estimation Techniques}\label{sec:related-uncertainty}

We identified two different strategies of approximating uncertainty. \textit{Ensemble-based posterior approximation} uses several weak models to make naive predictions and combine them according to a consensus function into a more complex predictive model \cite{dietterich2000ensemble}. One of the most popular approaches to uncertainty estimation based on ensembles is Monte Carlo (MC) Dropout. It drops neurons on every layer during training and test phases \cite{gal2016dropout}.

\textit{Generative models for aleatory modelling} use an additional latent variable to make stochastic predictions and evaluate the uncertainty of the model. Generative Adversarial Networks (GANs) \cite{goodfellow2014generative} play a minmax game where the discriminator needs to distinguish between real examples and the generated outcomes. GANs have state-of-the-art performances and we build on top of that by attaching hyperspace mapping layers to estimate the uncertainty of the model. Another interesting approach to estimate uncertainty is by using energy-based models \cite{du2019implicit,salakhutdinov2009deep,xie2016theory}. They learn an energy function that models the compatibility of the input and the output. Our method transcends energy-based models because the integrated hyperbolic uncertainty mechanism does not suffer from cold- nor warm-start problems \cite{zhang2021dense} which undermine the training complexity \cite{xie2018cooperative}.

\subsection{Anomaly Detection in Time Series}\label{sec:related_anomaly}
We identified five categories of methods proposed in the literature for anomaly detection in time series. \textit{Distance-based} outlier detectors consider the distance of a point from its k-nearest neighbours \cite{conf/vldb/KnorrN98, angiulli2002fast,ghoting2008fast}. \textit{Density-based} methods~\cite{lof,papadimitriou2003loci,He2003DiscoveringCL, 10.1145/3447548.3467137, 10.1145/3292500.3330672,WangCNLCT21, su2019robust} take into account the density of the point and its neighbours. \textit{Prediction-based methods} \cite{benkabou2021local,unsupahmad,conf/kdd/HundmanCLCS18} calculate the difference between the predicted value and the true value to detect anomalies. \textit{Reconstruction based} methods \cite{An2015VariationalAB,Malhotra2016LSTMbasedEF,10.1145/3447548.3467174} compare the input signal and the reconstructed one in the output layer typically by using autoencoders. These methods assume that anomalies are difficult to reconstruct and are lost when the signal gets mapped to lower dimensions, thus a higher reconstruction error means higher anomaly score. \cite{Malhotra2016LSTMbasedEF} uses an LSTM autoencoder for multi-sensor anomaly detection. \cite{chen2017outlier,sarvari2021unsupervised} use an ensemble of autoencoders to boost performances by focusing on learning the inlier characteristics at each iteration.~\cite{li2021multivariate} uses a hierarchical variational auto-encoder with two stochastic latent variables to learn the temporal and inter-metric embeddings for multivariate data.SISVAE~\cite{li2020anomaly} uses a variational auto-encoder with a smoothness-inducing prior over possible estimations to capture latent temporal structures of time series without relying on the assumption of constant noise. Recently, \textit{GANs} have been employed to detect anomalies in time series data. Our method also lies in this category. MAD-GAN \cite{Li2019MADGANMA} combines the discriminator output and reconstruction error to detect anomalies in multivariate time series. BeatGAN~\cite{ijcai2019-616} uses encoder-decoder generator with a modified time-warping-based data augmentation to detect anomalies in medical ECG inputs. TadGAN~\cite{geiger2020tadgan} uses a cycle-consistent GAN architecture with an encoder-decoder generator and additionally proposes several ways to compute reconstruction error and its combination with the critic outputs. We build on top of TadGAN's architecture by incorporating the hyperbolic mapping layer to the reconstructed time-windows to assess the uncertainty of the detector.

\subsection{Hyperbolic Neural Networks}\label{sec:related_hnn}
Deep representation learning in hyperspaces has gained momentum after the pioneering work of hyperNNs~\cite{NEURIPS2018_dbab2adc} that generalises Euclidean operations (e.g. matrix multiplications) to their counterparts in hyperspace. The authors propose analogue counterparts in hyperspace of neural network components such as fully-connected (FC) layers, multinomial logistic regression (MLR) and recurrent neural networks. Furthermore, methods like Einstain midpoint~\cite{gulcehre2018hyperbolic} and Fréchet mean~\cite{Lou2020DifferentiatingTT} propose different ways of aggregating features in hyperspace. The work in \cite{shimizu2021hyperbolic} extends hyperNN and proposes Poincaré split/concatenation operations, generalising the convolutional layer to hyperspace.~\cite{NEURIPS2019_103303dd,chami2019hyperbolic} propose hyperbolic graph neural networks, leveraging hyperNNs.

Thus formulated, hyperNNs have mainly been adopted to improve performance by leveraging hierarchies and uncertainty in zero-shot learning \cite{Liu_2020_CVPR}, re-identification \cite{Khrulkov_2020_CVPR}, and action recognition \cite{9157196}. Of particular interest, \cite{suris2021hyperfuture} has leveraged hyperNNs to model a hierarchy of actions from unlabeled videos. To the best of our knowledge, this is the first work to have applied hyperNNs for sequence modelling with the goal of anomaly detection.

redpajama_set_name: RedPajamaArXiv


is both a scientist and a journalist, helping to drive Circle of Blue's reporting with data and research. She holds an MS in Environmental Engineering from Michigan Technological University, and she brings proficiency in ESRI's ArcGIS mapping software.
Wildfires rage in drought-stricken California and are becoming more prevalent across the country.
More than half of the country's lakes and rivers are not meeting water quality standards.