\section{Introduction}
 Let $f\colon {\mathbb{C}}\to{\mathbb{C}} $ be a transcendental entire function. The set
    \[ I(f)\defeq \{z\in {\mathbb{C}}\colon \lim_{n\to\infty} f^n(z) = \infty\} \]
  is called the \emph{escaping set} of $f$, where
     \[ f^n = \underbrace{f\circ\dots\circ f}_{n\text{ times}}\]
    denotes the $n$-th iterate of $f$. The escaping set  was first studied by Eremenko~\cite{alexescaping} and has been the subject of intensive research in recent years;
     see e.g.~\cite{oberwolfach,rrrs,bergweilermeasure,ripponstallarderemenkopoints} and their references. The topological study of $I(f)$ turns out to be surprisingly
     intricate. For example Eremenko \cite[p.~343]{alexescaping} asked whether, for every
     transcendental entire function, every
     connected component of $I(f)$ is unbounded. This question has become known as \emph{Eremenko's conjecture}, and is one of the most famous open problems
     in transcendental dynamics. Strengthened versions of the conjecture have since been proved false in general; compare~\cite[Theorem~1.1]{rrrs}
     and~\cite[Theorem~1.6]{arclike}.
     On the other hand, Rippon and Stallard~\cite{boundariesescaping}
     have shown that $I(f)\cup\{\infty\}$ is always connected, so any counterexample to Eremenko's original conjecture would have rather
     subtle topological properties.

   The \emph{fast escaping set} is a certain subset of the escaping set, first introduced by Bergweiler and Hinkkanen~\cite{bergweilerhinkkanen}. This set has also received considerable attention recently; in part because it
    appears to be more tractable than the full escaping set
      $I(f)$. In particular, the analogue of Eremenko's conjecture holds for $A(f)$ \cite{ripponstallardfatoueremenko}. One difference between the two
      objects lies in the topological complexity of their definitions. Indeed, $A(f)$ can be written as a countable union of closed sets~\cite[Formula~(1.3)]{ripponstallardfast};
       so it is an $F_{\sigma}$ set. On the other hand, by definition
      \begin{align*} I(f) &= \{z\in{\mathbb{C}}\colon \forall M\geq 0\, \exists N\geq 0 \colon \lvert f^n(z)\rvert \geq M \text{ for $n\geq N$}\} \\
            &=\bigcap_{M= 0}^{\infty} \bigcup_{N=0}^{\infty}  \bigcap_{n=N}^{\infty} f^{-n}({\mathbb{C}}\setminus D(0,M)), \end{align*}
    where $D(0,M)$ denotes the disc of radius $M$ around $0$.
      So $I(f)$ is $F_{\sigma \delta}$: a countable intersection of countable unions of closed sets.
     This raises the following question, posed by Rippon in 2009~\cite[Problem~8, p.~2960]{oberwolfach}.

     \begin{question}\label{qu:Fsigma}
       Is there a transcendental entire function $f$ for which $I(f)$ itself is $F_{\sigma}$?
     \end{question}
         Since any closed subset of ${\mathbb{C}}$ is a countable union of compact sets, it is equivalent to ask whether
         $I(f)$ is ever \emph{$\sigma$-compact} (a countable union of compact sets); compare also \cite{lipham}. It is well-known that
        $I(f)$ cannot be a $G_{\delta}$ (a countable intersection of
        open sets); see Lemma~\ref{lemma:Gdelta} below.

   As far as we are aware, there is no function $f$ for which Question~\ref{qu:Fsigma} has been previously resolved; the goal of this paper is to
     give a negative answer in general. In fact, we prove a stronger result, as follows.
     Let $\operatorname{UO}(f)$ consist of all $z\in{\mathbb{C}}$ whose orbit $\{f^n(z)\colon n\geq 0\}$ is unbounded,
     and let $\operatorname{BU}(f) = \operatorname{UO}(f)\setminus I(f)$ denote the ``bungee set''; see \cite{bungee}. Recall that the \emph{Julia set} $J(f)$ is the set of non-normality
     of the iterates of~$f$.
     \begin{thm}\label{thm:main}
       Let $f$ be a transcendental entire function.
        Then every $\sigma$-compact subset of $\operatorname{UO}(f)$ omits some point of $I(f)\cap J(f)$ and some point of $\operatorname{BU}(f)\cap J(f)$.

      In particular, the sets $I(f)$, $\operatorname{UO}(f)$, $\operatorname{BU}(f)$ and their intersections with $J(f)$ are not $\sigma$-compact.
     \end{thm}

     \subsection*{Acknowledgements} I thank David Lipham and Phil Rippon for interesting discussions. I also thank David Mart\'i-Pete, James Waterman and the referee for
          helpful comments that improved
  		the presentation of the paper.

\section{Proof of the Theorem}
  We require a result on the existence of arbitrarily slowly escaping points,
   which follows from work of Rippon and Stallard~\cite{ripponstallardannular}; see also~\cite[Theorem~1]{ripponstallardslow}.
   \begin{thm}\label{thm:slow}
      Let $f$ be a transcendental entire function, and let $R_0\geq 0$. Then there exists $M_0>R_0$
       with the following property.
        If $(a_m)_{m=0}^{\infty}$ is a sequence with $a_m\to\infty$ and $a_m\geq M_0$ for all $m$,
        then there are $\zeta \in J(f)\cap I(f)$ and $\omega\in J(f)\cap \operatorname{BU}(f)$ with
              \[ R_0 \leq \lvert f^m(\zeta)\rvert \leq a_m\quad\text{and}\quad R_0 \leq \lvert f^m(\omega)\rvert \leq a_m \quad\text{for $m\geq 0$}. \]
   \end{thm}
    \begin{proof}
    The result is an easy consequence of~\cite{ripponstallardannular},
      which studies ``annular itineraries'' of entire functions.
      Fix some $R>0$ such that $M(r)>r$ for $r\geq R$, where $M(r)$
      denotes the maximum modulus of $f$ on the disc $\overline{D(0,r)}$.
      Then the \emph{annular itinerary} of a point $z$ is the sequence
        $(s_m)_{m=0}^{\infty}$ such that
        $s_m=0$ if $\lvert f^m(z)\rvert < R$, and
        \[ M^{s_m-1}(R) \leq \lvert f^m(z) \rvert < M^{s_m}(R) \]
        otherwise. (Here $M^k$ is the $k$-th iterate of $r\mapsto M(r)$.)

      Theorem 1.2 of~\cite{ripponstallardannular} implies that for suitable $R\geq R_0$, for an entry $s_m=n\geq 0$ in an annular itinerary,
       there is a subset $X_n\subset \{0,\dots,n+1\}$ of allowable next entries $s_{m+1}$, and that, for infinitely many $n$, all entries but at most one
       are allowable. More precisely,
           \begin{enumerate}[(a)]
              \item any sequence $(s_m)_{m=0}^{\infty}$ with $s_{m+1}\in X_{s_m}$ for all $m$ is the annular itinerary of some $z\in J(f)$;\label{item:admissible}
              \item $n+1\in X_n$ for all $n$;
              \item there is an increasing sequence $(n_j)_{j=1}^{\infty}$ such that $\# X_{n_j} \geq n+1$.
            \end{enumerate}
       (With the notation of~\cite{ripponstallardannular}, $X_{n_j}=\{0,\dots,n_j+1\}\setminus I_j$, where $\# I_j \leq 1$, and
           $X_n = \{n+1\}$ when $n\neq n_j$ for all $j$.) We may suppose that the sequence $(n_j)$ is chosen such that $n_1 \geq 2$.

     Set $M_0\defeq M^{n_1}(R)$ and
      let $(a_m)_{m=0}^{\infty}$ be a sequence as in the statement of the theorem. We may assume that $(a_m)$ is non-decreasing.
      Similarly as in the proof of \cite[Corollary~1.3~(d)]{ripponstallardannular}, we can construct
         an annular itinerary $(s_m)_{m=0}^{\infty}$ satisfying~\ref{item:admissible} such that $s_m\to\infty$ and
         \begin{equation}\label{eqn:slowgrowth} M^{s_m}(R) \leq a_m \qquad\text{for } m\geq 0. \end{equation}
       Indeed, for each $j$, either $n_j\in X_{n_j}$ or $n_{j}-1\in X_{n_j}$; let us denote this element of $X_{n_j}$ by $\tilde{n}_j$. Then any sequence of the form
        \[ (s_m)_{m=0}^{\infty} = \underbrace{n_1, \tilde{n}_1, n_1, \tilde{n}_1,\dots, n_1}_{\text{length }N_1},n_{1}+1, n_1+2, \dots,
            \underbrace{n_{2}, \tilde{n}_{2}, \dots, n_{2}}_{\text{length }N_2} ,n_{2} + 1, n_{2} + 2, \dots \]
      satisfies~\ref{item:admissible}. If $N_j$ is chosen sufficiently large that
               \begin{equation}\label{eqn:Nj} a_m \geq M^{n_{j+1}}(R) \quad\text{for } m\geq N_j, \end{equation}
        then~\eqref{eqn:slowgrowth} holds for $m\geq N_1$. It also holds for $m\leq N_1$, since $s_m\leq n_1$ for these values of $m$, and
     $a_m \geq M_0 = M^{n_1}(R)$ by assumption.

      Let $\zeta \in J(f)$ have annular itinerary $(s_m)_{m=0}^{\infty}$; then $\zeta\in I(f)$ and
         \[
            R_0\leq R \leq M^{s_m-1}(R) \leq \lvert f^m(\zeta)\rvert  < M^{s_m}(R) \leq a_m \quad \text{for all $m$}. \]
       Hence
       $\zeta$ has the desired properties.
   To obtain $\omega$, we instead use an unbounded but non-escaping sequence of the form
        \[(\tilde{s}_m)_{m=0}^{\infty} = B_1,n_{1}+1, n_1+2, \dots, n_{2},
            \hat{n}_2, B_2 , n_1+1,n_1+2,\dots,n_3,\hat{n}_3, B_3,\dots, \]
     where $\hat{n}_j\in X_{n_j}$ is either $n_1$ or $n_1-1$, and $B_j$ is a block alternating between entries
      $n_1$ and $\tilde{n}_1$. To ensure that the sequence satisfies~\ref{item:admissible}, $B_j$ should end in $n_1$; it should begin
      with $n_1$ if $\hat{n}_j=n_1-1$, and with $\tilde{n}_1$ if $\hat{n}_j = n_1$. The length $N_j$ of the block $B_j$ is again chosen to satisfy~\eqref{eqn:Nj}.
    \end{proof}

\begin{proof}[{Proof of Theorem~\ref{thm:main}}]
 If $D\subset{\mathbb{C}}$ is closed and $z\in{\mathbb{C}}$, we set
    \[ n_D(z) \defeq \min\{n\geq 0\colon f^n(z)\notin D\} \leq \infty, \]
     with the convention that $n_D(z)=\infty$ if no such $n$ exists. Observe that
       $n_D$ depends upper semicontinuously on $z$, as the infimum of
       upper semicontinuous functions. Indeed,
        \[ n_D(z) = \min\{\chi_n(z) \colon n \geq 0\}, \qquad\text{where}\qquad
            \chi_n(z) = \begin{cases}
                             \infty & \text{if } z\in f^{-n}(D) \\
                             n & \text{if } z\notin f^{-n}(D),\end{cases} \]
      and each $\chi_n$ is upper semicontinuous since $f^{-n}(D)$ is closed.
      Note also that
     $z\in \operatorname{UO}(f)$ if and only if $n_D(z)<\infty$ for every compact $D\subset{\mathbb{C}}$.

  Now let $R_0 \geq 0$ be arbitrary, and choose $M_0$ as in Theorem~\ref{thm:slow}.
   Let $X\subset \operatorname{UO}(f)$ be $\sigma$-compact; say
   $X = \bigcup_{j=0}^{\infty} K_j$ where each $K_j$ is compact.
     Define $M_j \defeq M_0 +j$ and $D_j \defeq \overline{D(0,M_j)}$. Then
     $n_{D_j}(z)<\infty$ for every $z\in K_j$.

     Since $K_j$ is compact and $n_{D_j}$ is
     upper semicontinuous,
     \[ n_j \defeq \max_{z\in K_j} n_{D_j}(z) \]
     exists for every $j$. Set $N_0\defeq n_0$ and $N_{j+1}\defeq \max\{n_{j+1},N_j+1\}$.
     For $n\geq 0$, define $j(n)\defeq\min\{j\colon N_j\geq n\}$ and
      \[ a_n \defeq M_{j(n)} = \min\{ M_j \colon  N_j\geq n \}. \]
     Then $a_n\geq M_0$ for all $n$ and $\lim_{n\to\infty} a_n = \infty$. So by Theorem~\ref{thm:slow}, there are $\zeta\in I(f)\cap J(f)$ and $\omega\in \operatorname{BU}(f)\cap J(f)$ such that
       $\lvert f^n(z)\rvert \leq a_n$ for $z\in \{\zeta,\omega\}$ and all $n\geq 0$.

      Let $j\geq 0$. Then, for $n\leq n_j\leq N_j$, we have
        $\lvert f^n(z)\rvert \leq a_n \leq M_j$, and hence $f^n(z) \in D_j$. So $n_{D_j}(z) > n_j$, and $z\notin K_j$ by choice of $n_j$.
       Thus $z\notin X = \bigcup_{j=0}^{\infty} K_j$, as claimed.
\end{proof}

By the expanding property of the Julia set, we also obtain the following answer to a question of Lipham (personal communication).

\begin{cor}\label{cor:nowheresigmacompact}
  Let $f$ be a transcendental entire function and let $Y$ be  one of the sets $I(f)\cap J(f)$, $\operatorname{UO}(f)\cap J(f)$ and $\operatorname{BU}(f)\cap J(f)$.
   Then $Y$ is nowhere $\sigma$-compact. That is, if $X\subset Y$ is $\sigma$-compact, then
    $X$ does not contain a non-empty relatively open subset of $Y$.
\end{cor}
\begin{proof}
   We prove a  slightly stronger statement. Let $Y$ be as in the statement and  let
   $X\subset \operatorname{UO}(f)\cap J(f)$ be $\sigma$-compact. If $U\subset{\mathbb{C}}$ is open with $U\cap Y \neq \emptyset$, then $X$ omits some point of
      $U\cap Y$.

    Let $R_0$ and $M_0$ be as in Theorem~\ref{thm:slow}, and set $L_0\defeq J(f)\cap \overline{D(0,M_0)}\setminus D(0,R_0)$. If $R_0$ is chosen sufficiently large, then
      $L_0$ contains no Fatou exceptional point of $f$. (Recall that a Fatou exceptional point is a point with finite backward orbit,
      and that $f$ has at most one finite exceptional point \cite[p.~156]{waltersurvey}.) Since $U\cap J(f)\neq \emptyset$, it follows that
     there is some $n$ such that $f^n(U)\supset L_0$.

     The proof of Theorem~\ref{thm:main} shows that no $\sigma$-compact subset of $\operatorname{UO}(f)\cap J(f)$ contains $L_0\cap Y$. (Note that we may take $a_0=M_0$ in the proof
      to ensure that $\lvert \zeta \rvert,\lvert \omega\rvert \leq M_0$.) Moreover, $f^n(X)$ is $\sigma$-compact as the
	image of a $\sigma$-compact set under a continuous function. So $f^n(X)$ omits some points of $L_0\cap Y$. Since $Y$ is backward-invariant,
	$X$ omits some points of $U\cap Y$, as claimed.
\end{proof}

We remark that the proof of Theorem~\ref{thm:main} establishes the following very general principle:
\begin{prop}[Abstract version of the main theorem]\label{prop:abstract}
  Let $f\colon U\to V$ be continuous, where $U,V$are non-empty topological spaces and $U$ is
  $\sigma$-compact. Let $\operatorname{UO}(f)$ denote the set of $z\in U$ such that $f^n(z)$ is defined and in $U$ for all $n\geq 0$, but
   the orbit $\{f^n(z)\}$ is not contained in any compact subset of $U$.

  Suppose that $X\subset \operatorname{UO}(f)$ is $\sigma$-compact and $\Delta\subset U$ is compact. Then there is a sequence
  $(\Delta_n)_{n=0}^{\infty}$ of compact subsets $\Delta_n\subset U$ with $\Delta\subset \Delta_0\subset \Delta_1\subset \dots$ and
    $\bigcup_{n=0}^{\infty} \Delta_n = U$ such that $X$ contains no point $\zeta\in \operatorname{UO}(f)$ with
  $f^n(\zeta)\in \Delta_n$ for all $n\geq 0$.
  \end{prop}
  \begin{remark}
    Note that we are not assuming any relation between the spaces $U$ and $V$. However, the statement is vacuous when $\operatorname{UO}(f)=\emptyset$, and in
      particular when $U\cap V\neq \emptyset$.
  \end{remark}
  \begin{sketch}
     Let $(D_j)_{j=0}^{\infty}$ be an increasing sequence of compact subsets $D_j\subset U$ with $\bigcup _{j=0}^{\infty} D_j= U$.
      Define $j(n)$ as in the proof of Theorem~\ref{thm:main}; then the sets $\Delta_n \defeq D_{j(n)}$ have the desired property.
  \end{sketch}

  It follows that the set of escaping points is not $\sigma$-compact in any
  setting where an analogue of  Theorem~\ref{thm:slow} holds.
  This includes:
  \begin{enumerate}[(a)]
    \item transcendental meromorphic functions ${\mathbb{C}}\to\hat{\C}$ \cite{ripponstallardslow};\label{item:mero}
    \item transcendental self-maps of the punctured plane \cite[Theorem~1.2]{martipeteCstar};\label{item:Cstar}
    \item quasiregular self-maps $f\colon {\mathbb{R}}^d\to{\mathbb{R}}^d$ of transcendental type
        \cite{nicksslow};
    \item continuous functions $\phi\colon [0,\infty)\to [0,\infty)$ with $\phi(t)\not\to\infty$ as $t\to\infty$,
       and such that $I(\phi)\neq \emptyset$ \cite[Theorem~2.2]{osborneripponstallard}.
  \end{enumerate}

   A more general setting than both~\ref{item:mero} and \ref{item:Cstar} is provided by
     the \emph{Ahlfors islands maps} of Epstein; see e.g.~\cite{exotic}.
     These are maps $f\colon W\to X$, where $X$ is a compact one-dimensional
     manifold, $W\subset X$ is open, and $f$ satisfies certain transcendence
     conditions near $\partial W$. We may define the escaping set $I(f)$
     as the set of points $z\in W$ with $f^n(z)\in W$ for all $n$ and
     $\operatorname{dist}(f^n(z),\partial W)\to 0$ as $n\to\infty$. It is plausible that an analogue of Theorem~\ref{thm:slow} holds for
     Ahlfors islands maps with $W\neq X$, using a similar proof
    as in~\cite{ripponstallardslow}. This would mean, by Proposition~\ref{prop:abstract}, that $I(f)$ is not
    $\sigma$-compact for such functions.

 For completeness, we conclude by giving the simple proof that $I(f)$ is never a $G_{\delta}$ set; compare also \cite[Corollary~3.2]{liphametds}.
\begin{lem}\label{lemma:Gdelta}
 Let $f$ be a transcendental entire function. Then $I(f)$ and $I(f)\cap J(f)$ are not $G_{\delta}$ sets.
\end{lem}
\begin{proof}
  Every closed subset of ${\mathbb{C}}$ (or any metric space) is $G_{\delta}$; so $J(f)$ is $G_{\delta}$.
   The intersection of two $G_{\delta}$ sets is again $G_{\delta}$, so it is enough to prove the claim for $I(f)\cap J(f)$.
 By \cite[Theorem~2]{alexescaping}, $I(f)\cap J(f)$ is nonempty, and hence dense in $J(f)$ by Montel's theorem.
  By Baire's theorem, any two dense $G_{\delta}$ subsets of $J(f)$ must intersect. Hence it is enough
    to observe that $\operatorname{BU}(f)\cap J(f)$ contains a dense $G_{\delta}$ by Montel's theorem, namely the set of points whose orbits are dense in $J(f)$.
      (See \cite[Lemma~1]{bakerdominguezresidual}.)
\end{proof}

 Lipham has pointed out the following reformulation of Corollary~\ref{cor:nowheresigmacompact}.
 \begin{cor}
    Any $G_{\delta}$ set
    $A\subset Y\defeq J(f)\setminus I(f)$ is nowhere dense in $Y$.

    In particular, $Y$ is $G_{\delta \sigma}$ but not \emph{strongly $\sigma$-complete}; that is, it cannot be written as a countable
    union of relatively closed $G_{\delta}$ subsets.
  \end{cor}
  \begin{proof}
    Suppose, by contradiction, that $\overline{D(\zeta,\varepsilon)}\cap Y \subset A$ for some $\zeta\in J(f)$ and $\varepsilon>0$. The complement
      of a $G_{\delta}$ set is $F_{\sigma}$, so
    $B \defeq J(f)\cap \overline{D(\zeta,\varepsilon)} \setminus A$ is an $F_{\sigma}$ set
     with \[ D(\zeta,\varepsilon)\cap I(f)\cap J(f)\subset B \subset I(f)\cap J(f),\] which contradicts Corollary~\ref{cor:nowheresigmacompact}.

   The set $Y$ is $G_{\delta \sigma}$ by definition.
     As mentioned in the proof of Lemma~\ref{lemma:Gdelta}, $Y$ contains a $G_{\delta}$ subset $U$ that is dense in $J(f)$.
     On the other hand, if $(A_k)_{k=0}^{\infty}$ is a sequence of relatively closed $G_{\delta}$ subsets of $Y$, then
     $\overline{A_k}$ is closed and nowhere dense in $J(f)$. Hence
        \[  \bigcup_{k=0}^{\infty} A_k \subset \bigcup_{k=0}^{\infty} \overline{A_k} \not\supset U \subset Y, \]
      by Baire's theorem, and $Y$ is indeed strongly $\sigma$-complete.
  \end{proof}

\providecommand{\href}[2]{#2}


\section*{Acknowledgements}

This work was supported by NSF Award CPS-1953740 and a Siebel Energy Institute
grant.

\section{Information on Datasets}

\begin{table}[H]
\centering
\caption{Data representation dimensions for each dataset.}
\label{tab:feature_dimensions}
\begin{tabular}{|c|c|c|c|c|}
\toprule
Device & \Cell{IAT,\$IAT$-FFT} & \Cell{SIZE,\$SIZE$-FFT} & \Cell{STATS} & \Cell{SAMP-NUM,\\SAMP-NUM-FFT,\\SAMP-SIZE} \\
\midrule
\multirow{5}{*}\ PC1 & 14&15&10&14 \\
\ PC2 & 16&17&10&16 \\
\ PC3 & 15&16&10&15 \\
\ PC4 & 17&18&10&17 \\
\ PC5 & 17&18&10&17 \\
\midrule
\ 2Rsps & 8&9&10&8\\
\midrule
\ 2PCs & 25&26&10&25\\
\midrule
\multirow{5}{*}\ TV\&PC & 12&13&10&12 \\
\ GHom & 19&20&10&19 \\
\ SCam & 6&7&10&6 \\
\ SFrig & 30&31&10&30 \\
\ BSTch & 148&149&10&148  \\
\bottomrule
\end{tabular}
\end{table}

\begin{table}[H]
\centering
\caption{Train and Test sample set sizes.}
\label{tab:train_test_sets}
\begin{tabular}{|c|c|l|l|}
\toprule
Reference & Devices & Train set size & Test set sizes  \\
\midrule
\multirow{5}{*}{UNB IDS } & PC1 & Normal: 10000 & Normal: 179, Attack: 179 \\
 & PC2 & Normal: 10000 & Normal: 257, Attack: 257 \\
 & PC3 & Normal: 10000 & Normal: 221, Attack: 221 \\
 & PC4 & Normal: 10000 & Normal: 209, Attack: 209 \\
 & PC5 & Normal: 10000 & Normal: 360, Attack: 360 \\
\midrule
CTU IoT  & 2Rsps & Normal: 3991 & Normal: 400, Novel device: 400 \\
\midrule
MAWI  & 2PCs & Normal: 824 & Normal: 400, Novel device: 400 \\
\midrule
\multirow{5}{*}{Private IoT } & TV\&PC & Normal: 4636 & Normal: 346, Novel device: 346 \\
 & GHom & Normal: 7376 & Normal: 513, Novel activity: 513 \\
 & SCam & Normal: 10000 & Normal: 197, Novel activity: 197 \\
 & SFrig & Normal: 2952 & Normal: 296, Novel activity: 296 \\
 & BSTch & Normal: 997 & Normal: 236, Novel activity: 236  \\
\bottomrule
\end{tabular}
\end{table}

\newpage
\section{Evaluation with Best Parameters}

\begin{longtable}{|c|c|g|c|g|c|g|c|}
\caption{AUCs for 4 ML approaches on basic feature representations with best parameters.}
\label{tab:OCSVM_IF_AE_KDE_with_best_parameters}
\endfirsthead
\endhead
\toprule
Detector & Dataset & \Cell{STATS} & \Cell{SIZE} & \Cell{IAT} &
    \Cell{IAT+\$SIZE$} & \Cell{SAMP-\\NUM} & \Cell{SAMP-\$SIZE$}  \\
\midrule
\multirow{5}{*}{OCSVM}
&UNB(PC2) & 0.64 & 0.75 & 0.82 & 0.81 & 0.82 & 0.85  \\
&UNB(PC3) & 0.59 & 0.85 & 0.87 & 0.88 & 0.83 & 0.85 \\
&UNB(PC5) & 0.47 & 0.73 & 0.84 & 0.86 & 0.76 & 0.81 \\
\cmidrule{2-8}
&GHom & 0.76 & 0.97 & 0.70 & 0.70 & 0.95 & 0.96\\
&SCam & 0.58 & 0.61 & 0.56 & 0.62 & 0.56 & 0.61\\
\midrule
\multirow{5}{*}{IF}
&UNB(PC2) & 0.68 & 0.61 & 0.68 & 0.75 & 0.81 & 0.80  \\
&UNB(PC3) & 0.62 & 0.63 & 0.70 & 0.79 & 0.81 & 0.80 \\
&UNB(PC5) & 0.55 & 0.61 & 0.69 & 0.74 & 0.79 & 0.74 \\
\cmidrule{2-8}
&GHom & 0.88 & 0.77 & 0.44 & 0.54 & 0.96 & 0.96\\
&SCam & 0.52 & 0.51 & 0.53 & 0.54 & 0.60 & 0.56\\
\midrule
\multirow{5}{*}{AE}
&UNB(PC2) & 0.78 & 0.66 & 0.71 & 0.83 & 0.83 & 0.77  \\
&UNB(PC3) & 0.76 & 0.58 & 0.71 & 0.90 & 0.85 & 0.80 \\
&UNB(PC5) & 0.82 & 0.66 & 0.73 & 0.87 & 0.88 & 0.87 \\
\cmidrule{2-8}
&GHom & 0.84 & 0.69 & 0.35 & 0.86 & 0.97 & 0.96\\
&SCam & 0.63 & 0.61 & 0.57 & 0.65 & 0.61 & 0.60\\
\midrule
\multirow{5}{*}{KDE}
&UNB(PC2) & 0.61 & 0.73 & 0.82 & 0.80 & 0.81 & 0.80  \\
&UNB(PC3) & 0.53 & 0.84 & 0.84 & 0.87 & 0.82 & 0.85 \\
&UNB(PC5) & 0.53 & 0.69 & 0.84 & 0.86 & 0.75 & 0.82 \\
\cmidrule{2-8}
&GHom & 0.60 & 0.97 & 0.71 & 0.72 & 0.96 & 0.96\\
&SCam & 0.52 & 0.50 & 0.56 & 0.63 & 0.54 & 0.60\\
\bottomrule
\end{longtable}

\newpage
\begin{longtable}{|c|c|g|c|g|c|g|c|}
\caption{AUCs for GMM and PCA on basic feature representations with best parameters.}
\label{tab:GMM_and_PCA_basic_representation_with_best_parameters}
\endfirsthead
\endhead
\toprule
 Detector & Dataset & \Cell{STATS} & \Cell{SIZE} & \Cell{IAT} &
    \Cell{IAT+\$SIZE$} & \Cell{SAMP-\\NUM} & \Cell{SAMP-\$SIZE$}  \\
\midrule
\multirow{12}{*}{GMM} &UNB(PC1) & 0.55 & 0.59 & 0.62 & 0.75 & 0.84 & 0.74  \\
&UNB(PC2) & 0.84 & 0.72 & 0.71 & 0.74 & 0.83 & 0.84  \\
&UNB(PC3) & 0.87 & 0.78 & 0.75 & 0.91 & 0.83 & 0.81 \\
&UNB(PC4) & 0.68 & 0.59 & 0.61 & 0.65 & 0.89 & 0.86 \\
&UNB(PC5) & 0.60 & 0.64 & 0.72 & 0.79 & 0.85 & 0.76 \\
\cmidrule{2-8}
&CTU & 0.98 & 0.96 & 0.90 & 0.93 & 0.91 & 0.88\\
\cmidrule{2-8}
&MAWI & 0.80 & 0.63 & 0.53 & 0.59 & 0.62 & 0.69\\
\cmidrule{2-8}
&TV\&PC & 1.00 & 1.00 & 0.97 & 0.99 & 0.99 & 0.99\\
\cmidrule{2-8}
&GHom & 0.96 & 0.93 & 0.90 & 0.93 & 0.97 & 0.96\\
&SCam & 0.62 & 0.61 & 0.59 & 0.63 & 0.63 & 0.65\\
&SFrig & 0.97 & 0.97 & 0.95 & 0.97 & 0.96 & 0.95 \\
&BSTch & 0.99 & 0.99 & 0.98 & 0.99 & 0.98 & 0.97  \\
\midrule
\multirow{12}{*}{PCA} &UNB(PC1) & 0.74 & 0.41 & 0.35 & 0.93 & 0.86 & 0.85  \\
&UNB(PC2) & 0.86 & 0.67 & 0.71 & 0.82 & 0.85 & 0.81  \\
&UNB(PC3) & 0.75 & 0.56 & 0.70 & 0.88 & 0.83 & 0.82 \\
&UNB(PC4) & 0.72 & 0.52 & 0.65 & 0.82 & 0.89 & 0.88 \\
&UNB(PC5) & 0.79 & 0.64 & 0.73 & 0.87 & 0.91 & 0.89 \\
\cmidrule{2-8}
&CTU & 0.95 & 0.95 & 0.84 & 0.97 & 0.84 & 0.87\\
\cmidrule{2-8}
&MAWI & 0.66 & 0.71 & 0.49 & 0.74 & 0.59 & 0.54\\
\cmidrule{2-8}
&TV\&PC & 1.00 & 1.00 & 0.96 & 1.00 & 0.94 & 0.94\\
\cmidrule{2-8}
&GHom & 0.85 & 0.71 & 0.34 & 0.97 & 0.97 & 0.96\\
&SCam & 0.63 & 0.61 & 0.57 & 0.65 & 0.62 & 0.59\\
&SFrig & 0.97 & 0.95 & 0.79 & 0.97 & 0.96 & 0.96 \\
&BSTch & 0.96 & 0.98 & 0.99 & 0.99 & 0.97 & 0.97  \\
\bottomrule
\end{longtable}

\newpage
\subsection{Effect of Fourier Domain Representation}

The figures below show the difference in AUC for the FFT vs. raw timeseries
representations with the best model parameters for each model. FFT
transformations have little effect on model accuracy.

\begin{figure}[H]
\centering
\includegraphics[width=0.99\textwidth ]{OCSVM_IF_AE_KDE_with_best_parameters_on_basic_representations_rest_diff.pdf}
\caption{Difference in AUC for {\bf FFT vs. raw time series} representations with best parameters.}
\label{fig:OCSVM_IF_AE_KDE_with_best_parameters_on_basic_representations_diff}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=0.99\textwidth]{GMM_PCA_with_best_parameters_on_basic_representations_diff.pdf}
\caption{Difference in AUC for {\bf FFT vs. raw time series} representations generated by GMM and PCA with best parameters.}
\label{fig:GMM_PCA_with_best_parameters_on_basic_representations_diff}
\end{figure}

\subsection{Effect of Packet Size Information}

The figures below show the difference in AUC for including vs. excluding
packet size information for different models: IAT and SAMP. Including packet
size can significantly improve model accuracy in certain cases.

\begin{figure}[H]
\centering
\includegraphics[width=0.99\textwidth ]{Effect_of_Size_OCSVM_IF_AE_KDE_with_best_parameters_rest_diff.pdf}
\caption{Differences in AUC due to {\bf including vs. excluding packet size} information with best parameters.}
\label{fig:Effect_of_Size_OCSVM_IF_AE_KDE_with_best_parameters_diff}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=0.99\textwidth]{Effect_of_Size_GMM_PCA_with_best_parameters_diff.pdf}
\caption{Differences in AUC due to {\bf including vs. excluding packet size} information generated by GMM and PCA with best parameters.}
\label{fig:Effect_of_Size_GMM_PCA_with_best_parameters_diff}
\end{figure}

\subsection{Effect of Packet Header}
The figures below show the difference in AUC for including vs. excluding
packet header information (i.e., IP TTL, TCP flags) for different models:
STATS, IAT, and SAMP. Including packet
header information almost always improves model accuracy for these models.

\begin{figure}[H]
\centering
\includegraphics[width=0.99\textwidth ]{Effect_of_Packet_Header_OCSVM_IF_AE_KDE_with_best_parameters_rest_diff.pdf}
\caption{Differences in AUC due to {\bf including vs. excluding packet header} information with best parameters.}
\label{fig:Effect_of_Packet_Header_OCSVM_IF_AE_KDE_with_best_parameters_diff}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=0.99\textwidth]{Effect_of_Packet_Header_GMM_PCA_with_best_parameters_diff.pdf}
\caption{Differences in AUC due to {\bf including vs. excluding packet header} information generated by GMM and PCA with best parameters.}
\label{fig:Effect_of_Packet_Header_GMM_PCA_with_best_parameters_diff}
\end{figure}

\newpage
\section{Evaluation with Default Parameters}

\begin{scriptsize}
\begin{longtable}{|c|c|g|c|g|c|g|c|}
\caption{AUCs for 4 ML approaches on basic feature representations with default parameters.}
\label{tab:OCSVM_IF_AE_and_KDE_with_default_parameters} \\
\toprule
 Detector & Dataset & \Cell{STATS} & \Cell{SIZE} & \Cell{IAT} & \Cell{IAT+S\\IZE} & \Cell{SAMP-\\NUM} & \Cell{SAMP-\$SIZE$}  \\
\midrule
\multirow{12}{*}{OCSVM} &UNB(PC1) & 0.56 & 0.20 & 0.40 & 0.89 & 0.66 & 0.74  \\
&UNB(PC2) & 0.43 & 0.66 & 0.82 & 0.74 & 0.79 & 0.85  \\
&UNB(PC3) & 0.30 & 0.83 & 0.82 & 0.85 & 0.82 & 0.85 \\
&UNB(PC4) & 0.42 & 0.58 & 0.82 & 0.76 & 0.67 & 0.78 \\
&UNB(PC5) & 0.41 & 0.64 & 0.84 & 0.82 & 0.75 & 0.74 \\
\cmidrule{2-8}
&CTU & 0.96 & 0.97 & 0.95 & 0.96 & 0.88 & 0.88\\
\cmidrule{2-8}
&MAWI & 0.43 & 0.38 & 0.56 & 0.38 & 0.58 & 0.59\\
\cmidrule{2-8}
&TV\&PC & 1.00 & 1.00 & 0.94 & 0.98 & 0.70 & 0.86\\
\cmidrule{2-8}
&GHom & 0.73 & 0.97 & 0.10 & 0.11 & 0.95 & 0.94\\
&SCam & 0.58 & 0.51 & 0.54 & 0.61 & 0.59 & 0.58\\
&SFrig & 0.91 & 0.90 & 0.77 & 0.84 & 0.94 & 0.95 \\
&BSTch & 0.96 & 0.98 & 0.94 & 0.98 & 0.92 & 0.92  \\
\midrule
\multirow{12}{*}{IF} &UNB(PC1) & 0.47 & 0.45 & 0.48 & 0.69 & 0.65 & 0.65  \\
&UNB(PC2) & 0.65 & 0.61 & 0.68 & 0.74 & 0.80 & 0.75  \\
&UNB(PC3) & 0.61 & 0.62 & 0.70 & 0.73 & 0.80 & 0.72 \\
&UNB(PC4) & 0.45 & 0.46 & 0.61 & 0.61 & 0.81 & 0.70 \\
&UNB(PC5) & 0.47 & 0.59 & 0.69 & 0.70 & 0.78 & 0.72 \\
\cmidrule{2-8}
&CTU & 0.94 & 0.92 & 0.78 & 0.88 & 0.83 & 0.84\\
\cmidrule{2-8}
&MAWI & 0.86 & 0.68 & 0.53 & 0.66 & 0.56 & 0.55\\
\cmidrule{2-8}
&TV\&PC & 0.95 & 0.98 & 0.89 & 0.99 & 0.73 & 0.78\\
\cmidrule{2-8}
&GHom & 0.86 & 0.68 & 0.44 & 0.47 & 0.96 & 0.96\\
&SCam & 0.51 & 0.51 & 0.52 & 0.54 & 0.56 & 0.56\\
&SFrig & 0.96 & 0.95 & 0.54 & 0.94 & 0.93 & 0.93 \\
&BSTch & 0.94 & 0.98 & 0.94 & 0.98 & 0.96 & 0.96  \\
\midrule
\multirow{12}{*}{AE} &UNB(PC1) & 0.66 & 0.39 & 0.33 & 0.75 & 0.78 & 0.75  \\
&UNB(PC2) & 0.74 & 0.54 & 0.56 & 0.81 & 0.78 & 0.76  \\
&UNB(PC3) & 0.65 & 0.56 & 0.71 & 0.81 & 0.83 & 0.78 \\
&UNB(PC4) & 0.60 & 0.39 & 0.63 & 0.74 & 0.85 & 0.86 \\
&UNB(PC5) & 0.79 & 0.25 & 0.37 & 0.79 & 0.84 & 0.85 \\
\cmidrule{2-8}
&CTU & 0.91 & 0.95 & 0.80 & 0.96 & 0.80 & 0.76\\
\cmidrule{2-8}
&MAWI & 0.45 & 0.50 & 0.48 & 0.60 & 0.57 & 0.54\\
\cmidrule{2-8}
&TV\&PC & 1.00 & 1.00 & 0.96 & 0.99 & 0.77 & 0.88\\
\cmidrule{2-8}
&GHom & 0.77 & 0.62 & 0.33 & 0.70 & 0.96 & 0.95\\
&SCam & 0.61 & 0.61 & 0.56 & 0.61 & 0.59 & 0.55\\
&SFrig & 0.92 & 0.78 & 0.71 & 0.91 & 0.96 & 0.96 \\
&BSTch & 0.95 & 0.87 & 0.97 & 0.98 & 0.96 & 0.95  \\
\midrule
\multirow{12}{*}{KDE} &UNB(PC1) & 0.28 & 0.37 & 0.51 & 0.92 & 0.74 & 0.75  \\
&UNB(PC2) & 0.59 & 0.63 & 0.82 & 0.74 & 0.77 & 0.80  \\
&UNB(PC3) & 0.52 & 0.78 & 0.82 & 0.83 & 0.82 & 0.84 \\
&UNB(PC4) & 0.37 & 0.51 & 0.71 & 0.77 & 0.81 & 0.84 \\
&UNB(PC5) & 0.33 & 0.62 & 0.84 & 0.84 & 0.74 & 0.81 \\
\cmidrule{2-8}
&CTU & 0.96 & 0.94 & 0.93 & 0.97 & 0.82 & 0.84\\
\cmidrule{2-8}
&MAWI & 0.66 & 0.38 & 0.56 & 0.41 & 0.57 & 0.56\\
\cmidrule{2-8}
&TV\&PC & 1.00 & 1.00 & 0.95 & 0.98 & 0.66 & 0.83\\
\cmidrule{2-8}
&GHom & 0.55 & 0.94 & 0.05 & 0.46 & 0.96 & 0.95\\
&SCam & 0.49 & 0.51 & 0.55 & 0.60 & 0.56 & 0.57\\
&SFrig & 0.94 & 0.91 & 0.73 & 0.77 & 0.94 & 0.94 \\
&BSTch & 0.96 & 0.96 & 0.95 & 0.98 & 0.92 & 0.91  \\
\bottomrule
\end{longtable}
\end{scriptsize}

\begin{longtable}{|c|c|g|c|g|c|g|c|}
\caption{AUCs for 2 additional ML approaches (GMM and PCA) on basic feature representations with default parameters.}
\label{tab:GMM_and_PCA_basic_representation_with_default_parameters}
\endfirsthead
\endhead
\toprule
 Detector & Dataset & \Cell{STATS} & \Cell{SIZE} & \Cell{IAT} & \Cell{IAT+S\\IZE} & \Cell{SAMP-\\NUM} & \Cell{SAMP-\$SIZE$}  \\
\midrule
\multirow{12}{*}{GMM} &UNB(PC1) & 0.37 & 0.23 & 0.16 & 0.81 & 0.76 & 0.73  \\
&UNB(PC2) & 0.56 & 0.52 & 0.65 & 0.71 & 0.85 & 0.54  \\
&UNB(PC3) & 0.52 & 0.61 & 0.75 & 0.90 & 0.81 & 0.56 \\
&UNB(PC4) & 0.40 & 0.47 & 0.60 & 0.57 & 0.81 & 0.82 \\
&UNB(PC5) & 0.36 & 0.43 & 0.70 & 0.71 & 0.71 & 0.73 \\
\cmidrule{2-8}
&CTU & 0.95 & 0.92 & 0.89 & 0.84 & 0.88 & 0.84\\
\cmidrule{2-8}
&MAWI & 0.65 & 0.37 & 0.52 & 0.40 & 0.60 & 0.58\\
\cmidrule{2-8}
&TV\&PC & 1.00 & 0.88 & 0.86 & 0.57 & 0.60 & 0.80\\
\cmidrule{2-8}
&GHom & 0.33 & 0.30 & 0.06 & 0.39 & 0.96 & 0.96\\
&SCam & 0.47 & 0.46 & 0.52 & 0.60 & 0.54 & 0.59\\
&SFrig & 0.73 & 0.91 & 0.82 & 0.93 & 0.96 & 0.95 \\
&BSTch & 0.96 & 0.95 & 0.93 & 0.99 & 0.93 & 0.90  \\
\midrule
\multirow{12}{*}{PCA} &UNB(PC1) & 0.66 & 0.29 & 0.27 & 0.65 & 0.72 & 0.85  \\
&UNB(PC2) & 0.59 & 0.67 & 0.43 & 0.32 & 0.84 & 0.77  \\
&UNB(PC3) & 0.71 & 0.48 & 0.70 & 0.48 & 0.73 & 0.68 \\
&UNB(PC4) & 0.65 & 0.22 & 0.64 & 0.63 & 0.89 & 0.69 \\
&UNB(PC5) & 0.73 & 0.29 & 0.44 & 0.78 & 0.66 & 0.67 \\
\cmidrule{2-8}
&CTU & 0.95 & 0.94 & 0.80 & 0.97 & 0.77 & 0.79\\
\cmidrule{2-8}
&MAWI & 0.65 & 0.71 & 0.46 & 0.54 & 0.59 & 0.51\\
\cmidrule{2-8}
&TV\&PC & 0.98 & 0.97 & 0.40 & 0.99 & 0.94 & 0.94\\
\cmidrule{2-8}
&GHom & 0.20 & 0.71 & 0.30 & 0.97 & 0.96 & 0.95\\
&SCam & 0.55 & 0.61 & 0.49 & 0.64 & 0.62 & 0.58\\
&SFrig & 0.96 & 0.90 & 0.76 & 0.97 & 0.96 & 0.95 \\
&BSTch & 0.95 & 0.98 & 0.97 & 0.98 & 0.97 & 0.96  \\
\bottomrule
\end{longtable}

\newpage
\subsection{Effect of Fourier Domain Representation}

The figures below show differences in model accuracy for Fourier domain
representation vs. raw timeseries representations for different models and
datasets. Fourier domain representations in general do not improve model
accuracy.

\begin{figure}[H]
\centering
\includegraphics[width=0.99\textwidth]{OCSVM_IF_AE_KDE_with_default_parameters_on_basic_representations_diff.pdf}
\caption{Difference in AUC for {\bf FFT vs. raw time series} representations with default parameters.}
\label{fig:OCSVM_IF_AE_KDE_with_default_parameters_on_basic_representations_diff}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=0.99\textwidth]{GMM_PCA_with_default_parameters_on_basic_representations_diff.pdf}
\caption{Difference in AUC for {\bf FFT vs. raw time series} representations generated by GMM and PCA with default parameters.}
\label{fig:GMM_PCA_with_default_parameters_on_basic_representations_diff}
\end{figure}

\newpage
\subsection{Effect of Packet Size Information}

The figures below show differences in model accuracy
as a result of including packet size information for different models and
datasets.  Packet size information generally improves model accuracy.

\begin{figure}[H]
\centering
\includegraphics[width=0.99\textwidth ]{Effect_of_Size_OCSVM_IF_AE_KDE_with_default_parameters_diff.pdf}
\caption{Differences in AUC due to {\bf including vs. excluding packet size} information with default parameters.}
\label{fig:Effect_of_Size_OCSVM_IF_AE_KDE_with_default_parameters_diff}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=0.99\textwidth]{Effect_of_Size_GMM_PCA_with_default_parameters_diff.pdf}
\caption{Differences in AUC due to {\bf including vs. excluding packet size} information generated by GMM and PCA with default parameters.}
\label{fig:Effect_of_Size_GMM_PCA_with_default_parameters_diff}
\end{figure}

\newpage
\subsection{Effect of Packet Header}

The figures below show differences in model accuracy
as a result of including packet header information for different models and
datasets.  Packet header information generally improves model accuracy.

\begin{figure}[H]
\centering
\includegraphics[width=0.99\textwidth ]{Effect_of_Packet_Header_OCSVM_IF_AE_KDE_with_default_parameters_diff.pdf}
\caption{Differences in AUC due to {\bf including vs. excluding packet header} information with default parameters.}
\label{fig:Effect_of_Packet_Header_OCSVM_IF_AE_KDE_with_default_parameters_diff}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=0.99\textwidth]{Effect_of_Packet_Header_GMM_PCA_with_default_parameters_diff.pdf}
\caption{Differences in AUC due to {\bf including vs. excluding packet header} information generated by GMM and PCA with default parameters.}
\label{fig:Effect_of_Packet_Header_GMM_PCA_with_default_parameters_diff}
\end{figure}

\section{Conclusion}\label{sec:conclusion}

Novelty detection problems in computer networking require making important
decisions about how to represent traffic data. Traffic data are captured as
``raw'' packet captures, where each packet has an individual timestamp, thus
taking the format of an irregularly sampled timeseries. The data itself must
be subsequently transformed into a representation for model input; often, the
representation of the input data can be as important for model accuracy as the
choice of the model or model parameters.

This paper takes a first step towards exploring the effects of network traffic
representation for novelty detection, focusing on a range of novel behaviors
in the Internet of Things (IoT)---security events, novel devices, and novel
human behaviors and interactions. We use a collection of datasets to evaluate
different traffic representations across a range of models.  Our work reveals
several important findings: (1)~popular Fourier domain representations do not
significantly improve accuracy over simple time-domain representations;
(2)~representations that include traffic sizes and packet header fields (e.g.,
IP TTL, TCP flags) can significantly improve model accuracy as measured by
area under the curve (AUC).

This study has important implications for the design of future measurement and
analysis systems that use network traffic data for novelty detection, in the
IoT and more generally for deployed network systems. From a systems
perspective, capturing full packet traces can introduce significant costs, yet
aggressive aggregation can preclude certain data representations that are
important for model accuracy.  In particular, we find that raw packet traces
can be summarized in terms of flow-level statistics in short time windows or
inter-arrival times, and augmented with sequences of packet sizes and flags;
augmentation of simple timeseries representations with this metadata
significantly improves accuracy.

\section{Introduction}

An increasing number of interconnected ``Internet-of-Things'' (IoT) are being
deployed to monitor and manage homes, buildings, infrastructure, and even
outdoor spaces. An intention of many such deployments is to monitor the
environment for new, unusual, or anomalous activities---in other words, patterns
that have previously not been observed. Such new or unusual activities can be
indicative of many types of noteworthy events, including network or physical
security incidents, device or infrastructure malfunctions, even behavioral
changes in the users of the devices that might be indicative of abnormal human
conditions or behaviors. The general goal of detecting such new, unseen events
falls under the general category of {\em novelty detection} problems in
machine learning, for which there are a wide variety of possible procedures,
including Gaussian mixture models (GMM), one-class support vector machines
(OCSVM), kernel density estimation (KDE), and others. The success of any of
these procedures---and, in fact, for {\em any} machine learning
approach---depends fundamentally on how the data are represented as input
features to the model. Often the choice of features and how they are
represented are more fundamental to the effectiveness of the approach than the
model itself.

IoT devices provide useful ``digital exhaust'' that can be used to detect a
broad class of new, unseen events and anomalies. Specifically,
Internet-connected devices must necessarily generate network traffic as they
communicate with other devices on the network (e.g., other devices on the home
network, cloud-hosted services that control these devices). Unusual activities
or incidents may be reflected in changes to the underlying traffic itself. As
a simple example, an Internet-connected light switch generates associated
traffic when it is turned on and off, effectively generating a timeseries of
network traffic over time. The data itself is timestamped, and
multi-dimensional---for example, the device may talk to more than one
destination on the network; the timestamped nature of the dataset makes
the data itself irregularly sampled in the time domain. Intuitively, one could
assume that the things that a human (or computer program) does to interact
with such a device might follow some normal patterns, and that novelty
detection might be interested in identifying deviations in these patterns,
where those deviations might reflect anything from a malware infection to
someone entering the room at an unusual or unexpected time. As
Internet-connected devices become increasingly ubiquitous, the digital exhaust
that these devices produce provides a wealth of signal for detecting a wide
variety of anomalies for just about any activity in our daily lives.
Approaches to capture novelty in the traffic that these devices
generate are less well explored.

Unfortunately, representations of network traffic that are most effective for
novelty detection remain a priori unclear.
By default, representation choices involve various statistics on traffic
volumes, decisions on packet sampling rates, and choices of observation
intervals; however, which of these choices retain predictive information
remains a priori unclear, especially given the large variety of IoT devices
with vastly different traffic patterns. For instance, if we decide to record
and build on \emph{number of packets per second}, then changes in traffic
patterns at the micro-second level might go undetected; on the other hand,
recording at the microsecond level would result in higher dimensional
observations that machine learning procedures might fail to scale up to, due
to well-known challenges with high dimensional data\footnote{The so-called
\emph{curse of dimensionality} in machine learning is the observed fact that the
higher the dimension of the input, the harder it is to achieve good accuracy;
in particular, we might then require an amount of training data exponential as
a function of dimension.}.

Our goal is to explore how different data representations for
network traffic affect novelty detection rates, across a range of IoT devices,
types of novelty, and unsupervised machine learning models.  Given raw traffic
inputs, we focus on the problem of identifying novel network
\emph{flows}---sequences of packets from a fixed source to a fixed
destination, a common, intuitive unit of identification in network traffic
that, at a high-level, represents a single exchange of data between two
network endpoints. From a practical perspective, the ability to identify a
novel flow of course lends itself to detecting other types of novelty;
for example, a device that generates novel traffic flows may itself be
considered novel. There are many possible ways to represent raw traffic
traces (i.e., packets) as inputs. The data itself can be aggregated on various
time windows, elided, transformed into different bases, sampled, and so forth.

\paragraph{Results and contributions.}
We compare flow-based data
representations as combined with various machine learning procedures for
novelty detection. To compare the performance of different machine learning
procedures, we report the area under the ROC
curve (AUC) for each approach and each dataset.
Because machine learning procedures are very sensitive to hyperparameter choices
(e.g., bandwidth choice in OCSVMs, or the number of dimensions in PCA), we consider both the
best possible choices of such hyperparameters, which unfortunately are hard
to find in practice without labeled data, and default choices that might be
made in practice based on rule-of-thumbs. To ensure an apples-to-apples
comparison across procedures, we ensure
that all data representations result in feature vectors of a similar order in
dimension, in the range of 10 to 30 dimensions.
Our primary results are as follows:
\begin{itemize}
\itemsep=-1pt
    \item No single data representation works best across all devices and
        models.
    \item Performing a Fourier transform on the raw network packet traces
        generally has minimal effect on model accuracy, regardless of the
        representation of the data in the time domain.
    \item Incorporating packet size and TCP transport header information (e.g.,
        IP TTL, TCP flags) into the timeseries can significantly improve the accuracy
        of novelty detection, across a variety of models and outlier detection
        techniques.
\end{itemize}

\paragraph{Outline} Section~\ref{sec:data_representations} outlines the common
ways of representing network traffic flows in the literature. It is difficult
to be completely exhaustive, yet we focus on the most established approaches
for representing traffic data both in general network anomaly detection and in
the growing literature on novelty detection for IoT, comparing tuning and
configuration and tuning decisions that might be made in practice.
Section~\ref{sec:ml_procedures} outlines the machine learning models and performance
metrics that we use for our comparative analysis, as well as the metrics that
we use to evaluate these models.
Section~\ref{sec:experimentalsetup} describes the setup of our experiments,
including how we selected hyperparameters for various novelty detection
models, as well as the datasets that we used in our evaluation and how we
processed those datasets. Section~\ref{sec:results} describes the results from
our experiments, and Section~\ref{sec:conclusion} concludes with a summary and
directions for future work.

\section{Machine Learning Models and Performance Metrics}
\label{sec:ml_procedures}

We describe the novelty detection procedures that we evaluate in our study, as
well as the performance metrics that we use to compare each of these
procedures.

\subsection{Novelty Detection Procedures}

We consider popular novelty detection procedures from Machine Learning and Statistics, approaches based on {Isolation Forests} (IF), {One-Class Support Vector Machines} (OCSVM), {Kernel Density Estimation} (KDE), {Gaussian Mixture Models} (GMM), {Principal Component Analysis} (PCA), and {Auto Encoders} (AE). Important common aspects of all such detection procedures are as follows:
\begin{enumerate}
\item \emph{Data representation as features.} Detection procedures invariably
    require data---here network flows---represented as a vector $X \in \mathbb{R}^d$ of $d$ predictive features.
While many procedures operate directly on that feature representation (e.g.,
        IF, KDE, GMM), others (e.g., OCSVM, PCA, AE) remap vectors $X$ into
        refined representations where certain patterns---clusters of similar
        network activities---might be more evident.

\item \emph{Training and testing phase.} Detection procedures are trained in a first phase using $n$ training datapoints $\{X_i\}_{i=1}^n$ -- i.e., $n$ copies $X_i$ of typical values of \emph{normal} features $X$ (e.g., each $X_i$ representing a normal traffic flow), from which they produce a \emph{scoring function $S(X)$};  high scores $S(X)$ are expected to be typical of normal observations such as the training datapoints $X_i$,
while low-scores would be indicative of the query $X$ being an outlier (novelty). Subsequently, in the so-called testing phase (or deployment phase), a threshold $t$ might be chosen so that any query $X$ is flagged as \emph{novelty} whenever $S(X) < t$.

Note that, in practical use cases, the right choice of threshold can be difficult since the available training data is assumed to be all \emph{normal} making it hard to assess \emph{false negative rates}. Instead, one might choose a threshold based on the \emph{false positive rates} the application can tolerate, which can be estimated for each choice of $t$ based on the \emph{normal data} (i.e., data known to not have positive label). Related issues are discussed in Section \ref{sec:performancemetric} on performance metrics.

\end{enumerate}

\paragraph{Normality scores in ML.} In KDE or GMM, the training data is used to estimate the \emph{density} function $f$ of the data in $\mathbb{R}^d$ (that is their spatial distribution), where by definition $f(x)$ is low in those regions of space that have little data. This density would then serve as a scoring function $S(X) \doteq f(X)$. IF works similarly by first partitioning the space $\mathbb{R}^d$ into ensembles of high and low density regions, and flagging any query $X$ as a novelty if it lies in a low density region.

On the other hand, PCA and AE work by identifying a lower-dimensional space ${\cal X}$ embedded in the representation space $\mathbb{R}^d$ on which most of the training datapoints $X_i$ appear to lie. In PCA $\cal X$ is a linear space, while in AE it can be highly nonlinear (using nonlinear activation functions in a neural network). The score $S(X)$ then denotes how \emph{close} $X$ is to $\cal X$ with lowest scores for points farthest from $\cal X$.

OCSVM stands farther apart as it proceeds by first remapping all $X_i$'s -- through a transformation $\Phi(X_i)$ into a higher dimensional feature space $\cal H$ (possibly of infinite dimension) and then essentially identifying regions of high density in $\cal H$.
The underlying assumption is that patterns might be more visible in $\cal H$, for instance, all $\Phi (X_i)$ might cluster together into a \emph{normal} region (either a half space in $\cal H$ or an enclosing ball; the score $S(X)$ is then lower for queries $X$ that map as $\Phi(X)$ outside the \emph{normal} region of $\cal H$.

\subsection{Performance Metric}\label{sec:performancemetric}
\paragraph{Area Under the Curve (AUC).}
As explained above, detection procedures flag new queries $X$ as novel if it
scores below a certain threshold $t$ (i.e., if $S(X) < t$). Now notice a
tension: the higher the threshold $t$, the more likely it is that all novel
$X$ are correctly detected, but unfortunately, the more likely it is also that
normal datapoints are flagged as novel (a false alarm). Thus, the best
performing approaches---combination of data representation and choice of
detection procedure---are those that alleviate this tension, performing
accurate detection while minimizing false positives.

Such tradeoffs are well captured by a Receiver Operating Characteristic (ROC)
curve, which plots the detection rate against the false alarm rate as $t$ is varied from small to large; a large area under the curve (AUC) indicates that good tradeoffs are possible under the given detection approach. We therefore adopt AUC as a sensible measure of performance.

\paragraph{Sensitivity to Hyperparameters.} All detection procedures come with hyperparameters, i.e., configuration choices that can greatly affect performance. Important hyperparameters are e.g., bandwidth in KDE or OSCVM, number of partitions in IF, number of components in GMM, embedding dimension in PCA and AE. We therefore have to carefully pick such parameters in practice, although the right approach remains unclear, especially in \emph{unsupervised learning} problems such as novelty detection where we typically do not have a labeled validation set to evaluate choices.

Since hyperparameter tuning in unsupervised learning remains the subject of ongoing research in Machine Learning and Statistics, \emph{we decided to present results for the best choices of parameters}, picked a posteriori as those choices that result in the best performance on a test set, as described in Section \ref{sec:hyperparamTuning} below.

More configuration details such as range or parameters are given in Section \ref{sec:hyperparamTuning}. Other configuration choices based on common rule-of-thumbs are covered in the appendix along with relevant results.

\section{Data Representation}
\label{sec:data_representations}

The importance of data representation has in fact fostered many subfields such as \emph{metric learning}, \emph{representation learning} which aim to \emph{automatically} find the right representation to be coupled with given procedures, so far with mixed results, but with much success in application areas such as computer vision and speech processing {\color{red} \cite{}}. No such automatic approach has yet been developed in the fledgling application area of novelty detection for IoT, and a main aim of this work is to bring attention to the importance of data representation, and identify representations that work well in general while meeting practical needs for succinct representations.

Machine learning procedures require for the most part, \emph{feature vectors} of the same dimension, i.e., every flow has to be represented as a vector of a fixed number of carefully chosen features -- i.e., information extracted from the flow.

We consider various feature-representations of \emph{forward flows}, where a forward flow aggregates all packets, with a same \emph{five-tuple} identifier, sent from a device being monitored to another device or server. The five-tuple identifier consists of source IP address, source port, destination IP address, destination port, and protocol. The basic representations considered are as follows.

\begin{itemize}
    \item {\bf STATS:} a set of statistical quantities compiled from a flow. In particular, we choose 10 of the most common such statistics in the literature (see e.g., \cite{Moore2005}), namely,
flow duration, number of packets sent per second, number of bytes per second, and the following statistics on \emph{packet sizes} (in bytes) in a flow: mean, standard deviation, the first to third quantiles, the minimum, and maximum.

\item {\bf SIZES:} a flow is represented as a time series of packet sizes in bytes.

\item {\bf IAT:} a flow is represented as a time series of inter-arrival times between packets, i.e., elapsed time in seconds between any two packets in the flow.

\item {\bf SAMP-NUM:} a flow is partitioned into small time intervals of equal length $\delta t$, and the number of packets in each interval is recorded; thus a flow is represented as a time series of packet counts in small time intervals. Here, $\delta t$ might be viewed as a choice of \emph{sampling rate} for the time series, hence the nomenclature.

\item {\bf SAMP-SIZE:} a flow is partitioned into small time intervals of
    equal length $\delta t$, and the total packet size (i.e., byte count) in
        each interval is recorded; thus, a flow is represented as a time series of byte counts in small time intervals.

\end{itemize}

Remark that any of the above time series representations (i.e., all but STATS) might be given in time domain (which we choose by default) or in Fourier domain, in which case we append \emph{FFT} to the representation name, i.e.,
IAT-FFT, SAMP-NUM-FFT, and SAMP-SIZE-FFT.

When using ML procedures for novelty detection, we have to ensure that
all of the above representations result in feature-vectors of the same dimension
for any given device being monitored. This is automatically the case for
STATS, but not for the other representations, in light of variations in
flow durations and lengths (number of packets) for a given device.
To ensure that each sample has the same dimension,
we select a fixed time duration for all flows corresponding to the 90th percentile of all
flow lengths or durations for a device. We use this duration so as to capture
the typical behavior of most device instances of a particular type. Any
definition of typicality may be appropriate here, depending on the desired
compactness and complexity of the corresponding representation. Using the 90th
percentile of all flow durations results in a comparable number of dimensions
(typically 10--30) for most of the devices that we consider in this study.
Details on achieving fixed feature dimensions are given in Section
\ref{sec:experimentalsetup} on experimental setup.

\begin{table}[t]
\centering
\caption{Data representations that are commonly used for anomaly detection.}
\begin{tabular}{|l|p{1.5cm}|p{1cm}|p{1cm}|p{2cm}|p{1cm}|p{1cm}|}
\toprule
Reference & Flow duration & IAT & SIZES & FFT of IAT and SIZES & SAMP-NUM & SAMP-SIZES \\
\midrule
    {\bf Network Intrusion Detection} \\
\midrule
    Lakshari {\em et al.}~\cite{lashkari2018toward} & \cmark & \cmark& \cmark & \xmark & \xmark & \xmark  \\
\midrule
    Mukar {\em et al.}~\cite{kumar2018novel} & \xmark & \cmark& \xmark & \xmark & \xmark & \xmark  \\
    Mirsky {\em et al.}~\cite{mirsky2018kitsune} &\xmark & \xmark& \xmark & \xmark & \cmark & \cmark  \\
\midrule
\midrule
    {\bf IoT Anomaly Detection} \\
\midrule
    Al Jawarneh {\em et al.}\cite{aljawarneh2018garuda} & \cmark & \xmark& \cmark & \xmark & \xmark & \xmark  \\
    Doshi {\em et al.}~\cite{doshi2018machine} & \xmark & \cmark& \cmark & \xmark & \xmark & \xmark  \\
\midrule
    Meidan {\em et al.}~\cite{meidan2018n} &\xmark & \cmark& \xmark & \xmark & \cmark & \cmark  \\
\midrule
    Bhatia {\em et al.}~\cite{bhatia2019unsupervised} & \xmark & \xmark& \cmark & \xmark & \cmark & \xmark  \\
\midrule
    Thamilarasu {\em et al.}~\cite{thamilarasu2019towards} & \xmark & \xmark& \cmark & \xmark & \xmark & \xmark  \\
\midrule
\midrule
    {\bf DDoS Detection} \\
\midrule
    Fouladi {\em et al.}~\cite{fouladi2016frequency} &\xmark & \xmark& \xmark & \cmark & \xmark & \xmark  \\
\midrule
    Lima {\em et al.}~\cite{lima2019smart} &\xmark & \xmark& \cmark & \xmark & \xmark & \xmark  \\
\bottomrule
\end{tabular}
\label{features}
\end{table}

\section{Results} \label{sec:results}

In this section, we explore how data representation affects the performance of
different novelty detection algorithms.
We are interested in the difference in AUC due to the presence or absence of
particular feature types with respect to a set of basic feature representations, namely
STATS, SIZE, IAT, SUMP-NUM, and SAMP-SIZE (Table
\ref{tab:OCSVM_IF_AE_and_KDE_with_best_parameters} also includes IAT+SIZE as a
baseline for future comparisons). We will see that across ML procedures, (1)
Fourier domain representations have little advantage over raw time series
(despite being generally thought of as a better way to capture important
trends in timeseries), while the effect of (2) packet size information, and
(3) packet header information tend to be significant overall. We focus on some
of the most popular ML approaches (OCSVM, IF, AE, KDE) in the main text, under
{\bf OPT} tuning, while results for omitted ML approaches, for {\bf Default}
tuning, and omitted datasets are given in the appendix.
We substantiate messages (1), (2), and (3) in subsections \ref{sec:fft},
\ref{sec:size}, and \ref{sec:header}.  Table
\ref{tab:OCSVM_IF_AE_and_KDE_with_best_parameters} gives baselines for
comparisons.

\paragraph{Statistical significance.} Throughout this section, we often
display error bars of length $1/\sqrt{n}$ ($n$ denoting test sample sizes as
given in Table \ref{tab:train_test_sets}), which is a ballpark for standard
deviation in AUC computed on a test sample of size $n$.

We first explore the effect of Fourier domain representations; then we explore
how including packet sizes and fields from the packet header ultimately
affect the performance of different novelty detection approaches.

\begin{table}[t] \centering

\caption{AUCs for 4 ML approaches on basic feature representations.}
    \label{tab:OCSVM_IF_AE_and_KDE_with_best_parameters}

\begin{tabulary}{0.99\textwidth}{|C|C|>{\columncolor[gray]{0.9}}C|C|>{\columncolor[gray]{0.9}}C|C|>{\columncolor[gray]{0.9}}C|C|}
\toprule Detector & Dataset & \Cell{STATS} & \Cell{SIZE} & \Cell{IAT} &
    \Cell{IAT+\$SIZE$} & \Cell{SAMP-\\NUM} & \Cell{SAMP-\$SIZE$}  \\ \midrule
    \multirow{7}{*}{OCSVM} &UNB(PC1) & 0.56 & 0.70 & 0.78 & 0.92 & 0.74 & 0.76
    \\ &UNB(PC4) & 0.48 & 0.66 & 0.82 & 0.80 & 0.81 & 0.79 \\ \cmidrule{2-8}
    &CTU & 0.98 & 0.97 & 0.95 & 0.96 & 0.88 & 0.88\\ \cmidrule{2-8} &MAWI &
    0.88 & 0.71 & 0.58 & 0.43 & 0.60 & 0.61\\ \cmidrule{2-8} &TV\&PC & 1.00 &
    1.00 & 0.96 & 0.99 & 0.95 & 0.93\\ \cmidrule{2-8} &SFrig & 0.98 & 0.93 &
    0.81 & 0.85 & 0.96 & 0.95 \\ &BSTch & 0.97 & 0.99 & 0.97 & 0.99 & 0.96 &
    0.95  \\ \midrule \multirow{7}{*}{IF} &UNB(PC1) & 0.49 & 0.49 & 0.51 &
    0.74 & 0.66 & 0.70  \\ &UNB(PC4) & 0.49 & 0.51 & 0.62 & 0.65 & 0.81 & 0.79
    \\ \cmidrule{2-8} &CTU & 0.94 & 0.93 & 0.81 & 0.88 & 0.83 & 0.85\\
    \cmidrule{2-8} &MAWI & 0.86 & 0.70 & 0.54 & 0.67 & 0.56 & 0.55\\
    \cmidrule{2-8} &TV\&PC & 0.95 & 0.98 & 0.90 & 0.99 & 0.74 & 0.79\\
    \cmidrule{2-8} &SFrig & 0.96 & 0.96 & 0.56 & 0.94 & 0.93 & 0.93 \\ &BSTch
    & 0.97 & 0.98 & 0.94 & 0.98 & 0.98 & 0.96  \\ \midrule \multirow{7}{*}{AE}
    &UNB(PC1) & 0.72 & 0.40 & 0.50 & 0.93 & 0.81 & 0.77  \\ &UNB(PC4) & 0.70 &
    0.52 & 0.64 & 0.84 & 0.83 & 0.86 \\ \cmidrule{2-8} &CTU & 0.91 & 0.96 &
    0.85 & 0.96 & 0.83 & 0.87\\ \cmidrule{2-8} &MAWI & 0.56 & 0.58 & 0.50 &
    0.70 & 0.58 & 0.55\\ \cmidrule{2-8} &TV\&PC & 1.00 & 1.00 & 0.97 & 0.99 &
    0.85 & 0.95\\ \cmidrule{2-8} &SFrig & 0.95 & 0.85 & 0.72 & 0.92 & 0.96 &
    0.96 \\ &BSTch & 0.97 & 0.93 & 0.98 & 0.99 & 0.96 & 0.96  \\ \midrule
    \multirow{7}{*}{KDE} &UNB(PC1) & 0.34 & 0.58 & 0.68 & 0.92 & 0.74 & 0.76
    \\ &UNB(PC4) & 0.47 & 0.63 & 0.76 & 0.80 & 0.81 & 0.85 \\ \cmidrule{2-8}
    &CTU & 0.96 & 0.93 & 0.94 & 0.97 & 0.82 & 0.84\\ \cmidrule{2-8} &MAWI &
    0.88 & 0.48 & 0.57 & 0.43 & 0.60 & 0.57\\ \cmidrule{2-8} &TV\&PC & 1.00 &
    1.00 & 0.96 & 0.99 & 0.88 & 0.93\\ \cmidrule{2-8} &SFrig & 0.97 & 0.91 &
0.79 & 0.78 & 0.95 & 0.95 \\ &BSTch & 0.97 & 0.98 & 0.95 & 0.98 & 0.95 & 0.95
\\ \bottomrule \end{tabulary} \end{table}

\subsection{Effect of Fourier Domain Representation}\label{sec:fft}

\begin{figure}[t] \centering
    \includegraphics[width=0.99\textwidth]{OCSVM_IF_AE_KDE_with_best_parameters_on_basic_representations_diff.pdf}
    \caption{Difference in AUC for {\bf FFT vs. raw time series}
    representations.}
\label{fig:diffinFFT} \end{figure}

We compare Fourier domain representations---obtained by \emph{Fast Fourier
    Transforms} (FFT), where we retain as many FFT components as the number of
    time points---to raw time-domain representations of each flow IAT,
    SAMP-NUM, SAMP-SIZE. Figure \ref{fig:diffinFFT} presents the difference in
    AUC, i.e., the AUC obtained under FFT minus that obtained under the
    corresponding raw time series representation.

The same trend is observed across ML procedures: apart for a few device
    datasets (e.g., positive effect in SFrig and negative effect in BSTch
    w.r.t. IAT), FFT makes no significant difference in the achieved AUC over
    raw time series. In other words, the various ML procedures seem able to
    extract much of the same information already from the raw time series,
    i.e., their \emph{internal representations} of the data already account
    for major trends in the times series and thus do away with the need to
    preprocess through FFT.

Henceforth, we focus attention on raw representations for time series in
    subsequent discussions.

\subsection{Effect of Packet Size Information}\label{sec:size}

\begin{figure}[t] \centering
    \includegraphics[width=0.99\textwidth]{Effect_of_Size_OCSVM_IF_AE_KDE_with_best_parameters_diff.pdf}
\caption{Differences in AUC due to {\bf including vs. excluding packet size}
    information.}
\label{fig:diffinsize} \end{figure}

Referring back
to Table~\ref{tab:OCSVM_IF_AE_and_KDE_with_best_parameters}, packet size
information immediately appears to be important based on the generally higher
AUCs achieved under STATS (which primarily compiles statistics on packet sizes
in a flow) to those achieved under basic representations such as IAT and
SAMP-NUM (devoid of packet size information). Interestingly, SIZE, a time
series of packet sizes in a flow, does not perform as well as STATS does over
IAT, and SAMP-NUM.

As it turns out, adding packet size information to IAT or SAMP-NUM
representations improves on such representations alone.  Namely, in the case
of IAT, size information is added in by concatenating IAT and SIZE vectors,
which we denote IAT+SIZE. For SAMP-NUM, a time-series of \emph{number of
packets} in small fixed time intervals $\delta t$, size information is
accounted for by instead compiling \emph{total packet size} in fixed time
intervals $\delta t$.

Figure \ref{fig:diffinsize} presents the differences in AUC obtained using
packet size information, minus those obtained on corresponding representations
without size information. We observe significant positive differences for most
ML procedures on most datasets---which correspond to a range of
novelty detection problems, whether malicious attacks, or novel devices or
activity. Even in those cases when size information does not help, it
nonetheless does not hurt, except in rare cases such as MAWI under IAT when
employing OCSVM and KDE. However, even in this case, we observe (see Table
\ref{tab:OCSVM_IF_AE_and_KDE_with_best_parameters}) that the concatenation of
IAT+SIZE yields {\em worse} AUC than IAT or SIZE alone. The exact reason is unclear
and could be due to \emph{overfitting}, as the concatenation increases feature
dimension in ways that might affect \emph{nonparametric} models such as OCSVM
and KDE given the relatively modest training size of MAWI (800+ datapoints).

The most significant positive differences are observed for IF, AE, and PCA
in the case of IAT, while
no significant differences are observed for sampling-based representations,
such as
SAMP-SIZE vs. SAMP-NUM. Interestingly, for fixed datasets, trends are generally
consistent across ML methods, suggesting that whether size information helps
depends most on devices' characteristics, rather than the ML method
employed or the type of detection problem.

In general, the inclusion of packet sizes for novelty detection makes sense,
as the packet size can provide some indication about the nature of the
underlying activity. For example, for the case of the UNB dataset, attack
traffic may be unique in size relative to other types of activities. In the
case of the private IoT datasets, novel activities may be less likely to
produce individual packets of differing sizes unless the activities themselves
are new.

\subsection{Effect of Packet Header} \label{sec:header}

We now consider the effects of information in packet headers on novelty
detection, focusing on the information available in various TCP flags. We
focus on the following eight flags in each packet: FIN, SYN, RST, PSH, ACK,
URG, ECE, and CWR, and TTL. We do not include IP address or port information
in this part of our analysis.  Given the already established benefits (or at
least invariance in AUC) of including packet size information, we now take
representations such as STATS, IAT+SIZE and SAMP-SIZE as baselines to which we
concatenate packet header information (for each packet in a flow) and record
differences in performance.

In general, we observe no significant degradation in performance in adding
header information, and in fact, observe improvements for some datasets such
as SFrig under OCSVM and KDE (novel activity detection) where we might not
necessarily expect such header information to help.
The simple explanation is that in
fact, even for problems such as activity detection, a device might switch
destination IP or ports depending on the type of activity being performed.

Figure \ref{fig:diffpacketheader} presents the differences in AUC obtained
with packet header information minus that obtained without header information.
The same trends are observed across ML approaches. As might be expected, we
observe the most significant improvements on those datasets corresponding to
(1) detecting malicious activity (UNB)---where infected traffic
might be rerouted to new destinations, thereby resulting in traffic with
different distributions of packet TTL values corresponding to the different,
new destinations---or (2) detecting novel devices (CTU,
MAWI)---where header information such as certain TCP
options can a device or operating system pertinent to a
device.

Although TV \& PC seem to be an exception to (2), the general lack of
improvement is simply due to the fact that AUC's for the baseline
representations were already near perfect (close to 1) across ML approaches
(see Table \ref{tab:OCSVM_IF_AE_and_KDE_with_best_parameters}).  This happens
to also be the case (i.e., near perfect AUC's) for those baseline
representations where we observe little difference in AUC for the two
detection problems (1) and (2). The most improvement across ML methods is
observed for MAWI where the baseline representations yielded poor AUC's to
start with (Table \ref{tab:OCSVM_IF_AE_and_KDE_with_best_parameters}).

\begin{figure}[t] \centering
    \includegraphics[width=0.99\textwidth]{Effect_of_Packet_Header_OCSVM_IF_AE_KDE_with_best_parameters_diff.pdf}

\caption{Differences in AUC due to {\bf including vs. excluding packet header}
    information.}
\label{fig:diffpacketheader} \end{figure}

\begin{figure}[h] \centering
\includegraphics[width=0.99\textwidth]{correlation-bar.pdf} \caption{Top 6
correlation between each value in packet headers and ground truth ($y$).}
\label{fig:correlation} \end{figure}

Figure~\ref{fig:correlation} shows that certain packet header fields can be particularly useful for
novelty detection, depending on the anomaly; these results are evident in the
results. One of the more important findings from taking a closer look at the
correlations between features and ground truth is that the most important
packet header values will {\em depend on the novelty detection problem}. We
explore this in more detail below.

\paragraph{Attack detection.}
In the case of the UNB dataset, we see that the SYN flag can be very useful
in identifying novelty events related to security, as the SYN flag correspond
to a TCP handshake that occurs at the beginning of the connection and is
commonly related to certain types of denial of service attacks, such as a SYN
flood. Other packet header information that exhibits correlation include other
flags that are commonly associated with denial of service attacks, including
the RST and FIN flags, which victim hosts may send in response to a SYN. In
all of these cases, an increase in packets that include these flags could
likely correspond with a volumetric denial of service attack, such as a SYN
flood.

\paragraph{Novel device detection.}
The MAWI dataset, where the novelty detection problem involves detecting a
novel device, finds the TTL field in a packet header to be significant. This
also follows intuition, as the TTL value is a coarse indicator of network
topology (i.e., how many network-level ``hops'' a device is from a particular
destination endpoint) and naturally two different devices on the network may
be located at different network attachment points and thus be different
distances away from common network locations. Such topological differences
would be apparent in the TTL field, and a new device attaching at a different
location on the network could appear as a sudden influx of network traffic
that bears a different distribution in TTL values.  In particular, sudden
deviations in distributions of TTL values in the dataset may thus reflect the
injection of traffic from new devices, or possibly attackers.  On the other
hand, we see in this dataset that the SYN and TTL fields are less useful for
novelty detection concerning activities (e.g., the SFrig dataset, which
involves detection of activities in the dataset).

\paragraph{Novel activity detection.} If someone begins to interact with a
connected IoT device in new ways---triggering new types of activities---the
traffic itself may bear differences from previously seen traffic. Differences
in the traffic itself may arise because activities themselves may bear unique
signatures in the underlying network traffic. For example, the amount and
nature of traffic generated by opening and closing a refrigerator door in
SFrig will generate different types of traffic than the kind of traffic that
is generated by activities that do not involve human interaction (e.g., a
software update).  These fundamental differences may appear in different ways,
such as differences in traffic volume or timing, and in some cases they may
appear in the TCP header flags themselves. In the case of SFrig, we see that
the ACK flag has a high correlation with ground truth, indicating that changes
in the prevalence of ACKs (i.e., packet acknowledgments) may represent changes
in underlying activities. This characteristic likely results from the
underlying behavior of TCP, the Internet's transport protocol, and how it
handles packet acknowledgments for flows of different sizes and timings. For
example, for large traffic flows, TCP will sometimes optimize its
acknowledgment behavior, using techniques such as {\em delayed ACK} to improve
network performance.  Highly interactive behavior and the traffic it generates
may exhibit very different qualities than automated traffic (e.g., a software
update involving the transfer of a large amount of data) and thus different
behavior in ACK traffic.  The distribution of ACK fields in the traffic may
reflect higher traffic volumes during periods of novel activity.

\section{Experiment Setup} \label{sec:experimentalsetup}

We first describe the process by which we choose hyperparameters for each of
the novelty detection procedures that we evaluate in this study. We then
describe the datasets we use to evaluate each novelty detection method,
including how we process each of these datasets and select parameters for each
corresponding feature representation.

\subsection{Hyperparameter Choices for Novelty Detection Procedures}
\label{sec:hyperparamTuning}
All detection procedures are implemented in the scikit-learn Python package. We now describe configuration choices, both for the best hyperparameter choice (OPT) -- picked a posteriori using test data as described in Section \ref{sec:performancemetric} above -- and a rule-of-thumbs choice (Default). The results in the main paper are presented for the OPT choice, while results for Default are presented in the appendix.

\begin{itemize}
    \item OCSVM \cite{scholkopf2000support} (implemented by
        \cite{zhao2019pyod}). We choose a Gaussian kernel of the form $K(x,
        x') \propto \exp(-\|x-x'\|^2/2\sigma^2)$, and pick $\sigma$ as
        follows. We consider the quantiles $[0.1, 0.2, \ldots, 0.9] \cup
        \{0.95\}$ on increasing distances between pairs of points in the
        training sample.\\ {\bf OPT:} The parameter $\sigma$ is then picked as
        that quantile which yields the best AUC.\\ {\bf Default:} $\sigma$ is
        picked as the 0.3 quantile of inter-point distances.

    \item KDE \cite{latecki2007outlier, scott2005multidimensional}. We use a Gaussian kernel, of the same form as OCSVM above, and pick the \emph{bandwidth} $\sigma$ exactly the same way for both {\bf OPT} and {\bf Default}.
    \item GMM \cite{Bishop2006Pattern}. We choose the number of Gaussian components $k$ on a range
    $[2, 5, 8, 11, 14, 17, 20, 23, 26, 30]$. \\
    {\bf OPT:} $k$ is chosen as the value in the range maximizing AUC. \\
    {\bf Default:} $k$ is obtained as the number of high-density regions (or modes) in the training data, as obtained by the \texttt{quickshift++} procedure of \cite{jiang2018quickshift++}.

    \item IF \cite{liu2012isolation} (implemented by \cite{zhao2019pyod}). We
        pick the number $k$ of trees in the range 30 to 300 with increments of
        10. \\
    {\bf OPT:} $k$ is chosen as the value in the range maximizing AUC. \\
    {\bf Default:} $k$ is 100, which is the same as the default value in pyod.

    \item PCA \cite{aggarwal2015outlier} For $X \in \mathbb{R}^d$, i.e., having $d$ features, we pick the projection dimension $\text{dim}$ in a range. We pick $\text{dim}$ in a range
    $\{\lceil 1 + i\cdot (d -2)/9 \rceil \}_{i = 0}^9$
    of (up to 10) values (approximately evenly spaced) between $1$ and
        $d-1$.\\
    {\bf OPT:} $\text{dim}$ is chosen as the value in the range maximizing
        AUC. \\
    {\bf Default:} $\text{dim}$ is estimated by maximum likelihood (MLE) in scikit-learn.

    \item AE \cite{zhou2017anomaly} (from PyTorch \cite{NEURIPS2019_9015} using LeakyRelu for activation functions). For $X\in \mathbb{R}^d$, we use the following architecture with 5 layers, determined by a tuning parameter $\text{dim}$: one input layer of size $d$ (number of \emph{neurons}), followed by a hidden layer of size $\text{h-dim} \doteq \text{min}\{(d-1), \lceil 2\cdot \text{dim}\rceil \}$, a latent layer of size $\text{dim}$, a subsequent hidden layer of size $\text{h-dim}$, and finally an output layer of size $d$. {We pick $\text{dim}$ in a range
    $\{\lceil 1 + i\cdot (d -2)/9 \rceil \}_{i = 0}^9$
    of (up to 10) values (approximately evenly spaced) between $1$ and $d-1$}.
        \\
    {\bf OPT:} $\text{dim}$ is chosen as the value in the range maximizing
        AUC. \\ {\bf Default:} $\dim$ is $\lceil d/2\rceil $.

\end{itemize}

\subsection{Data}

We consider a combination of publicly available traffic traces and
traces collected on private consumer IoT devices. We aim to evaluate
representative set of devices, from multi-purpose devices such as laptop PCs,
and Google Home, to less complex electronics and appliances such as smart
cameras and TVs. Furthermore we aim at a a representative set of novelty, from
benign novelties (new activity, or a new device type), to novelties due to
malicious activities (DDoS attack).  Table \ref{dataset_sources} describes
these datasets, and types of novelty being detected.

\begin{table}[t!]
\centering
\caption{Datasets.}
\begin{tabular}{|p{1.5cm}|p{6.1cm}|p{3.6cm}|p{3.6cm}|}
\toprule
Reference & \begin{tabular}[c]{p{6.1cm}}{Description} \end{tabular} & \begin{tabular}[c]{p{3.5cm}}{Devices } \end{tabular}& \begin{tabular}[c]{p{3.6cm}}{Type of Novelty} \end{tabular}  \\
\hline
UNB IDS~\cite{cicids2017}  & \begin{tabular}{p{6.1cm}} Normal and DDoS attack traces from personal computers (PC), which are chosen from Monday trace and Friday trace.  \end{tabular} &
\begin{tabular}{p{3.6cm}}Five PCs (noted as PC1, PC2, PC3, PC4, and PC5, respectively)  \end{tabular}& \begin{tabular}{p{3.5cm}} DDoS attack\end{tabular}\\
\hline
CTU IoT~\cite{ctu2019iot}  & \begin{tabular}{p{6.1cm}}
 Bitcoin-Mining and Botnet traffic traces generated by two Raspberries; we aim to distinguish the two devices by their attack traces (i.e., Mirai and CoinMiner); hence we use one generated Mirai as \emph{normal}, the other generated CoinMiner as \emph{novel}.\end{tabular} & \begin{tabular}{p{3.6cm}}Two Raspberries \end{tabular} & \begin{tabular}{p{3.5cm}} Novel (infected) device \end{tabular} \\
\hline
MAWI~\cite{mawi2019normal} & \begin{tabular}{p{6.1cm}} Normal traffic generated by two PCs, which are collected on Dec. 07, 2019; we use one PC (whose IP address is 202.171.168.50) as \emph{normal}, the other (whose IP address is 202.4.27.109) as \emph{novel}. \end{tabular} & \begin{tabular}{p{3.6cm}}Two PCs  \end{tabular} &  \begin{tabular}{p{3.5cm}} Novel (normal) device \end{tabular}\\
\hline
\multirow{5}{*}{\Cell{Lab \\IoT}} & \begin{tabular}{p{6.3cm}}Data traces are generated by IoT devices (e.g., fridge) in a private lab environment.
We choose two kinds of normal traffic generated by a smart TV and a PC.
\end{tabular} & \begin{tabular}{p{3.6cm}}A smart TV and a PC (noted as TV\&PC) \end{tabular} & \begin{tabular}{p{3.5cm}}Novel device\end{tabular}  \\
\cline{2-4}
 & \begin{tabular}[c]{p{6.3cm}} Four smart-home devices, each with two types of traffic traces labeled as \emph{normal} when there is no human interaction, and \emph{novel} when being operated by a human.

 \end{tabular} & \begin{tabular}[l]{p{3.6cm}} Google home (GHom), Samsung camera (SCam), Samsung fridge (SFrig), and Bose soundtouch (BSTch)\end{tabular} & \begin{tabular}{p{3.5cm}}Novel activity \end{tabular} \\
\bottomrule
\end{tabular}

\label{dataset_sources}
\end{table}

\paragraph{Obtaining flows.}
We parse upstream flows from datasets in Table \ref{dataset_sources} using
    Scapy \cite{scapy}. Given that certain devices can have arbitrarily long
    flows, we truncate each flow from a given dataset to have duration at most
    that of the 90th upper-percentile of flow durations in the dataset.
    Henceforth, a \emph{flow} refers to these choices of flows involving
    truncation.

\paragraph{Selecting feature representation dimension.}
Each flow is then represented using any of the feature-representation choices described in Section \ref{sec:data_representations}. Here, we have to ensure that, given a representation choice, e.g., IAT,
all flows result in vectors of the same dimension $d$ for each dataset. We apply a similar approach to enforce such fixed dimension $d$, as outlined below. For a fixed dataset, let $d_0$ denote the 90th upper-percentile of number of packets per flow.
\begin{itemize}
    \item STATS: automatically results in vectors of the same dimension $d$,
        equal to the number of statistics (on packet size) computed on a flow.
    \item IAT, or SIZE: dimension is fixed to $d = d_0-1$ for IAT, and $d = d_0$ for size; namely, for long flows, we truncate the number of packets in a flow to the first $d_0$; for short flows where the number of packets is less than $d_0$, we append $0$'s to the IAT representation to arrive at $d$ features.
    \item IAT + SIZE: just concatenates IAT and SIZE representations to dimension $d = 2d_0 -1$.
    \item SAMP-NUM or SAMP-SIZE: dimension is fixed to $d = d_0-1$ as in the case of IAT. Now we pick
    a fixed window size $\delta t$, and each flow is divided into up to $d$ windows of length $\delta t$; for flows of short duration, where $\delta t$ is large w.r.t. duration, we simply append $0$'s.

    \emph{Choice of $\delta t$:} we start with candidate choices $\Delta \doteq \{t_f/d\}$, over durations $t_f$ of flows $f$ in a given dataset. We then consider 10 choices of $\delta_t$ in $\Delta$, each choice an $i$-th quantile of values in $\Delta$, for each $i \in \{10, 20, 30, 40, 50, 60, 70, 80, 90, 95\}$. Thus, each such choice of $\delta t$ yields one (SAMP-NUM or SAMP-SIZE) representation for all flows; since we do not know which such representation (i.e. choice of $\delta t$) is best a priori\footnote{In fact we do not know how such choice might be automated in practice, as automating \emph{sampling rates} remains an largely open research problem.} we report the best AUC resulting out of all 10 choices of $\delta t$.
\end{itemize}

Each of the above representations, besides STATS, can then be viewed as fixed length time series, and therefore admit a Fourier domain representation; the corresponding Fourier representation is then set to the same dimension $d$, i.e., we retain as many Fourier components as the original time series.

The resulting representation dimensions for each dataset are reported in Table \ref{tab:feature_dimensions} in the appendix.