\section{Introduction}

IoT (Internet of Things) is an ecosystem of interconnected devices, each embedded with computational tools such as sensors or processing units, to collect, store, and exchange data over the internet \cite{lee2015internet}. Each device is performing a task to deliver the overall function or purpose of the IoT ecosystem, such as smart homes, healthcare, or industrial operations. According to a research by the International Data Corporation (IDC), due to the demand for IoT networks, it is estimated that the number of IoT devices would reach up to 41.6 billion in 2025 \cite{wu2020convergence}. IoT ecosystem can deliver great potential to businesses and consumers due to its digital and automated intelligence capabilities, increasing operational agility and efficiency. However, securing IoT networks has become one of the main challenges of implementing such networks \cite{zhang2014iot}. The equipped devices are collecting and storing sensitive data which are often breach targets for attackers. The current state of IoT networks makes it harder for businesses and consumers to secure their digital assets. As IoT connects the digital and physical worlds, hacking into IoT devices may lead to critical concerns \cite{alaba2017internet}.

Network Intrusion Detection Systems (NIDSs) aim to protect and preserve digital IoT networks from network threats \cite{chaabouni2019network}. Traditional NIDS aims to analyse incoming traffic signatures and match them to one of the known attack signatures. This method provides high detection accuracy of known attacks, however, it fails to detect unseen threats to IoT networks, known as zero-day attacks. Therefore, researchers have applied emerging Machine Learning (ML) technologies to learn and detect the harmful patterns of network traffic to detect intrusions \cite{sinclair1999application}. Plenty of academic research has been applied in this domain, many ML-based NIDS have been developed, mostly achieving high detection accuracy when applied to certain datasets. However, the number of practical deployments of such systems into networks is vastly low \cite{sommer2010outside}. Several factors are contributing to the lack of translating academic research into operation. However, two main reasons seem to top the list; the lack of understanding of internal ML-based NIDS operations \cite{amarasinghe2018toward} and the following of unreliable evaluation methodologies \cite{sarhan2021towards}.

Currently, ML-based NIDSs are offered as a complex 'black-box' that achieve promising detection results on certain datasets. The complexity of ML models makes them hard to explain the rationale behind the predictions made. Consequently, organisations are reluctant to trust ML decisions in a highly sensitive area such as intrusion detection \cite{amarasinghe2018toward}. Therefore, eXplainable Artificial Intelligence (XAI) methods are being applied in modern systems to explain and interpret the decisions made by ML models. Explainable ML outputs can help maintain and troubleshoot the deployment of an ML-based NIDS by understanding and modifying the factors contributing to the model's decision. Moreover, the datasets used in the evaluation stage of such systems are often generated with a unique set of data features. Thus, evaluating a learning model's detection capability using a targeted feature set across multiple datasets is limited. This removes the conclusion that the model's performance will generalise across network environments and attack types.

A recent work \cite{sarhan2021towards} proposed a standard feature set, which enables the evaluation of proposed systems across multiple datasets using a common feature set. However, the performance of ML models on the proposed NetFlow-based feature set has never been compared to another set of features across multiple datasets. Therefore, this paper aims to compare the feature set designed by the CICFlowMeter \cite{lashkari2017characterization} to the NetFlow feature set proposed in \cite{sarhan2021towards} across three datasets, i.e. CSE-CIC-IDS-2018, ToN-IoT and BoT-IoT. Two datasets have been generated as part of the experiment, named CIC-ToN-IoT and CIC-BoT-IoT, and have been made publicly available at \cite{netflow_datasets_2020}. This will accommodate for a reliable evaluation of the proposed ML model using the NetFlow and CICFlowMeter feature sets across multiple datasets. Additionally, the classification results achieved by the ML classifiers have been explained by calculating the Shapley values of each feature using the SHapley Additive exPlanations (SHAP) methodology. This will aid in the identification of key features utilised in the model's predictions.

\section{Related Work}
\label{rw}

In this section, some of the works that focused on explaining and evaluating the generalisability of feature sets of ML-based NIDS are discussed. In \cite{moustafa2015significant}, Moustafa et al. examined the attack detection performance of the UNSW-NB15 features against the features of the KDD-99 dataset. The experiment extracted the features of the UNSW-NB15 from the KDD99 dataset. The Association Rule Mining (ARM) feature selection technique is applied to determine the most important features across the two datasets. The detection accuracy is measured using a Naive Bayes (NB) model and Expectation-Maximization (EM) clustering. The results show that the original KDD-99 features are less efficient than the UNSW-NB15 features in terms of attack detection. The NB and EM models achieved an accuracy of 62.02\% and 52.54\% using the original KDD-99 features and 78.06\% and 58.88\% using the UNSW-NB15 features. The authors did not replicate the KDD-99 features on the UNSW-NB15 dataset to reliably evaluate the two feature sets.

In \cite{sarhan2020netflow}, Sarhan et al. highlighted the limitations of existing NIDS datasets, as their feature sets are unique and completely different from each other. Therefore, the evaluation methods of the proposed ML-based NIDS are often unreliable. The lack of a common ground feature set prohibits the evaluation of the ML model's performance ability to generalise across multiple datasets. As a solution, the paper generated and published four datasets; NF-UNSW-NB15, NF-BoT-IoT, NF-ToN-IoT, and NF-CSE-CIC-IDS2018, sharing the same 12 NetFlow-based features. NetFlow features are also faster to extract from network traffic due to their presence in packet headers compared to complex features requiring deep packet inspection. As a use case, an Extra Tree classifier is reliably evaluated across the four datasets using the common feature set. Results indicate a reliable performance while conducting a binary-class detection scenario. However, poor results are achieved following a multi-class detection scenario.

In \cite{sarhan2021towards}, the authors extended the feature set proposed in \cite{sarhan2020netflow}, by extracting a total of 43 NetFlow-based features. The additional extracted features contain an adequate amount of security events to enhance the ML model detection rate of attacks present in the datasets. The generated and labelled datasets named NF-UNSW-NB15-v2, NF-BoT-IoT-v2, NF-ToN-IoT-v2 and NF-CSE-CIC-IDS2018-v2 are published for research purposes. Their common feature set is proposed as a standard set to be used across future NIDS datasets. The authors argue the tremendous benefits of having a universal NetFlow-based standard feature set due to its wide usage in the networking industry, practical deployment, and scaling properties. As a use-case, an Extra Tree classifier is used to evaluate the proposed feature set and compare its detection accuracy with the limited NetFlow datasets generated in \cite{sarhan2020netflow} and the original feature set of the datasets. The results indicate that the proposed NetFlow feature set vastly outperforms the other feature sets in terms of attack detection accuracy using a single ML classifier.

In \cite{wang2020explainable}, Wang et al. aimed to explain the internal operations of ML-based NIDS. The experimental methodology utilised the SHAP technique to interpret the decisions of the NIDS. To their knowledge, this is the first paper in the intrusion detection field to utilise the SHAP method in the understanding of the judgement and structure of the NIDS. The paper also explored the differences in interpretation between a binary- and multi-class classifier. The paper used a Deep Feed Forward (DFF) with a ReLU activation function. The classifiers achieved an F1 score of 0.807 and 0.792 in binary- and multi-class experiments. However, the detection performance achieved by the model is weak and the utilised single dataset, i.e. NSL-KDD dataset in the experiment, has no significant value in representing modern network attacks due to its old age \cite{8672520}. The paper used a single attack class by selecting a random 100 samples of 'Neptune' attacks for the local explanation phase. The classifiers utilised different features to reach their prediction. Finally, the top 20 important features for each attack type used in the global explanation are listed and comprehensively analysed via related works in the paper.

In \cite{mane2021explaining}, the paper designed a DFF and proposed an eXplainable Artificial Intelligence (XAI) framework to add transparency. The DFF model is made up of 3 hidden layers performing a ReLU activation function and the output layer performing the softmax activation function. One aspect of the paper is the exploration of multiple XAI techniques, i.e. LIME, SHAP, Boolean Decision Rules via Column Generation (BRCG), and Contrastive Explanation Method (CEM). However, the paper utilised the NSL-KDD dataset to validate their methodology, which has been criticised in the field due to the lack of complex current network attack scenarios in the KDD-based datasets \cite{8672520}. The SHAP results deemed that a high value of the \textit{'same\_srv\_rate’} feature increases the probability of an attack prediction whereas a high value of the \textit{‘dst\_host\_serror\_rate’} feature increases the chances of a benign prediction. The authors applied the BRCG method to the dataset to extract model rules and achieved an accuracy of ~80\%. The LIME method generated a local explanation on specific data samples indicating which features contributed towards an attack or a benign prediction. Finally, the CEM showed how the prediction can be changed by altering feature values on a single benign data sample.

\section{Explainable NIDS}
\label{inter}

A great work of development has been put in the design of ML-based NIDS, often achieving great accuracy. However, the trust in their operations of securing computer networks has always been in question \cite{amarasinghe2018toward}. As a result, a very limited number of commercial NIDS are equipped with ML capabilities. It is believed this is due to the nature of such tools where the cost of errors are significantly higher compared to other domains. As the operation of NIDS requires to be optimal at all times to avoid disasters. ML is often known to be a 'black box' technology where there is no clear understanding of what patterns are learned or why predictions are made. In the context of ML-based NIDS, the way models train, predict, and classify network traffic into an attack category is often magical. Therefore, organisations are reluctant to implement an ML-based tool. There are multiple reasons such as data source, data features,  class imbalance, and inaccurate datasets that can have a vast impact on the predictions made by ML models. This makes it critical to understand and gain insights into the ML internal operations and decisions made.

The motivation above led to the generation of an ongoing research area of XAI where they aim to analyse and explain the internal operations of ML models. Therefore, implementing the technologies of XAI into the ML-based NIDS is important to tighten the gap between the extensive academic research conducted and the number of operational deployments by increasing trust in ML. One way to interpret the detection results of an ML model is to determine which network data features contribute towards the decision of the classifier. By determining which features of the dataset were used by the classifier to differentiate between a benign and an attack sample is critical for many reasons. Firstly, it helps in identifying which features contain security events that should be used in the design of the model. On the other hand, invaluable features that contain a limited number of security events and should be omitted from datasets are also located. Moreover, the classification decision of the model will be justified based on the values of these features. This helps in troubleshooting the errors caused by the wrong predictions of the model. As it allows security experts to analyse the values of the influencing features that result in a miss-classification. Further tuning of the model parameters and utilised features can be conducted after such information.

\begin{equation}
\phi_{j}(v a l)=\sum_{S \subseteq\left\{x_{1}, \ldots, x_{p}\right\} \backslash\left\{x_{j}\right\}} \frac{|S| !(p-|S|-1) !}{p !}\left(\operatorname{val}\left(S \cup\left\{x_{j}\right\}\right)-\operatorname{val}(S)\right)
\label{eq1}
\end{equation}

The interpretation of the model's decisions through the calculation of each feature contribution will help in uncovering the 'black box' of ML. In this paper, the Shapley values are used to explain the importance of network data features in the detection of network attacks. The Shapley value was invented by Lloyd Shapley in 1953
which is the method of assigning players' payouts based on their contribution to the total game's payout. The theory behind the Shapley value has been adopted in ML, where the 'game' represents the prediction task of a single sample in a dataset. The 'payout' is the actual prediction for one sample minus the average predictions of all samples. The 'players' are the feature values of the samples that collaborate to obtain the 'payout'. Overall, the Shapley value is the weighted average of the respective contribution of a feature value. The Shapley value ($\phi_{j}$) is defined via Equation \ref{eq1}, where $S$ is a subset of features used by the model. $X$ represents a vector of feature values of the dataset sample and $p$ is the number of total features. Each $x_{j}$ is a feature value with $j$ = 1,...,$p$ and $val_x(S)$ is the prediction for feature values in subset $S$.

\begin{equation}
g\left(z^{\prime}\right)=\phi_{0}+\sum_{j=1}^{M} \phi_{j} z_{j}^{\prime}
\label{eq2}
\end{equation}

SHapley Additive exPlanations (SHAP) is a common XAI technique developed by Lundberg and Lee. It is based on an additive feature importance method of calculating the Shapley values, known as KernelSHAP and TreeSHAP. Compared to other XAI methods, SHAP has a strong theoretical foundation and can be used to explain the output of any ML model.  SHAP presents new methods that have shown enhanced performance than alternative techniques. KernelSHAP is a kernel-based calculation approach for Shapley values, inspired by local surrogate models used to explain any ML model. TreeSHAP is used to explain tree-based ML models such as decision trees, random forest, and extra trees by leveraging their internal 'tree' structure to speed up the explanation process. When implemented, SHAP explains the prediction of each data sample $x$ by calculating the importance of each feature based on its contribution to the prediction process. Equation \ref{eq2} defines the explanation as specified by SHAP, where $g$ is the ML model. $z^{\prime} \in\{0,1\}^{M}$ is the coalition vector of features used, $M$ represents the maximum coalition size, and $\phi_j \in R$ is the Shapley value of feature $j$. When using SHAP to determine feature importance, the features with larger Shapley values are more important. SHAP calculates the average of importance per feature across the dataset.

\section{Datasets}
\label{ds}

Network data features that control the information represented by datasets have a great impact on the final quality of the ML model \cite{binbusayyis_vaiyapuri_2019}. These features are required to represent a sufficient amount of security events to aid the model's classification tasks. To reliably evaluate an ML-based NIDS performance, the model needs to be evaluated across multiple datasets using the same feature set. Multiple datasets will aid in evaluating the generalisation ability of the model's detection to different attack types and network environments. Currently, NIDS datasets have different feature sets, which are often completely different from each other. It is important to note that it will be unfeasible to extract all of them when the model is deployed over a practical network. Therefore, using a single feature set across multiple datasets is crucial in the design of the model, increasing the evaluation reliability and the chances of potential deployment. NIDS datasets need to be similar in terms of information represented to facilitate reliable experiments. The information represented by the datasets is determined by the choice of network features that make up the dataset. As dataset authors have applied their domain knowledge when selecting the network features, most available datasets consist of a unique almost exclusive set of features.

Common feature sets enable reliable experimental evaluation of ML models across various datasets and attack types. Currently, there are four NIDS datasets that share a common NetFlow-based feature set. The features are proposed as a standard set to be used across future NIDS datasets. However, the performance of ML models using the NetFlow feature set has never been evaluated with another set of features. Therefore, in this paper, the features designed by the CICFlowMeter tool \cite{lashkari2017characterization} are compared to the standard NetFlow feature set across three datasets. As part of the experiment, the CICFlowMeter tool has been utilised to extract the features from the ToN-IoT and BoT-IoT datasets. The generated data flows from the packet capture (pcap) files have been labelled in a binary- and multi-class manner, using the ground truth events. The generated datasets have been named CIC-ToN-IoT and CIC-BoT-IoT, respectively, and made publicly available for research purposes at \cite{netflow_datasets_2020}. The chosen datasets; NF-CSE-CIC-IDS2018-v2, NF-ToN-IoT-v2, NF-BoT-IoT-v2, CSE-CIC-IDS2018, CIC-ToN-IoT and CIC-BoT-IoT will allow for an evaluation of ML experiments on two common feature sets across multiple network environments and attack types. It will also comprehensively evaluate both feature sets and compare the performances in enabling ML models for intrusion detection.

\begin{itemize}

    \item \textbf{CSE-CIC-IDS2018} \cite{sharafaldin-habibi-lashkari-ghorbani-2018}- A well-known NIDS dataset was released in 2018 in a project involving the Communications Security Establishment (CSE) \& Canadian Institute for Cybersecurity (CIC). The testbed used to emulate the network traffic was set up in an organisational network manner involving multiple departments. Attack types such as brute-force, bot, DoS, DDoS, infiltration, and web attacks were conducted from an external source. The dataset contains 75 features extracted using the CICFlowMeter-v3 tool \cite{lashkari2017characterization}. There are 16,232,943 data flows in total, where 13,484,708 (83.07\%) are benign and 2,748,235 (16.93\%) are attack samples.

    \item \textbf{CIC-ToN-IoT}- A dataset generated as part of this paper, where the feature set of the CICFlowMeter was extracted from the pcap files of the ToN-IoT dataset \cite{fesz-dm97-19}. The CICFlowMeter-v4 tool \cite{lashkari2017characterization} was utilised to extract 83 features. There are 5,351,760 data samples where 2,836,524 (53.00\%) are attacks and 2,515,236 (47.00\%) are benign samples. This dataset has been generated as a part of this paper and made available at \cite{netflow_datasets_2020}.

    \item \textbf{CIC-BoT-IoT}- The CICFlowMeter-v4 \cite{lashkari2017characterization} was used to extract 83 features from the BoT-IoT dataset \cite{DBLP:journals/corr/abs-1811-00701} pcap files. The dataset contains 13,428,602 records in total, containing 13,339,356 (99.34\%) attack samples and 89,246 (0.66\%) benign samples. The attack samples are made up of four attack scenarios inherited from the parent dataset, i.e. DDoS, DoS, reconnaissance, and theft. This dataset has been published as a part of this paper at \cite{netflow_datasets_2020}.

    \item \textbf{NF-CSE-CIC-IDS2018-v2} \cite{sarhan2021towards} - The CSE-CIC-IDS2018 dataset \cite{sharafaldin-habibi-lashkari-ghorbani-2018} have been converted into 43 NetFlow-based features using nProbe \cite{Ntopng2017} to generate NF-CSE-CIC-IDS2018-v2. There are 18,893,708 total number of flows where  2,258,141 (11.95\%) are attack samples and 16,635,567 (88.05\%) are benign ones. There are six attack categories such as bruteforce, bot, DoS, DDoS, infilteration, and web attacks.

    \item \textbf{NF-ToN-IoT-v2} \cite{sarhan2021towards} - An IoT dataset generated based on 43 NetFlow features released in 2021. The features are extracted using nProbe \cite{Ntopng2017} from the pcaps of the original parent (ToN-IoT) dataset \cite{fesz-dm97-19}, generated at the Cyber Range Lab by the Australian Centre for Cyber Security (ACCS). The total number of attack data flows 10,841,027 (63.99\%) and 6,099,469 (36.01\%) are benign dataflows, adding up to a total of 16,940,496 samples. There are nine attack categories known as backdoor, DoS, DDoS, injection, MITM, password, ransomware, scanning, and XSS.

    \item \textbf{NF-BoT-IoT-v2} \cite{sarhan2021towards} - A newly generated IoT dataset based on 43 NetFlow features was released in 2021. The features are extracted using nProbe \cite{Ntopng2017} from the pcaps of the original dataset, known as BoT-IoT, generated at the Cyber Lab of the ACCS \cite{DBLP:journals/corr/abs-1811-00701}. It contains 37,763,497 labelled network data flows, where the majority are attack samples; 37,628,460 (99.64\%) and, 135,037 (0.36\%) are benign. There are four attack categories in the dataset, i.e. DDoS, DoS, reconnaissance, and theft.

\end{itemize}

\section{Evaluation}
\label{results}
During the experiments, Deep Feed Forward (DFF) and Random Forest (RF) classifiers are utilised to classify the network data flows present in the datasets. The aim is to evaluate both feature sets in consideration among the three datasets in the CICFlowMeter format (CSE-CIC-IDS2018, CIC-BoT-IoT, and CIC-ToN-IoT) and their respective datasets in the NetFlow format (NF-CSE-CIC-IDS2018-v2, NF-BoT-IoT-v2, and NF-ToN-IoT-v2). Additionally, the results will be analysed and explained by utilising the SHAP method to identify the key features that contributed to the models' final predictions. To avoid learning bias towards the attack and victim devices, the source/destination IPs and ports are dropped. In addition, the timestamp and flow ID features are dropped as they are unique to each data sample. The min-max scaler is applied to normalise all values between zero and one. This is necessary to avoid increased attention to larger values. Several binary classification metrics are collected such as accuracy,  \textit{F1 Score}, \textit{Detection Rate (DR)}, \textit{False Alarm Rate (FAR)}, \textit{Area Under the Curve (AUC)}, and the \textit{Prediction Time} required to predict a single data sample in microseconds (\textmu s). For a fair evaluation, five cross-validation splits are conducted and the mean results are measured.

\begin{table}[ht] \scriptsize
\centering
\caption{RF classification results}
\label{tab:my-table}
\resizebox{\textwidth}{!}{%
\begin{tabular}{l|r|r|r|r|r|r|}
\cline{2-7}
 &
  \multicolumn{1}{l|}{\textbf{Accuracy}} &
  \multicolumn{1}{l|}{\textbf{F1 Score}} &
  \multicolumn{1}{l|}{\textbf{DR}} &
  \multicolumn{1}{l|}{\textbf{FAR}} &
  \multicolumn{1}{l|}{\textbf{AUC}} &
  \multicolumn{1}{l|}{\textbf{Prediction Time}} \\ \hline

\multicolumn{1}{|l|}{\textbf{NF-CSE-CIC-IDS2018-v2}} & 99.47\% & 0.98 & 96.82\% & 0.17\% & 0.9833 & 20.98\textmu s \\
\multicolumn{1}{|l|}{\textbf{CSE-CIC-IDS2018}}       & 98.01\% & 0.93 & 94.75\% & 1.42\% & 0.9667 & 22.39\textmu s \\ \hline
\multicolumn{1}{|l|}{\textbf{NF-ToN-IoT-v2}}         & 99.66\% & 1.00    & 99.80\% & 0.58\% & 0.9961 & 7.57\textmu s  \\
\multicolumn{1}{|l|}{\textbf{CIC-ToN-IoT}}           & 99.33\% & 0.99 & 99.80\% & 1.22\% & 0.9929 & 10.14\textmu s \\ \hline
\multicolumn{1}{|l|}{\textbf{NF-BoT-IoT-v2}}         & 100.00\%   & 1.00    & 100.00\%   & 0.25\% & 0.9988 & 3.60\textmu s  \\
\multicolumn{1}{|l|}{\textbf{CIC-BoT-IoT}}           & 98.24\% & 0.99 & 98.24\% & 1.53\% & 0.9836 & 7.07\textmu s  \\ \hline
\end{tabular}%
}
\end{table}

For evaluation, DFF and RF classifiers are designed to classify the dataset samples into attack and benign categories. Tables \ref{tab:my-table} and \ref{tab:my-table2} list the attack detection results of the six datasets using the RF and DFF classifiers respectively. Both classifiers have achieved a higher detection accuracy on the NF-CSE-CIC-IDS2018-v2 dataset compared to the CSE-CIC-IDS2018 dataset by increasing the DR and lowering the FAR. Resulting in an increase of the F1 score from 0.93 to 0.98 and 0.90 to 0.97 using the RF and DFF classifiers respectively.  These affirm the results published in \cite{sarhan2021towards}. The RF classifier achieved very similar detection results on the ToN-IoT and BoT-IoT datasets in both their NetFlow and CICFlowMeter feature sets. The F1 score increases from 0.99 to 1.00 in both datasets. The FAR drops from 1.22\% in CIC-ToN-IoT to 0.58\% in NF-ToN-IoT-v2 and from 1.53\% in CIC-BoT-IoT to 0.25\% in NF-BoT-IoT-v2. It is also noted that the NetFlow features require a significantly lower prediction time compared to the CICFlowMeter features in both datasets. The DFF model achieved a significantly increased DR in the NF-ToN-IoT-v2 dataset of 95.37\% compared to 92.29\% in the CIC-ToN-IoT dataset, which resulted in an F1 Score increase from 0.94 to 0.96. This was also the scenario in the NetFlow features of the NF-BoT-IoT-v2 dataset, which a high DR of 99.54\% is achieved compared to CIC-BoT-IoT of only 95.99\% DR.

\begin{table}[ht]\scriptsize
\centering
\caption{DFF classification results}
\label{tab:my-table2}
\resizebox{\textwidth}{!}{%
\begin{tabular}{l|r|r|r|r|r|r|}
\cline{2-7}
 &
  \multicolumn{1}{l|}{\textbf{Accuracy}} &
  \multicolumn{1}{l|}{\textbf{F1 Score}} &
  \multicolumn{1}{l|}{\textbf{DR}} &
  \multicolumn{1}{l|}{\textbf{FAR}} &
  \multicolumn{1}{l|}{\textbf{AUC}} &
  \multicolumn{1}{l|}{\textbf{Prediction Time}} \\ \hline

\multicolumn{1}{|l|}{\textbf{NF-CSE-CIC-IDS2018-v2}} & 99.24\%             & 0.97              & 94.67\%       & 0.14\%         & 0.9831       & 8.23\textmu s                     \\
\multicolumn{1}{|l|}{\textbf{CSE-CIC-IDS2018}} & 97.05\% & 0.90  & 85.71\% & 0.96\% & 0.9502 & 8.31\textmu s \\ \hline
\multicolumn{1}{|l|}{\textbf{NF-ToN-IoT-v2}}   & 94.74\% & 0.96 & 95.27\% & 6.08\% & 0.9843 & 8.12\textmu s \\
\multicolumn{1}{|l|}{\textbf{CIC-ToN-IoT}}     & 93.80\%  & 0.94 & 92.29\% & 4.49\% & 0.9782 & 7.26\textmu s \\ \hline
\multicolumn{1}{|l|}{\textbf{NF-BoT-IoT-v2}}   & 99.54\% & 1.00    & 99.54\% & 0.20\%  & 0.9996 & 8.37\textmu s \\
\multicolumn{1}{|l|}{\textbf{CIC-BoT-IoT}}     & 96.01\% & 0.98 & 95.99\% & 1.20\%  & 0.9907 & 9.88\textmu s \\ \hline
\end{tabular}%
}
\end{table}

The lower prediction time required by the NetFlow features compared to the CICFlowMeter features can be explained by the smaller total number of features that make up the dataset. The higher DR of attacks and lower FAR indicates that the NetFlow-based features contain more amount or better quality of security events that aid the ML models in efficient network intrusion detection. Overall, the constantly higher achieved detection accuracies demonstrate that the proposed NetFlow feature set in \cite{sarhan2021towards} is in a better capacity to assist ML models in identifying the attacks present in all three datasets. Figure \ref{tt} visually presents the benefits of using the NetFlow feature set in attack detection scenarios compared to the CICFlowMeter feature set. The results are grouped by Figures \ref{uu} and \ref{u} based on the DFF and RF classifiers respectively. The F1 score is plotted on the y-axis and the datasets on the y-axis, the figures show that the NetFlow features constantly achieve a higher F1 score across the three datasets compared to the CICFlowMeter features.

\begin{figure*}[ht]
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=7cm, height=4cm]{Files/rf.png}
  \caption{RF}
  \label{uu}
\end{subfigure}
\hfill
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=7cm, height=4cm]{Files/dnn.png}
  \caption{DFF}
  \label{u}
\end{subfigure}
\caption{Classification performance of two feature sets across three NIDS datasets}
\label{tt}
\end{figure*}

The continuous improvement of the ML models' detection accuracy, when trained and evaluated using the NetFlow feature set over the CICFlowMeter features, demonstrates the advantages of standardising it in the NIDS community. The NetFlow feature set enables two ML models; DFF and RF, following deep and shallow learning structures, to achieve a reliable performance over three datasets designed over different network environments and include a large number of different attack types. The CICFlowMeter results are slightly less superior in the BoT-IoT and ToN-IoT datasets and less efficient to a certain extent in the detection of attacks in the CSE-CIC-IDS2018 dataset. Moreover, another advantage of the NetFlow feature set is it contains a lower number of features of 43 compared to the CICFlowMeter of 83 features. A lower number of features will aid in an enhanced operation of extraction, analysis, and storage of features over an operational network. Finally, the NetFlow features are naturally present in network packet headers that do not require additional tasks to collect, unlike the CICFlowMeter features that include statistically measured features based on sum, maximum, minimum, standard deviation and average calculations, which might be unfeasible to generate in a live high-speed network.

\section{Explaination}
\label{SHAP}
In this section, the classification results listed above will be zoomed in and explained by utilising the SHAP technique. SHAP is used to determine the features that contribute towards the final prediction, and hence the classification performance of the ML models. The Shapley value defines the amount of contribution of a feature value per data sample. The mean Shapley value is the average of the Shapley values across all test samples. A higher mean Shapley value indicates a stronger influence of the feature value over the final prediction of the model. Hence, a more important feature to analyse and investigate is indicated by a larger Shapley value.  The top 20 features are displayed in Figures \ref{cic nf}-\ref{bot cic}. The features (y-axis) are ranked based on their mean Shapley values (x-axis) across the whole test data samples used to evaluate the ML models. Each figure presents the mean Shapley values as determined by the KernelSHAP and TreeSHAP methods for the DFF and RF models respectively. For efficient comparison, the mean Shapley values have been normalised to a scale from 0 to 1. The results will be analysed separately for each dataset due to a similar feature set and attack type.

\begin{figure*}[h!]
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=7cm, height=5.5cm]{Files/cic_cic_rf.png}
  \caption{RF}
  \label{1}
\end{subfigure}
\hfill
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=7cm, height=5.5cm]{Files/cic_cic_dnn.png}
  \caption{DFF}
  \label{11}
\end{subfigure}
\caption{SHAP top 20 features of CSE-CIC-IDS2018}
\label{cic cic}
\end{figure*}

In Figure \ref{cic cic}, the features of the CSE-CIC-IDS2018 dataset have been ranked by their respective mean Shapley values. Figure \ref{1} and \ref{11} explain the RF and DFF classifiers respectively. Using the RF classifier, features containing security events relating to the forward direction of the flow make up the top four features influencing the model's decision. In particular, the \textit{'Fwd Seg Size Min'} feature has almost double the influence of any other feature. The RF classifier seems to assign a relatively distributed weight to features, causing the gradual decrease of Shapley values. However, the DFF classifier, as conducted by its natural operations, it assigns heavier weights to features based on their importance, hence a steep decrease of Shapley values is noticed. Causing the top ten features to have almost all influence on the model's decision. The top two features present the number of forward and backward packets per second, followed by the \textit{'Fwd Seg Size Min'} feature which is the dominating feature in influencing the RF classifier.

\begin{figure*}[h!]
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=7cm, height=5.5cm]{Files/cic_nf_rf.png}
  \caption{RF}
  \label{2}
\end{subfigure}
\hfill
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=7cm, height=5.5cm]{Files/cic_nf_dnn.png}
  \caption{DFF}
  \label{22}
\end{subfigure}
\caption{SHAP top 20 features of NF-CSE-CIC-IDS2018-v2}
\label{cic nf}
\end{figure*}

The attack detection performance of the NF-CSE-CIC-IDS2018-v2 dataset using the RF and DFF classifiers are explained in Figures \ref{2} and \ref{22} respectively. There are 13 common features in the top 20 influencing features for both classifiers. This indicates that they contain key security events that can be utilised in the detection of attacks present in the NF-CSE-CIC-IDS2018-v2 dataset. The common features mainly include Transmission Control Protocol (TCP)- and Time To Live (TTL)-based features. The \textit{'TCP\_WIN\_MAX\_OUT'} is the first and second most influencing feature in the DFF and RF classifiers respectively, where it presents the maximum TCP window from the destination to the source host. Overall, the DFF nature of assigning weights to features can be seen in the steep decrease of Shapley values leading the top 10 features to have most of the influence over the classifier's decision.

\begin{figure*}[h!]
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=7cm, height=5.5cm]{Files/ton_cic_rf.png}
  \caption{RF}
  \label{3}
\end{subfigure}
\hfill
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=7cm, height=5.5cm]{Files/ton_cic_dnn.png}
  \caption{DFF}
  \label{33}
\end{subfigure}
\caption{SHAP top 20 features of CIC-ToN-IoT}
\label{ton cic}
\end{figure*}

In Figure \ref{ton cic}, the impact of the features of the CIC-ToN-IoT is explained by their respective mean Shapley values. The RF classifier has determined that the \textit{'Idle Mean, Min, and Max'} as the key features, influencing more than 50\% of the model's final predictions. The features present the average, minimum, and maximum time the flow was idle before becoming active. However, the DFF failed to fully utilise neither the idle-based features in the attack detection stage. Where there are only two idle-based features in the top 20 list. This can explain the lower performance of the DFF classifier compared to the RF classifier. Further analysis and attention to the values of the idle-based features is required.

\begin{figure*}[h!]
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=7cm, height=5.5cm]{Files/ton_nf_rf.png}
  \caption{RF}
  \label{4}
\end{subfigure}
\hfill
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=7cm, height=5.5cm]{Files/ton_nf_dnn.png}
  \caption{DFF}
  \label{44}
\end{subfigure}
\caption{SHAP top 20 features of NF-ToN-IoT-v2}
\label{ton nf}
\end{figure*}

In Figure \ref{ton nf}, the top six and two features of the NF-ToN-IoT-v2 dataset generate more than 50\% of the total impact on the RF and DFF classifiers respectively. In Figure \ref{4}, the \textit{'LONGEST and SHORTEST\_FLOW\_PKT} features present the top first and fifth influencing features to the RF classifier. They indicate the longest and shortest packet in terms of bytes transmitted in the traffic flow. The DFF classifier utilises the values of the protocol- and TCP-based features as the top four features to classify network attacks in the NF-ToN-IoT-v2 dataset. Out of the top 20 features affecting the RF and DFF classifiers respectively, there are 17 common features, this indicates that they withhold key security events to aid the ML model detection performance.

\begin{figure*}[h!]
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=7cm, height=5.5cm]{Files/bot_cic_rf.png}
  \caption{RF}
  \label{5}
\end{subfigure}
\hfill
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=7cm, height=5.5cm]{Files/bot_cic_dnn.png}
  \caption{DFF}
  \label{55}
\end{subfigure}
\caption{SHAP top 20 features of CIC-BoT-IoT}
\label{bot cic}
\end{figure*}

In Figure \ref{bot cic}, the Shapley values of the CIC-BoT-IoT dataset features are calculated based on the RF and DFF classifiers.  The forward directional-based security events are presented in four out of the top five features that impact the RF classifier decisions. The Information Access Technology (IAT)-based features account for nine out of the top 20 features that aid the RF classifier to detect attacks present in the CIC-BoT-IoT dataset. In particular, the \textit{'Fwd IAT Min and Mean'} features are the top two, representing the minimum and average time between two packets sent in the forward direction. Followed by the \textit{'Fwd and Bwd Header Length'} features that present the total number of bytes used for the header in the forward and backward directions. The DFF classifier likewise utilised the IAT-based features that made up eight features out of the top 20. However, the \textit{'SYN Flag Count'} and \textit{'Protocol'} features had mainly the largest influence over the model's decision.

\begin{figure*}[h!]
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=7cm, height=5.5cm]{Files/bot_nf_rf.png}
  \caption{RF}
  \label{6}
\end{subfigure}
\hfill
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=7cm, height=5.5cm]{Files/bot_nf_dnn.png}
  \caption{DFF}
  \label{66}
\end{subfigure}
\caption{SHAP top 20 features of NF-BoT-IoT-v2}
\label{bot nf}
\end{figure*}

In Figure \ref{bot nf}, the RF classifier has a slightly decreasing Shapley value as shown in Figure \ref{6}. The top three features are level 7 protocol, maximum, and minimum TTL. The feature weight allocation can be highly noted using the DFF classifier where one feature (\textit{FLOW\_DURATION\_MILLISECONDS}) dominate the model's decision. The rest of the features have a Shapley value smaller than 0.30, with the most being insignificant to the final predictions. There are 14 common features amongst the top 20 features as identified by SHAP, when implemented to explain the RF and DFF classifiers' decisions.

Overall, a global interpretation has been conducted by analysing the Shapley values of six NIDS datasets. The datasets sharing two common feature sets (CICFlowMeter and NetFlow) have been utilised using two ML classifiers (RF and DFF). The results indicate that some features have contributed more towards the final ML predictions than others. Some features have been vastly insignificant in the classification's process. This relates to the amount and quality of security events that these features present relating to the detection of attack types in the dataset. Therefore, the amount of each feature's contribution is different based on the datasets. However, there are some common findings across datasets, such as the RF classifier being heavily influenced by forward directional-based features in the CICFlowMeter feature set.

\section{Conclusion}
ML-based NIDS have achieved outstanding attack detection performance in the research community. However, the number of operational deployments has been lacking. The limited evaluation of a common feature set across multiple datasets and the missing explanation of classification results contribute to the failed translation of research into real-world deployments. In this paper, the proposed NetFlow-based feature set has been evaluated and compared with the feature set designed by the CICFlowMeter. The evaluation has been conducted over three datasets (CSE-CIC-IDS2018, ToN-IoT, and BoT-IoT) using two ML classifiers (RF and DFF). Two datasets, CIC-BoT-IoT and CIC-ToN-IoT, have been generated and published to conduct the experiments. This kind of reliable comparison demonstrates the importance and necessity of having common feature sets across NIDS datasets, such as evaluating the generalisability of ML models' performance across different network environments and attack scenarios. The classification results generated by the ML models indicate a constant superiority of the NetFlow feature across the three NIDS datasets. Where both the DFF and RF classifiers achieved a higher attack detection accuracy in a lower prediction time. In addition, the SHAP method is used to explain the prediction results of the ML models by measuring the feature importance. The key features influencing the predictions of the models have been identified for each dataset.

\printbibliography

\end{document}