text:
\subsection{Levels of Abstraction}
The CWE weaknesses model is composed of four layers of abstraction: Pillar, Class, Base, and Variant. There is also the notion of a Compound Element. The abstraction reflects to what extent issues are being described in terms of five dimensions: behavior, property, technology, language, and resource. Variant weaknesses are at the most specific level of abstraction; they describe at least three dimensions. Base weaknesses are more abstract than variants and more specific than classes; they describe two to three dimensions. Class weaknesses are very abstract; they describe one to two dimensions, typically not specific about any language or technology. Pillar weaknesses are the highest level of abstraction. A Compound Element associates two or more interacting or co-occurring CWEs \cite{MitreGlossary}.

\subsection{Ranking Lists}
It contains two pillars (\cwelink{682} and \cwelink{697}) and 36 classes; as well as 81 bases, three variants (\cwelink{415}, \cwelink{416}, and \cwelink{401}), and  two compounds (\cwelink{352} and \cwelink{384}).
MITRE Top 25 \cite{MitreTop25} ranks CWE items across all the layers of  abstraction from view \cwelink{1003}, where some of  these items have child-parent relationships. It lists seven classes, 16 bases, one variant, and one compound; and there is separate counting for children's occurrences in CVEs and the occurrences of their parent class CWE. For example, the count for the top class \cwelink{119} (rank 1, count 1048) does not include the counts of its children \cwelink{125} (rank 5, count 678) and \cwelink{787} (rank 12, count. 473). Analogously, the count for the class \cwelink{287} (rank 13, count 299) does not include the counts of its children base \cwelink{798} (rank 19, count 91) and base \cwelink{295} (rank 25, count 77). This kind of raking is rather difficult to interpret. Counting and comparing only items that are at the same level of abstraction would be more reasonable.

The NIST Top 20 lists rank view \cwelink{1003} using the mitigated MDSE' equation while separating pillars and classes from basses, variants, and compounds (Tables  \ref{tab:Top20Classes} \& \ref{tab:Top20Bases}).The Top 20 Classes comprise 19 class CWEs and the pillar \cwelink{682} (rank 9) -- see Table \ref{tab:Top20Classes}. Only three CVEs are directly described with the pillar but it is in the list appears in the list because there is a set of severe CVEs described with its children (see subsection "Next Most Dangerous CWEs"). The Top 20 Bases comprise 17 bases, the variants \cwelink{416} (rank 7) and \cwelink{415} (rank 14), and the compound \cwelink{352} (rank 10) --see Table \ref{tab:Top20Bases}. Each list properly compares items of the same kind, and each CWE from Top 20 Bases is a child of a CWE from Top 20 Classes. However, not all classes follow the same order as their children. One interesting example is \cwelink{74}, which we discuss in section "The Most Dangerous CWEs -- Injection vs Memory Errors". The only kind of discrepancies left relate the impossibility to inverse the propagation, as it is a surjective function. So, CVEs described by class CWEs do not get reflected in the Top20 Bases list.
\subsection{Ranking Reordered}
There are two main reasons for the reordering of the items in the Top 20 lists (Tables  \ref{tab:Top20Classes} \& \ref{tab:Top20Bases}) compared to the items in Top 25 \cite{MitreTop25}: (1) the Top 25 list is biased towards the frequency of a CWE occurring in CVEs; (2) the Top 20 Classes list counts all basses and classes that are children of the ranked classes.

Top 20 Classes brings in the class CWEs with different weights compared to those in Top 25, as it is generated via the propagation approach from [Peter paper]. \cwelink{89} (SQL Injection) and \cwelink{502} (Deserialization of Untrusted Data) climb up in the Top 20 Basses due to their highest  severities of 8.89 and 9.01. \cwelink{913} (Improper Control of Dynamically-Managed Code Resources) does not even appear in Top 25, as it has only three direct occurrences in CVE. However, it climbs up to first position in Top 20 Classes due to its highest severity of 8.81 and its 188 propagated occurrences. Its main child contributor is base \cwelink{502} with frequency of 83 and severity of 9.01. The Top 25 \cwelink{119} (Improper Restriction of Operations within the Bounds of a Memory Buffer), while widely used with 2745 propagated occurrences in the CVEs, is quite less severe than \cwelink{913} and drops to rank 2 in Top 20 classes.

All the classes from Top 25 with children from Top 25 appear also in Top 20 Classes: class \cwelink{119} with children base \cwelink{125} and base \cwelink{787}, and class \cwelink{287} with children base \cwelink{798} and base \cwelink{295}. While base \cwelink{120} (a child of \cwelink{119}) comes into Top 20 Bases; the base \cwelink{295} (a child of \cwelink{287}) drops out of Top 20 Bases. Classes that are in Top 25 but do not appear in  Top 20 Classes  are: \cwelink{200} and \cwelink{732}. Basses that are in Top 25 but do not appear in Top 20 bases are \cwelink{79}, \cwelink{476}, \cwelink{772}, \cwelink{426}, \cwelink{295}.

\subsection{The Two Most Dangerous CWEs: Injection vs Memory Errors}
The two most distinctive groups of weaknesses both in the Top 25 list and the two Top 20 lists are injection and memory errors. However, the use of mitigated MSDE' and the split in Top 20 Basses (Table \ref{tab:Top20Bases}) and Top 20 Classes (Table \ref{tab:Top20Classes}) considerably reorders them, as well as brings new CWEs and drops some CWEs.

Injection is the most dangerous type of weakness, represented by \cwelink{89} (SQL Injection), \cwelink{78} (OS Command Injection), \cwelink{94} (Code Injection), and \cwelink{611} (Improper Restriction of XML External Entity Reference), with ranks 1, 4, 6, and 17 respectively (see Table \ref{tab:Top20Bases}). An injection happens when a command string gets assembled partially from input with language-specific special elements and parsed into an invalid construct \cite{bojanova2016bugs}. The invalid constructs for the four most dangerous injection CWEs are respectively SQL Query, OS Command, Code, XML Entity. Top 25 also contains these four CWEs, however the rankings of the first two are 6 and 11 due to the lower frequencies of 397 and 217. The unbiased accounting of their high severity scores of 8.86 and 8.56 moved them several positions up in the Top 20 Bases. In addition, the second ranked in Top 25 \cwelink{79} (Cross-site Scripting), which invalid construct is a generated Web Page, drops out of Top 20 Bases, as although it has the highest frequency of 1571, its severity score of 5.83 is relatively low. The Top 20 Classes brings up also \cwelink{116} (Improper Encoding or Escaping of Output) and CWE 74 (Injection), ranked 14 and 16 (see Table \ref{tab:Top20Classes}). The reason for that is \cwelink{116} is a typical cause of injection and \cwelink{74} is the parent of \cwelink{78}, \cwelink{89}, and \cwelink{94}. Interestingly, the class \cwelink{74} has rank 16 among classes, while its children base \cwelink{89}, \cwelink{78}, and \cwelink{94} are ranked 1, 4, 6 among bases. The frequencies of 2455 for \cwelink{74}, 384 for \cwelink{89}, 194 for \cwelink{78}, and 100 for \cwelink{94}, leave 1777 injection CVEs that are described with CWEs that are either very infrequent or nonsevere. These are \cwelink{79} (Cross-site Scripting) with the low severity of 5.83, \cwelink{88} (Argument Injection) with the low frequency of 6, and \cwelink{91} (XML Injection) with the low frequency of 16. Being not too dangerous they bring the class \cwelink{74} down to rank 16.That same  base \cwelink{79} is ranked 2nd in Top 25 due to the frequency biasing.

The most dangerous memory weaknesses that compete with injection are \cwelink{787} (Out-of-bounds Write) and \cwelink{120} (Classic Buffer Overflow) with ranks 3 and 5 -- see Table \ref{tab:Top20Bases}. Both of them are new to Top 20 Bases compared to Top 25, due to the correction of the frequency bias towards proper reflection of their severity scores of 8.34 and 8.55. Compared to the most dangerous injection weaknesses again while \cwelink{787} is more frequent than \cwelink{89} -- 423 vs 384,-- the severity scores of 8.34 vs 8.89  pushes SQL Injection to the top of the list. The variants \cwelink{416} (Use After Free) and \cwelink{415} (Double Free) with ranks 9 and 14 are peers and main contributing children to \cwelink{672} (Operation on a Resource after Expiration or Release) with rank 4 in Top 20 Classes. They also contribute as grandchildren to class \cwelink{119} through their parent class \cwelink{825} (Expired Pointer Dereference), which is not in view \cwelink{1003}. \cwelink{787}, \cwelink{120}, and \cwelink{125} (Out-of-bounds Read) are the bases contributing to the class \cwelink{119} (Improper Restriction of Operations within the Bounds of a Memory Buffer) with rank 2 in Top 20 Classes. Lastly, class \cwelink{400} (Uncontrolled Resource Consumption) with rank 15 is mostly directly assigned to CVEs (see Table \ref{tab:Top20Bases}).

The majority of the base CWEs in this group  are memory use weakness -- \cwelink{120}, \cwelink{125}, and \cwelink{787} are buffer overflow (out of bounds read or write); \cwelink{416} is use after free (use of deallocate memory through a dangling pointer). Another base CWE is a memory deallocation weakness: \cwelink{415} is double free (deallocate of already deallocated memory). Class \cwelink{119} is a general memory corruption weakness, which includes buffer overflow, use after free and double free. Yet, another class CWE from this group is a memory allocation weakness: \cwelink{400} is memory overflow (stack/heap  exhaustion) \cite{BFMem}.

Compared to Top 25, the mitigated MDSE' brings up several injection weaknesses with much higher severity than that of any memory weaknesses. The related CVEs analysis confirms that the injection CVE are easier to exploit and with higher impact. An injection directly leads to arbitrary command, code, or script execution.Once a SQL injection is in place, there is no need of additional sophisticated attack crafting or use of glitches in the system. While It takes considerable extra effort for an attacker to turn a buffer overflow into an arbitrary code execution. The attacker also needs exceptional skills in applying spraying memory techniques.and more [SoK: War on Memory paper]. The possible damage from a SQL injection is also very high. It may expose huge amounts of structured data, which is always more valuable than raw data. Well formed structured data is easy to read, sort, search, and make sense of it. An attacker could modify a database -- insert, update, delete data, execute admin operations, recover file content, and even issue OS commands to the operating system \cite{OWASPInjection}.

\subsection{Next Most Dangerous CWEs}
Next groups of weaknesses in Top 20 Bases and Classes relate to file input and upload, authentication,  randomization, cryptography, arithmetics and conversion, and input validation.
The most dangerous file input and upload weaknesses are base \cwelink{502} (Deserialization of Untrusted Data) and base \cwelink{434} (Unrestricted Upload of File with Dangerous Type) with ranks 2 and 8, respectively. They are the main contributors to class \cwelink{913} and class \cwelink{669} with ranks 1 and 3, respectively.

Top 20 Classes brings up also the randomization weaknesses class \cwelink{330} (Use of Insufficiently Random Values) with rank 5, which is mostly directly assigned to CVEs.
The most dangerous authentication weakness is base \cwelink{798} (Use of Hard-coded Credentials) with rank 7; which is one of the contributors to the class \cwelink{287} (Improper Authentication) with the same rank 7 in Top 20 Classes.

The most dangerous cryptography weakness is base \cwelink{352} (Cross-Site Request Forgery -- CSRF)with rank 10, which relates to bugs in data verification. Top 20 Classes list also class \cwelink{326} (Inadequate Encryption Strength) with rank 18, which is directly assigned to 35 CVEs with severity 7.24.

The most dangerous arithmetics and conversion weaknesses are \cwelink{190} (Integer Overflow or Wraparound) and \cwelink{191} (Integer Underflow) with rank13 and 20, which are the main contributors to the pillar \cwelink{682} (Incorrect Calculation) with rank 9 in Top 20 Classes. Note that only three CVEs are directly described with the pillar \cwelink{682}, but it is in the Top 20 Classes list due to the set of severe CVEs described with its children bases \cwelink{131} (Incorrect Calculation of Buffer Size), \cwelink{190} (Integer Overflow or Wraparound), and \cwelink{191} (Integer Underflow -- Wrap or Wraparound) with severities 9.8, 7.95, and 8.47, respectively. \cwelink{131} though does not appear in the Top 20 Basses, as it has only one occurrence in the CVEs.
The most dangerous input validation weakness is \cwelink{129} (Improper Validation of Array Index) with rank 16.

\subsection{Dependent on CWE, CVE, and NVD}
Both Top 25 and Top 20 rankings heavily depend on how NVD assigns CWEs to particular CVEs. The CWE selection is restricted to view \cwelink{1003}. The lack of enough information about a CVE or the lack of a more specific CWE may lead to the CVE described with the closest class CWE or even with a pillar CWE. For example, It makes sense for class \cwelink{119} to be used for the memory corruption CVE-2019-7098, as there is not much  information, no code, no details about it -- it could be any memory use error or a double free. .However, there is enough information about  the use after free CVE-2019-15554 and it still gets described  with class \cwelink{119}, as there is no appropriate base CWE. A close base CWE is \cwelink{416} (Use After Free), but it does not really reflect memory safe languages like Rust. It is also possible for a class CWE to be assigned to a CVE even when a specific  base CWE is available. For example, the stack buffer overflow write CVE-2019-14363 is assigned class \cwelink{119}, although there is plenty of information and appropriate bases \cwelink{121} and \cwelink{120}.

\section{Appendix: Supplementary Table from Outside Source}
\input{MITRETop25CWE}

\end{appendix}

\subsection{Levels of Abstraction}

\subsection{High Level Summaries}
View 1003 contains two pillars (\cwelink{682} and \cwelink{697}) and 36 classes, as well as 81 bases, three variants (\cwelink{415}, \cwelink{416}, and \cwelink{401}), and  two compounds (\cwelink{352} and \cwelink{384}).

The MDSE Top 25 \cite{MitreTop25} ranks CWE items across all the layers of abstraction from view \cwelink{1003}. The list has seven classes, 16 bases, one variant, and one compound. Interestingly, some of these top CWEs have child-parent relationships among themselves.

A simple inspection of the list shows how parent CWEs do not receive CVE counts from their children.
For example, the count for the top class \cwelink{119} (rank 1, count 1048) does not include the counts of its children \cwelink{125} (rank 5, count 678) and \cwelink{787} (rank 12, count 473). Analogously, the count for the class \cwelink{287} (rank 13, count 299) does not include the counts of its children base \cwelink{798} (rank 19, count 91) and base \cwelink{295} (rank 25, count 77).

Our MSSW class list is comprised of 19 class CWEs and the pillar \cwelink{682} (rank 9) – see Table \ref{tab:Top20Classes}. Only three CVEs are directly described with the pillar, but it appears in the list because there is a set of severe CVEs described with its children (see subsection \ref{NextMostDangerous}). Our MSSW base list is comprised of 17 bases, the variants \cwelink{416} (rank 7) and \cwelink{415} (rank 14), and the compound \cwelink{352} (rank 10) – see Table \ref{tab:Top20Bases}. Each of the two lists properly compare items of the same kind. Interestingly but not surprisingly, each CWE from the base list is a child of a CWE from class list. However, the ordering of these parent-child pairs are not necessarily preserved between the two lists.

\subsection{Set Differences}
There are differences in the set of CWEs covered by our top 20 MSSW class and base lists and the MDSE list. The pillars/classes in the MDSE list that do not appear in the class list are: \cwelink{200} and \cwelink{732}. The bases/variants/compounds in the MDSE list that do not appear in the base list are: \cwelink{79}, \cwelink{476}, \cwelink{772}, \cwelink{426}, and \cwelink{295}. The base list contains base \cwelink{120} (a child of \cwelink{119}), which does not appear in the MDSE list.

Note that the two classes from the MDSE list with children in the same list are also in the class list (emphasizing their importance): class \cwelink{119} with children base \cwelink{125} and base \cwelink{787}, and class \cwelink{287} with children base \cwelink{798} and base \cwelink{295}.

\subsection{Reordered Rankings}
The relative orderings in the MDSE list often do not match the orderings in the MSSW class and base lists.
There are some notable reorderings. \cwelink{89} (Structured Query Language (SQL) Injection) and \cwelink{502} (Deserialization of Untrusted Data) climb up in the base list due to their highest severities of 8.89 and 9.01. \cwelink{913} (Improper Control of Dynamically-Managed Code Resources) does not even appear in the MDSE Top 25 list, as it has only three direct occurrences in the CVEs. However, it climbs up to first position in the class list due to its highest severity of 8.81 and its 188 propagated occurrences. Its main child contributor is base \cwelink{502} with frequency of 83 and severity of 9.01. \cwelink{119} (Improper Restriction of Operations within the Bounds of a Memory Buffer) in the MDSE list, while widely used with 2745 propagated occurrences in the CVEs, is quite less severe than \cwelink{913} and drops to rank 2 in the MSSW class list.

\subsection{The Two Most Dangerous CWEs: Injection vs. Memory Errors}
\label{2MostDangerous}
The two most distinctive groups of weaknesses both in the MDSE Top 25 list and the two MSSW Top 20 lists are injection and memory errors. However, the use of the MSSW equation and the split into  class and base lists considerably reorders these two groups, as well as brings in new CWEs and drops some CWEs.

\subsubsection{Injection Weaknesses}
Injection is the most dangerous type of weakness, represented by  bases) \cwelink{89} (SQL Injection),  \cwelink{502} (Deserialization of Untrusted Data), \cwelink{78} (OS Command Injection), \cwelink{94} (Code Injection), and \cwelink{611} (Improper Restriction of Extensible Markup Language (XML) External Entity Reference), with ranks 1, 2, 4, 6, and 17 respectively in the base list (see Table \ref{tab:Top20Bases}). The MDSE list also contains these five CWEs, however the rankings of the first three are 6, 16, and 11 due to their lower frequencies of 397, 85, and 217. The MSSW inclusion of their high severity scores of 8.89, 9.01,  and 8.58 moved them several positions up in the base list. Note that \cwelink{502} covers Object Injection.

Also of importance is that the second ranked in the MDSE list \cwelink{79} (Cross-site Scripting), is not in our MSSW base list. Although it has the highest frequency of 1571, its severity score of 5.83 is relatively low.

The MSSW class list includes \cwelink{913} (Improper Control of Dynamically-Managed Code Resources), \cwelink{116} (Improper Encoding or Escaping of Output), and \cwelink{74} (Injection), ranked 1, 14, and 16 (see Table \ref{tab:Top20Classes}). The reason for that is  \cwelink{913} is the parent of \cwelink{502}, \cwelink{116} is a typical cause of injection and \cwelink{74} is the parent of \cwelink{78}, \cwelink{89}, and \cwelink{94}. Interestingly, the class \cwelink{74} has rank 16 among classes, while its children bases \cwelink{89}, \cwelink{78}, and \cwelink{94} are ranked 1, 4, 6 among bases. The frequencies of 2455 for \cwelink{74}, 384 for \cwelink{89}, 194 for \cwelink{78}, and 100 for \cwelink{94}, leave 1777 injection CVEs that are described with CWEs that are either very infrequent or not severe. These are bases \cwelink{79} (Cross-site Scripting) with the low severity of 5.83, \cwelink{88} (Argument Injection) with the low frequency of 6, and \cwelink{91} (XML Injection) with the low frequency of 16. Being not too dangerous they bring the class \cwelink{74} down to rank 16. That same base \cwelink{79}, not included in the MSSW base list, is ranked 2nd in the MDSE list due to the frequency biasing.

\subsubsection{Memory Weaknesses}

The most dangerous memory weaknesses are \cwelink{787} (Out-of-bounds Write) and \cwelink{120} (Classic Buffer Overflow) with ranks 3 and 5 – see Table \ref{tab:Top20Bases}. Both of them are included in the base list but not the MDSE list, due to the correction of the frequency bias towards proper inclusion of their severity scores of 8.34 and 8.55.

The other memory weaknesses in the MSSW class and base lists are as follows:
\begin{itemize}
    \item bases \cwelink{125} and \cwelink{787} are buffer overflow (out of bounds read or write)
    \item variant \cwelink{416} is use after free (use of deallocated memory through a dangling pointer)
    \item variant \cwelink{415} is double free (deallocate of already deallocated memory)
    \item class \cwelink{119} is a general memory corruption weakness, which includes buffer overflow, use after free and double free.
    \item class \cwelink{400} is memory overflow (stack/heap  exhaustion) \cite{BFMem}
\end{itemize}

\subsubsection{Injection/Memory Weakness Comparison}
Compared to MDSE, the MSSW equation brings up several injection weaknesses with much higher severity than that of any memory weaknesses. The related CVE analysis confirms that the injection CVE are easier to exploit and with higher impact. An injection directly leads to arbitrary command, code, or script execution. Once a SQL injection is in place, there is no need of additional sophisticated attack crafting or use of glitches in the system. However, it takes considerable extra effort for an attacker to turn a buffer overflow into an arbitrary code execution. He or she would need to have  exceptional skills, such as to apply spraying memory techniques.
The possible damage from an Object injection or from an SQL injection or from is very high. Object injection could lead to remote code execution. An SQL injection may expose huge amounts of structured data, which is proven to be more valuable than raw data. Well formed structured data is easy to read, sort, search, and make sense of it. Via an SQL injection, an attacker could modify a database – insert, update, delete data, execute admin operations, recover file content, and even issue OS commands \cite{OWASPInjection}.

\subsection{Next Most Dangerous CWEs}
\label{NextMostDangerous}
The next most dangerous groups of weaknesses in the MSSW class and base lists relate to file input and upload, authentication,  randomization, cryptography, arithmetics and conversion, and input validation:
\begin{itemize}
    \item randomization – class \cwelink{330} (Use of Insufficiently Random Values) with rank 5 is the class mostly directly assigned to CVEs.
    \item authentication – base \cwelink{798} (Use of Hard-coded Credentials) has rank 7; it is one of the contributors to the class \cwelink{287} (Improper Authentication) with the same rank 7 in the class list.
    \item file upload – base \cwelink{434} (Unrestricted Upload of File with Dangerous Type) has rank 8. It is the main contributors to class \cwelink{669} with rank 3.
    \item cryptography – base \cwelink{352} (Cross-Site Request Forgery) has rank 10, which relates to bugs in data verification. The class list also has class \cwelink{326} (Inadequate Encryption Strength) with rank 18, which is directly assigned to 35 CVEs with severity 7.24.
    \item arithmetics and conversion – base \cwelink{190} (Integer Overflow or Wraparound) and base \cwelink{191} (Integer Underflow) have ranks 13 and 20. They are the primary contributors to pillar \cwelink{682} (Incorrect Calculation) with rank 9. Others in this group on the top lists are bases \cwelink{131} (Incorrect Calculation of Buffer Size), \cwelink{190} (Integer Overflow or Wraparound), and \cwelink{191} (Integer Underflow – Wrap or Wraparound).
    \item input validation - base \cwelink{129} (Improper Validation of Array Index) has rank 16.
\end{itemize}

\subsection{Mapping Dependencies}
Both the MDSE and MSSW rankings heavily depend on how NVD assigns CWEs to particular CVEs. The CWE selection is restricted to view \cwelink{1003}. Insufficient information about a CVE or an insufficiently specific CWE may lead to the use of the closest matching CWE class or pillar to describe the CVE. For example, it makes sense for class \cwelink{119} to be used for the memory corruption CVE-2019-7098, as there is not much information (no code and no details) – it could be any memory use error or a double free. However, there does exist enough information about the use after free CVE-2019-15554, but it is still mapped to class \cwelink{119} because there exists no appropriate base CWE. A close base CWE is \cwelink{416} (Use After Free), but it does not really reflect memory safe languages like Rust. It is also possible for a class CWE to be assigned to a CVE even when a specific base CWE is available. For example, the stack buffer overflow write CVE-2019-14363 is assigned class \cwelink{119}, although there is plenty of information to map it more specifically to bases \cwelink{121} and \cwelink{120}.

\section{Introduction}
\label{sec.Intro}

In 2019, there were over 17\,000 documented software vulnerabilities \cite{NVD} that enable malicious activity. While many are discovered, they map to a relatively small set of underlying weakness types. We posit that if the most significant of these types can be identified, developers of programming languages, software, and security tools can focus on preventing them and thus over time diminish the quantity and severity of newly discovered vulnerabilities.

In this work, we provide a metric to calculate the most significant security weaknesses (MSSW) in software systems. We define a `significant' weakness as one that is both frequently occurring among the set of publicly published vulnerabilities and results in high severity vulnerabilities (those that are easily exploitable and have high impact).
The set of security weakness types upon which we calculate significance comes from
the Common Weakness Enumeration (CWE) \cite{CWE}.
We also leverage the Common Vulnerabilities and Exposures (CVE) \cite{CVE} repository of publicly announced vulnerabilities, the Common Vulnerability Scoring System (CVSS) \cite{CVSSv3.1} to measure the severity of vulnerabilities, and the National Vulnerability Database (NVD) \cite{NVD} to map the CVEs to both CWEs and CVSS scores.

In the fall of 2019, the CWE community published an equation to calculate the `Top 25 Most Dangerous Software Errors' (MDSE) among the set of CWEs \cite{MitreTop25}. It follows the form of the common security risk matrix combining probability and severity (e.g., \cite{engert1999risk}). The MDSE equation claims to combine `the frequency that a CWE is the root cause of a vulnerability with the projected severity'; the equation description implies that both factors are weighed equally (making no mention of any bias). However, we empirically find that the equation highly biases frequency and almost ignores severity in generating top lists of varying sizes. This is due to the equation multiplying calculated frequency and severity values together though each has has very different distributions. Frequency distributions have a power law like curve, while severity distributions are more uniform. Our mitigation is to create a revised equation, named MSSW, that adjusts the frequency distribution using a double log function to better match it to the severity distribution. We also fix an error in how normalization is done in the MDSE equation.

We next improve upon the data collection approach used by the MDSE equation by leveraging published literature \cite{mell2020}. Lastly, we publish top lists of the most significant CWEs for 2019, provide an analysis of those software security weaknesses, and compare our top lists against previously published lists. It is our hope that our data and methodology will be adopted to focus our collective security resources in reducing the most significant software security weaknesses.

The rest of this work is organized as follows. Section \ref{sec.Background} provides background on CVE, CVSS, CWE, NVD, and the MDSE equation. Section \ref{sec.Limitations} discusses the limitations of the MDSE equation. Section \ref{sec.FixedEquation} presents our MSSW equation that mitigates the previously identified limitations. Section \ref{sec.TopWeaknesses} provides two lists of the most significant CWEs at two different levels of software flaw type abstractions. Section \ref{sec.Evaluation} provides a discussion and analysis of the most significant CWEs identified. Section \ref{sec.RelatedWork} presents related work, Section \ref{sec.FutureWork} discussed possible future research, and Section \ref{sec.Conclusions} concludes.

\section{Background}
\label{sec.Background}

\subsection{Common Vulnerabilities and Exposures}
The CVEs are a large set of publicly disclosed vulnerabilities in widely-used software. They are enumerated with a unique identifier, described, and referenced with external advisories \cite{CVE} \cite{baker1999CVE}.

\subsection{Common Vulnerability Scoring System}
CVSS `provides a way to capture the principal characteristics of a vulnerability and produce a numerical score reflecting its severity' \cite{CVSS}. The CVSS base score reflects the inherent risk of a vulnerability apart from any specific environment.
The base score is composed from two sub-scores that calculate exploitability (how easy it is to use the vulnerability in an attack) and impact (how much damage the vulnerability can cause to an affected component).

The exploitability score is determined by the following:
\begin{itemize}
    \item attack vector: `the context by which vulnerability exploitation is possible',
    \item attack complexity: `the conditions beyond the attacker's control that must exist in order to exploit the vulnerability',
    \item privileges required: `the level of privileges an attacker must possess before successfully exploiting the vulnerability', and
    \item user interaction: a human victim must participate for the vulnerability to be exploited.

\end{itemize}

The impact score is determined by measuring the impact to the confidentiality, integrity, and availability of the affected system. Also included is a scope metric that `captures whether a vulnerability in one vulnerable component impacts resources in components beyond its security scope'. The specifics on these metrics and the details for the three equations can be found in the CVSS version 3.1 specification at \cite{CVSSv3.1}.

\subsection{Common Weakness Enumeration}
\label{CWE}

The Common Weakness Enumeration (CWE) \cite{martin2008CWE} is a `community-developed list of common software security weaknesses'. `It serves as a common language, a measuring stick for software security tools, and as a baseline for weakness identification, mitigation, and prevention efforts' \cite{CWE}. It contains an enumeration, descriptions, and references for 839 software weaknesses that are referred to as CWEs, where each is labelled CWE-\textit{X} with \textit{X} being an integer.

The CWE weaknesses model has four layers of abstraction: pillar, class, base, and variant. There is also the notion of a compound, that associates two or more interacting or co-occurring CWEs \cite{MitreGlossary}. These abstractions reflect to what extent issues are described in terms of five dimensions: behavior, property, technology, language, and resource. Variant weaknesses are at the most specific level of abstraction; they describe at least three dimensions. Base weaknesses are more abstract than variants and more specific than classes; they describe two to three dimensions. Class weaknesses are very abstract; they describe one to two dimensions, typically not specific about any language or technology. Pillar weaknesses are the highest level of abstraction.

There are a set of taxonomies, called views, to help organize the CWEs. Two prominent CWE taxonomies are the `Research Concepts' (view 1000) and `Development Concepts' (view 699). There is also a view 1003 that was made specifically to describe the set of CVEs that contains 124 CWEs. It is called `CWE Weaknesses for Simplified Mapping of Published Vulnerabilities View'.

\subsection{National Vulnerability Database}
The CWE effort uses the National Vulnerability Database (NVD) \cite{NVD} as a repository of data from which to calculate the MDSE scores. The NVD contains all CVEs and for each CVE it provides a CVSS score along with the applicable CWE(s) that describe the weakness(es) enabling the vulnerability. For the empirical work in this paper, we use the complete set of
17\,308
CVEs published by NVD for 2019, that were available as of
2020-03-19.

\subsection{Most Dangerous Software Error Equation}
The MDSE equation is designed to balance the frequency and severity in ranking the CWEs. The frequency is determined by the number of CVEs that map to a given CWE in the time period of study. The severity is determined by the mean CVSS score for the CVEs mapped to a given CWE. The MDSE score for a CWE is produced by multiplying the normalized frequency by the normalized severity and then multiplying by 100. We now describe this metric more formally.

\subsubsection{Metric for Normalized Frequency}
\label{MDSE-Frequency}

Let $I$ designate the set of all CWEs and let $J$ be the set of all CVEs.

For CWE $i\in I$, let $N_i$ be the number of CVEs mapped to $i$, defined as follows:

\begin{equation}
N_i = \sum_{j\in J} e_{ij},
\label{eq:N_i}
\end{equation}
where

\begin{equation}
e_{ij} =
    \begin{cases}

          1, &         \text{if CVE $j$ is mapped to CWE $i$},\\
            0, &         \text{otherwise}.
    \end{cases}
    \label{eq:eij}
\end{equation}

Now let $F_i$ be the normalized frequency for CWE $i$, defined as follows:

\begin{equation}
F_i = \frac{N_i - \min\limits_{i' \in I} (N_{i'})}{\max \limits_{i' \in I} (N_{i'}) - \min\limits_{i' \in I} (N_{i'})}.
\label{eq:F_i}
\end{equation}

\subsubsection{Metric for Normalized Severity}
\label{MDSE-Severity}

Let $J$, $N_i$, and $e_{ij}$ be as defined above in Section \ref{MDSE-Frequency}.
Let $s_j$ be the CVSS base score for CVE $j$. For CWE $i\in I$, let $\overline{S_i}$ be the mean CVSS score, defined as follows:

\begin{equation}
\overline{S_i} =
\frac{\sum_{j\in J}s_j e_{ij}}{N_i}.
\label{eq:S_i_overline}
\end{equation}

Now let $S_i$ be the normalized severity for CWE $i$, defined as follows:

\begin{equation}
S_i = \frac{\overline{S_i} - \min\limits_{j \in J} (s_j)}{\max \limits_{j \in J} (s_j) - \min\limits_{j \in J} (s_j)}.
\label{eq:S_i}
\end{equation}

\subsubsection{Most Dangerous Software Error Metric}
Let $MDSE_i$ be the MDSE score for CWE $i$, defined as follows:

\begin{equation}
MDSE_i=F_i*S_i*100.
\label{eq:MDSE_i}
\end{equation}

\section{Limitations of the Equation}
\label{sec.Limitations}

\begin{figure}
\centerline{\includegraphics[scale=.6]{images/MDSE-Set-Difference.jpg}}
\caption{The Size of the Set Difference between Top Lists from the MDSE Equation Compared to Frequency Top Lists (red bottom line), Severity Top Lists (yellow middle line), and the Theoretical Maximum (blue top line)}
\label{fig:MDSE-Set-Difference}
\end{figure}

The MDSE equation was designed to and appears to combine both frequency and severity in determining the individual scores used to rank the CWEs. The frequency component is calculated in equation \ref{eq:F_i} and the severity component is calculated in equation \ref{eq:S_i}; both are brought together in equal proportions in equation \ref{eq:MDSE_i} to create the MDSE score. And both the severity and frequency are normalized in equations \ref{eq:F_i} and \ref{eq:S_i} to ensure that their scales match for the multiplication in equation \ref{eq:MDSE_i}.

However, we empirically find that the MDSE equation strongly biases frequency over severity. To demonstrate this, we calculate MDSE top CWE lists for all possible list sizes. While there exist 839 CWEs, the CVE data used as MDSE input is mapped only to 124 view 1003 CWEs (see section \ref{CWE})\footnote{This is expected as view 1003 was designed to cover the types of vulnerabilities in CVE.
}. Thus the maximum top list size is 124. We also calculate top CWE lists using just the frequency equation \ref{eq:F_i} and then just the severity equation \ref{eq:S_i}. For each CWE top list size, we perform a set difference between the MDSE top list and the frequency top list. We then also do this between the MDSE top list and the severity top list. The size of the set difference between the MDSE top list and the frequency top list (for all possible top list sizes) has a maximum difference of 3. The size of the set difference between the MDSE top list and the severity top list (for all possible top list sizes) has a maximum difference of 23. This is shown graphically in Figure \ref{fig:MDSE-Set-Difference}. The bottom red line represents the set difference using frequency and the yellow middle line represents the set difference using severity. The top blue line shows the maximal possible set difference that could be achieved using the 124 CWEs.

More qualitatively, the red line hovers close to a y-axis value of 0 which means that for all list sizes the top list generated using just frequency is almost identical to the top list generated using the MDSE equation. The middle yellow line being far from the y-axis value of 0 means that for all list sizes the top list generated using just severity is very different from the top list generated using the MDSE equation. Note that the yellow line shows an almost maximal difference for top list sizes of up to 15.

\begin{figure}
\centerline{\includegraphics[scale=.6]{images/Freq-vs-MDSE.jpg}}
\caption{CWEs Chosen (Red Triangles) and Not Chosen (Yellow Circles) for a MDSE Top 20 List Relative to Frequency}
\label{fig:MDSE-vs-Frequency}
\end{figure}

\subsection{Limitation 1: Distribution Differences}
\label{lim1}
The MDSE equation in practice biases frequency over severity, even though its equations treat them equally, because frequency and severity have very different distributions. The frequency distribution has the majority of CWEs at a very low frequency and a few at a very high frequency (somewhat resembling a power law curve). This can be seen in Figure \ref{fig:MDSE-vs-Frequency} by looking at how each CWE maps to the x-axis (note that most of the yellow dots overlap, there are 102 yellow dots and 20 red triangles). The figure shows the MDSE scores for each CWE and shows how (for a top list of size 20) the top scoring chosen CWEs are exactly the most frequent CWEs. This is not unique and occurs for many top lists (e.g., for sizes 11, 13, 15, 16, 20, 21, 32, and 38) as shown when the bottom red line is at 0 in Figure \ref{fig:MDSE-Set-Difference}. The other sizes of top lists produce graphs that are almost identical to that in Figure \ref{fig:MDSE-vs-Frequency}, with at most 3 yellow circles just to the right of the leftmost red triangles representing the chosen CWEs.
\begin{figure}
\centerline{\includegraphics[scale=.6]{images/CVSS-vs-MDSE.jpg}}
\caption{CWEs Chosen (Red Triangles) and Not Chosen (Yellow Circles) for a MDSE Top 20 List Relative to Severity}
\label{fig:MDSE-vs-CVSS}
\end{figure}

The severity distribution is more uniform within a limited range. It can be seen in Figure \ref{fig:MDSE-vs-CVSS} by looking at how the CWEs map to the x-axis. This figure shows how the top MDSE scoring chosen CWEs do not necessarily map to the CWEs with the highest severity. In fact, only 1 of the top 10 most severe CWEs made the MDSE top 20 list (note that many of the yellow circles lay on top of each other).

\subsection{Limitation 2: Normalization Error}
\label{lim2}
Equation \ref{eq:S_i} normalizes $S_i$ based on the maximum and minimum CVSS score found in the set of inputted CVEs. However, this does not lead to the expected and desired normalized distribution from 0 to 1. For our data the range is from .28 to .97, as can be seen from the mappings of the points onto the x-axis in Figure \ref{fig:MDSE-vs-CVSS}. The reason for this is that  $\overline{S_i}$ has a smaller range than the maximum and minimum CVSS score because each $\overline{S_i}$ represents the mean of the CVSS score for the CVEs that map to CWE $i$. This limitation, while of less consequence than the previous, constrains the range of $S_i$ values thus further lessening the influence that severity has in determining a MDSE score.

\section{Mitigated Equation}
\label{sec.FixedEquation}

We mitigate the limitations of the MDSE equation by replacing equations \ref{eq:F_i}, \ref{eq:S_i}, and \ref{eq:MDSE_i} with the five equations that follow:

\begin{equation}
k = \frac{1}{\log _{e} \log _{e} \max\limits_{i \in I} (N_{i}) },
\label{eq:k}
\end{equation}

\begin{equation}
F'_i =
    \begin{cases}
          \log _{e} N_i, & \text{if $N_i$ >= 1},\\
          0, & \text{otherwise},
    \end{cases}
\label{eq:F'_i}
\end{equation}

\begin{equation}
F''_i =
    \begin{cases}
          k\log _{e} F'_i, & \text{if $F'_i$ >= 1},\\
          0, & \text{otherwise},
    \end{cases}
\label{eq:F''_i}
\end{equation}

\begin{equation}
S'_i = \frac{\overline{S_i} - \min\limits_{i' \in I} (\overline{S_{i'}})}{\max \limits_{i' \in I} (\overline{S_{i'}}) - \min\limits_{i' \in I} (\overline{S_{i'}})},
\label{eq:S'_i}
\end{equation}

\begin{equation}
MSSW_i=F''_i*S'_i*100.
\label{eq:MSSW_i}
\end{equation}

\subsection{Explanation of Mitigated Equation}

\begin{figure}
\centerline{\includegraphics[scale=.6]{images/G0Freq-SLog-DLog-Distributions.jpg}}
\caption{Normalized Distributions of Frequency (bottom blue line), Log of Frequency (middle yellow line), and Double Log of Frequency (top red line).}
\label{fig:G0-distributions}
\end{figure}

Equation \ref{eq:F'_i} takes the log of the frequency using the natural log as the base. Equation \ref{eq:F''_i} then takes the log of equation \ref{eq:F'_i}, again using the natural log as a base and multiplies the result by $k$ (from equation \ref{eq:k}). The $k$ coefficient serves the purpose of normalizing the resulting values between 0 and 1 (to match the severity range in equation \ref{eq:S'_i}).

These three equations modify the power law like frequency distribution to make it more linear, thus addressing limitation 1 (from Section \ref{lim1}). This can be seen in Figure \ref{fig:G0-distributions}. Each value on the x-axis represents a particular CWE, ordered from least frequent to most frequent. The lower blue line represents the normalized frequency (i.e., number of CVEs mapped to a particular CWE). Note the slow increase in frequency up to the 100th CWE, followed by a rapid increase terminating in an almost vertical line (i.e. large derivative). This behavior creates large differences between the most frequent CWEs and almost no difference between the lowest CWEs.  If $g(x) = \log f(x)$ its derivative is
\begin{equation}
\frac{dg(x)}{dx} = \frac{1}{f(x)} \frac{df(x)}{dx},
\end{equation}
thus applying a log function over the frequency should minimize differences between the most frequent CWEs.

The middle yellow line represents taking the log of the frequency (equations not shown), which helps linearize but still results in an upwards curve on the right side. Thus, we apply a double log for further linearization (see the top red line). We note that this approach is not pseudo-linear for the most infrequent of CWEs. However, this does not cause problems as our goal is to identify the most significant and any such CWE must have at least a moderate frequency.

Our modified MDSE equation \ref{eq:MSSW_i} then multiplies frequency and severity as in the original MDSE equation, but it multiplies from two distributions that have a similar shape for the part of the functions that are of interest. This enables the MSSW equation to more fairly balance evaluating frequency and severity in scoring and ranking a CWE.

To address limitation 2 from Section \ref{lim2}, equation \ref{eq:S'_i} normalizes the severity using the maximum and minimum mean severity values. This gives the distribution a full 0 to 1 range which is not achieved in the MDSE equation \ref{eq:S_i}.

Equation \ref{eq:MSSW_i} is our final modified MDSE equation. We recommend its use in place of the published MDSE equation.

\subsection{Analysis of Mitigated Equation}

We now conduct three experiments to evaluate the effect of the MSSW equation in making the frequency and severity distributions more similar and in producing top lists with more equal inclusion of both frequency and severity. A fourth experiment involving correlation calculations is provided in Section \ref{sec.TopWeaknesses} (because it includes some variants introduced in that section).

\subsubsection{Risk Map Experiment}

\begin{figure}
\centerline{\includegraphics[scale=.6]{images/MITRE-risk-map.jpg}}
\caption{MDSE Equation Risk Map}
\label{fig:MDSE-Riskmap}
\end{figure}

\begin{figure}
\centerline{\includegraphics[scale=.6]{images/NIST-risk-map.jpg}}
\caption{MSSW Equation Risk Map}
\label{fig:MSSW-Riskmap}
\end{figure}

Figure \ref{fig:MDSE-Riskmap} shows an MDSE risk map for the evaluated CWEs. Each red dot represents a CWE positioned according to its $S_i$ severity and $F_i$ frequency. In general, CWEs towards the upper right are more significant and those towards the lower left are less significant. Note how the majority of the CWEs are squished very close to the x-axis as many have a very small frequency. Also, the range of x-values is constrained from .37 to .97 (when the normalization should make it from 0 to 1).

Figure \ref{fig:MSSW-Riskmap} shows the same risk map using our double log frequency $F_i''$ and our modified severity $S_i'$. Note how the CWEs are now more uniformly spread over the y-axis. Also, the range of x-axis values is now from 0 to 1. The MSSW equation that combines frequency and severity using the values shown in Figure \ref{fig:MSSW-Riskmap} will now more equally combine them than with the MDSE values shown in Figure \ref{fig:MDSE-Riskmap}.

\subsubsection{Set Difference Experiment}

\begin{figure}
\centerline{\includegraphics[scale=.6]{images/MSSW-Set-Difference.jpg}}
\caption{The Size of the Set Difference between Top Lists from the MSSW Equation Compared to Frequency Top Lists (red lower line), Severity Top Lists (yellow middle line), and the Theoretical Maximum (blue top line)}
\label{fig:MSSW-Set-Difference}
\end{figure}

In Figure \ref{fig:MSSW-Set-Difference} we show the size of the set difference between the MSSW top list and the severity top list (the mostly lower red line). We also calculate the set difference between the MSSW top list and the frequency top list (the middle yellow line). Note how the red and yellow lines are much closer together than in Figure \ref{fig:MDSE-Set-Difference} and how the red line does not hover close to 0 like it does in Figure \ref{fig:MDSE-Set-Difference}. This demonstrates that the MSSW equation is more evenly balancing inclusion of the top frequency and top severity CWEs.

Note that the goal is not to have the red and yellow lines match. The top list should not necessarily evenly include an equal number of both top frequency and top severity CWEs. Our point with this analysis is to show how the MDSE equation almost exclusively chooses the top frequency CWEs and how our MSSW equation factors in CWEs from both sets. The next subsection will evaluate this more equal inclusion in more detail, focusing on top lists of size 20.

\subsubsection{Chosen CWE Experiment}

\begin{figure}
\centerline{\includegraphics[scale=.6]{images/Freq-vs-MSSW.jpg}}
\caption{CWEs Chosen (Red Triangles) and Not Chosen (Yellow Circles) for a MSSW Top 20 List Relative to Frequency}
\label{fig:MSSW-vs-Frequency}
\end{figure}

\begin{figure}
\centerline{\includegraphics[scale=.6]{images/CVSS-vs-MSSW.jpg}}
\caption{CWEs Chosen (Red Triangles) and Not Chosen (Yellow Circles) for a MSSW Top 20 List Relative to Severity}
\label{fig:MSSW-vs-CVSS}
\end{figure}

Figure \ref{fig:MSSW-vs-Frequency} shows the MSSW scores plotted against the double log frequency $F''_i$ scores. Each point represents a CWE. The red triangles indicate the CWEs that were chosen for the MSSW top 20 list. Note how unlike in the analogous Figure \ref{fig:MDSE-vs-Frequency} for MDSE, there are many higher frequency CWEs that are not chosen for the top 20 list due to their severity not being high enough.

Likewise, Figure \ref{fig:MSSW-vs-CVSS} shows the MSSW scores plotted against the $S'_i$ normalized mean CVSS score for each CWE. Note how the range spreads from 0 to 1, unlike the analogous Figure \ref{fig:MDSE-vs-CVSS} for the MDSE equation. Also note how the MSSW equation chooses CWEs for the top 20 list from CWEs with generally higher CVSS scores. However, it excludes many high severity CWEs because their frequencies were too low.

\section{2019 Top 20 Lists of the Most Significant Weaknesses}
\label{sec.TopWeaknesses}

We now use our MSSW equation to generate lists of the most significant software security weaknesses. We choose a list size of 20, somewhat arbitrarily, to enable the lists to conveniently fit on a page. We performed the experiments on a variety of list sizes and did not discover any appreciable differences. We did not choose a size of 25 to match the CWE top list because we needed to produce two top lists of differing levels of abstraction to get the most accurate results (explained below) and thus were unable to produce a single list of 25 CWEs for an ideal comparison with the CWE top 25 list.

We follow the approach in \cite{mell2020} of separately providing a top list for CWEs of higher levels of abstraction (pillars and classes) apart from a list covering CWEs of lower levels of abstraction (bases, variants, and compounds). This was done in \cite{mell2020} to avoid errors in frequency calculations that exist in CWE's top 25 list. The paper argues that CWEs mapped to lower level abstractions (e.g., bases) also should count towards their parent abstractions (e.g., classes); this is not done with MDSE calculations. For example, class CWE-20 (Improper Input Validation) is a parent of base CWE-1289 (Improper Validation of Unsafe Equivalence in Input). If a vulnerability exists with CWE-1289, then CWE-20 also needs to be taken into account with the CWE frequency counts. However when this frequency propagation is performed, combining together the two abstractions results in a single top list with a bias towards parents with many children (especially popular children). Thus the 2 levels of abstraction need to be presented in separate top lists.

We will refer to the higher level abstraction list as the class list and the lower level abstraction list as the base list for convenience and because both lists are primarily composed of either classes or bases. We also follow \cite{mell2020} in using published CWE taxonomy views 1000 and 1008 (discussed in Section \ref{CWE}) to propagate CVE data from child CWEs to their parents (discussed above). This provides a more accurate mapping of CVEs onto the CWEs, providing a more accurate data foundation upon which to apply our MSSW equation.\footnote{This propagation especially helps the formulation of the class list since most classes have children. It has a lesser effect on the base list. Note that it is impossible to inverse the propagation of data. CWEs are labelled as specifically as possible by NVD analysts so CVEs described by pillar or class CWEs do not get reflected in the base list. It is even possible that they shouldn't because there may be unidentified bases missing from view 1003 that are still covered by the view 1003 classes.}

These modifications also alter the frequency and severity distributions which could potentially render our double log function invalid. However, Table \ref{tab:correlation} shows correlation results for using and not using all combinations of the modifications adopted from \cite{mell2020}. It shows that the MDSE equation is highly correlated to frequency (.97 or higher) with very little correlation to severity (.25 or lower) regardless of the modifications used or not used. It also shows that the MSSW equation is strongly correlated to both frequency (.81 or higher with one exception) and severity (.66 or higher) regardless of the modifications used. Our one exception is for the class list using propagation with MSSW; even here the frequency correlation was .55 (still strong but much less than the others).

Note that our objective is not for the correlations to necessarily be equal, but that there exists a strong correlation for both frequency and severity. Depending upon the data, the higher frequency CVEs may or may not also be the highest severity CVEs. If so, then the correlations to frequency and severity would both be very high and almost equal. If not, both should still be high but one may be higher than the other. What we do not want in these results is for one of frequency or severity to have a high correlation and the other to have a very low correlation (which can be seen with the MDSE equation).

\begin{table}
\centering
\caption{Measurements Showing the Pearson Correlation of MDSE and MSSW to Frequency and Severity}
\label{tab:correlation}
\begin{tabular}{lllll}
 &  &  & \multicolumn{2}{c}{Correlation} \\
Equation & Abstraction & Propagation & Frequency & Severity \\ \hline
MDSE & All & Yes & .99 & .08 \\
MDSE & All & No & .98 & .18 \\
MDSE & High & Yes & .99 & .10 \\
MDSE & High & No & .98 & .25 \\
MDSE & Low & Yes & .97 & .20 \\
MDSE & Low & No & .97 & .18 \\ \hline
MSSW & All & Yes & .81 & .70 \\
MSSW & All & No & .86 & .66 \\
MSSW & High & Yes & .55 & .96 \\
MSSW & High & No & .84 & .68 \\
MSSW & Low & Yes & .84 & .67 \\
MSSW & Low & No & .83 & .68
\end{tabular}
\end{table}

We also checked to see that the double log still linearized the frequency distribution when using both variants from \cite{mell2020}. While propagating CVEs over the CWEs using the CWE taxonomies and using all applicable CWEs (i.e., pillars, classes, bases, variants, and compounds), the results show that the double log does still linearize the frequency (see Figure \ref{fig:G2-distributions}). The same results were obtained while also performing the experiment using just the pillars/classes and then just the bases, variants, and compounds (graphs not shown).

\begin{figure}
\centerline{\includegraphics[scale=.6]{images/G2Freq-SLog-DLog-Distributions.jpg}}
\caption{Normalized Distributions of Frequency (bottom blue line), Log of Frequency (middle yellow line), and Double Log of Frequency (top red line).}
\label{fig:G2-distributions}
\end{figure}

Using our MSSW equation to aggregate the frequency and severity of CWEs, the top 20 class list for 2019 is shown in Table \ref{tab:Top20Classes}. The top 20 base list is shown in Table \ref{tab:Top20Bases}. These two lists use the modification from \cite{mell2020} where the CVEs are propagated up through the CWE taxonomies.
We claim that these two lists represent the most accurate measurement yet produced for determining the most significant software security weaknesses. Given that there is no ground truth for how to best combine frequency and severity and no ground truth upon which to establish the correctness of the CVSS metric, it is likely impossible to prove any such metric as maximally effective. We make our `most accurate measurement yet' claim based on the demonstrated limitations in the published MDSE equation and a lack of competing published alternatives.

\begin{table*}
\centering
\caption{2019 MSSW Top 20 Pillars/Classes, Propagating CVSS Data over CWE Taxonomies}
\begin{tabular}{llllll}
Rank & Identifier & CWE Description & MSSW Score & Frequency & Mean CVSS\\
1 & \cwelink{913} & Improper Control of Dynamically-Managed Code Resources & 78.31 & 188 & 8.81 \\
2 & \cwelink{119} & Improper Restriction of Operations within Bounds of a Memory Buffer & 71.14 & 2745 & 8.00 \\
3 & \cwelink{669} & Incorrect Resource Transfer Between Spheres & 64.86 & 181 & 8.31 \\
4 & \cwelink{672} & Operation on a Resource after Expiration or Release & 64.56 & 876 & 7.96 \\
5 & \cwelink{330} & Use of Insufficiently Random Values & 63.74 & 111 & 8.43 \\
6 & \cwelink{704} & Incorrect Type Conversion or Cast & 62.55 & 54 & 8.68 \\
7 & \cwelink{287} & Improper Authentication & 59.75 & 627 & 7.86 \\
8 & \cwelink{345} & Insufficient Verification of Data Authenticity & 54.60 & 483 & 7.73 \\
9 & \cwelink{682} & Incorrect Calculation & 51.94 & 215 & 7.78 \\
10 & \cwelink{269} & Improper Privilege Management & 50.57 & 258 & 7.70 \\
11 & \cwelink{610} & Externally Controlled Reference to a Resource in Another Sphere & 48.38 & 725 & 7.46 \\
12 & \cwelink{706} & Use of Incorrectly-Resolved Name or Reference & 39.04 & 358 & 7.23 \\
13 & \cwelink{20} & Improper Input Validation & 38.56 & 3960 & 6.99 \\
14 & \cwelink{116} & Improper Encoding or Escaping of Output & 32.13 & 2461 & 6.82 \\
15 & \cwelink{400} & Uncontrolled Resource Consumption & 32.07 & 272 & 7.01 \\
16 & \cwelink{74} & Improper Neutralization of Special Elements in Output ... ('Injection')  & 32.06 & 2455 & 6.82 \\
17 & \cwelink{754} & Improper Check for Unusual or Exceptional Conditions & 32.05 & 264 & 7.01 \\
18 & \cwelink{326} & Inadequate Encryption Strength & 28.21 & 35 & 7.24 \\
19 & \cwelink{668} & Exposure of Resource to Wrong Sphere & 26.59 & 2292 & 6.66 \\
20 & \cwelink{436} & Interpretation Conflict & 22.40 & 17 & 7.19 \\
\end{tabular}
\label{tab:Top20Classes}
\end{table*}

\begin{table*}
\centering
\caption{2019 MSSW Top 20 Bases/Variants/Compounds, Propagating CVSS Data over CWE Taxonomies}

\begin{tabular}{llllll}
Rank & Identifier & CWE Description & MSSW Score & Frequency & Mean CVSS\\
1 & \cwelink{89} & Improper Neutralization of Special Elements used  ...  ('SQL Injection') & 71.70 & 384 & 8.89 \\
2 & \cwelink{502} & Deserialization of Untrusted Data & 61.73 & 83 & 9.01 \\
3 & \cwelink{787} & Out-of-bounds Write & 61.57 & 423 & 8.34 \\
4 & \cwelink{78} & Improper Neutralization of Special ... ('OS Command Injection')  & 61.22 & 194 & 8.58 \\
5 & \cwelink{120} & Buffer Copy without Checking Size of ... ('Classic Buffer Overflow') & 59.35 & 162 & 8.55 \\
6 & \cwelink{94} & Improper Control of Generation of Code ('Code Injection') & 58.62 & 100 & 8.72 \\
7 & \cwelink{798} & Use of Hard-coded Credentials & 58.07 & 89 & 8.75 \\
8 & \cwelink{434} & Unrestricted Upload of File with Dangerous Type & 57.95 & 167 & 8.46 \\
9 & \cwelink{416} & Use After Free & 56.69 & 426 & 8.09 \\
10 & \cwelink{352} & Cross-Site Request Forgery  (CSRF) & 51.60 & 386 & 7.86 \\
11 & \cwelink{346} & Origin Validation Error & 51.51 & 430 & 7.82 \\
12 & \cwelink{613} & Insufficient Session Expiration & 51.08 & 402 & 7.82 \\
13 & \cwelink{190} & Integer Overflow or Wraparound & 48.79 & 164 & 7.95 \\
14 & \cwelink{415} & Double Free & 43.17 & 46 & 8.15 \\
15 & \cwelink{125} & Out-of-bounds Read & 42.34 & 658 & 7.28 \\
16 & \cwelink{129} & Improper Validation of Array Index & 41.97 & 25 & 8.50 \\
17 & \cwelink{611} & Improper Restriction of XML External Entity Reference & 41.47 & 100 & 7.69 \\
18 & \cwelink{918} & Server-Side Request Forgery (SSRF) & 41.05 & 74 & 7.78 \\
19 & \cwelink{22} & Improper Limitation of a Pathname to a Restricted ... ('Path Traversal') & 39.40 & 309 & 7.27 \\
20 & \cwelink{191} & Integer Underflow (Wrap or Wraparound) & 37.76 & 18 & 8.47 \\
\end{tabular}
\label{tab:Top20Bases}
\end{table*}

\section{Discussion and Analysis of the Most Significant Weaknesses}
\label{sec.Evaluation}

\input{IrenaSecMellRev}

\section{Related Work}
\label{sec.RelatedWork}

\input{AssaneSecMellRev}

\section{Future Work}
\label{sec.FutureWork}

This goal of this work is to identify and fix the unintended bias in the MDSE equation towards frequency. Thus we design the MSSW equation to, as evenly as possible, factor together frequency and severity. And this is rational as it models typical security risk matrices that equally combine probability and impact (e.g., \cite{engert1999risk}). However, it is possible that intentionally biasing towards either frequency or severity is more useful in this domain. Also, the CVSS severity equation is itself an aggregate of exploitability and impact. Future work should evaluate whether or not any intentional bias should be added between these 3 factors.

Also, future work should evaluate additional metrics that might be useful for determining the most significant CWEs. In particular, it would be useful to identify CWEs whose associated vulnerabilities are frequently used in actual and impactful breaches. We note that the CVSS temporal equations provide some of this, but these results are not commonly calculated and no public repository of this data exists. That said, some data does exist to support such mappings (e.g., \cite{Podjarny}).

\section{Conclusion}
\label{sec.Conclusions}

The field of security metrics is a difficult area of scientific research because there is often no ground truth, unlike disciplines such as physics and chemistry. This may lead one to focus on just taking simple low level measurements that are inherently defensible; that was the approach taken in \cite{mell2020}. However, creating aggregate metrics that compose multiple simple measurements is of practical importance for the field of security. In this work we did just that, aggregating frequency and severity (i.e., exploitability and impact) into a single metric. Our objective is not for the correlations to necessarily be equal, but that there exists a strong correlation for both factors which more evenly balances the inclusion of the top frequency and top severity CWEs. This seemingly simple task proved challenging because of the differing distributions of both simpler metrics. Indeed, the officially published CWE metric neglected this property and did not achieve its stated objective (almost exclusively choosing the most frequent CWEs). With our work, we claim to have addressed the limitations and to have produced the most accurate equation yet for measuring the most significant software security weakness.

redpajama_set_name: RedPajamaArXiv


If it's time to purchase new gutters for your home in Des Moines, Iowa, or another nearby area, turn to the gutter experts at Waddle Exteriors. Our factory- and manufacturer-trained employees will ensure that your new gutter system is installed perfectly, and to further support the premium quality of our work we offer a 100 percent satisfaction guarantee on all of the gutters that we install.
Seamless – We will custom-cut your new gutters to the exact dimensions of your roofline, so they're won't be any seams holding your gutter system together.
Installed with hidden hangers – Our hidden hanger system helps to make your gutters a more natural and visually-appealing part of your home.
Available in several different colors – Since we can cut gutters from many different kinds of metal, you'll be able to choose one that matches your home's aesthetic.
To help keep your new gutters clean, Waddle Exteriors offers Gutter Topper, which is the No. 1 consumer-rated gutter protection system on the market. Gutter Topper has a rounded nose design, which uses surface tension to channel water into your gutters, while preventing leaves and other potentially obstructive materials from causing a blockage.
If you're looking for new gutters, contact Waddle Exteriors today for expert installations and superior products. We proudly serve residents of Des Moines, IA, and other nearby areas.


If you are looking for a faster alternative to straighten your teeth, then Six Month Smiles may be the ideal choice for you. This innovative breakthrough in orthodontic technology makes use of tooth-colored wires and clear brackets that move your teeth into their appropriate alignment in as fast as six months.
Traditional braces are highly effective at correcting the alignment of the teeth. However, they come with a significant number of drawbacks that make them undesirable to some people. The metal brackets and wires are not visually appealing and also not very comfortable. Additionally, they typically need to be worn for a year or two, causing many people to avoid treatment altogether. With the Six Month Smiles program, treatment time is reduced to roughly half that of traditional braces.
The brackets used in Six Month Smiles are clear, and the wires are made using tooth-colored nickel-titanium which is a biocompatible material. The treatment works by slowly moving the teeth using a slight amount of force. There is no risk of damage to the tooth roots with these braces. Once the desired results have been achieved, and the teeth are properly aligned, the braces can be removed, and the retention phase of the treatment can be started in order to prevent the teeth from shifting back into their old positions.
Six Month Smiles is one of the best orthodontic treatments of modern times. With this comfortable and advanced form of therapy, you can attain successful results in the form of a beautiful smile in much less time!
If you would like more information on Six Month Smiles, or if want to schedule an evaluation with one of our two conveniently located offices, contact Heresi Dental Care at (540) 249-1849. Ask us about convenient options for financing your dental treatment.


Bar lines are drawn by MusiCAD as necessary determined by meter and note durations.
A special form of bar lines are bar lines with repeating dots at one or both sides. Bar line-repeats should always be paired with one exception for the first repeat at the start of a melody, where it is very uncommon to write down the repeating dots at the (missing) first bar line. Prima volta and secunda volta signs may be used in conjunction with bar lines having repeat-dots.
To change the appearance of a bar line you can use <\> repeatedly. It will change the bar line from single, to double, opening and closing bar lines.


On this page you'll find my travel blogs from countries in Asia. From Thailand to Vietnam and from Beijing to Japan. I'm sharing my personal experiences with you. I'll tell you what to do and where to eat in Asia. And as always I'm focussing on vegetarian food as well!


This gruesome crime – The Murders on Smuttynose Island, Isles of Shoals – is well known.
Now you can own the original front page of the Wednesday, March 19, 1873 edition of The Gardiner Home Journal printed in Gardiner, Maine, that details Louis Wagner and the murders he was executed for.
Wagner was then brought to Alfred, Maine for trial. After he was condemned to death, he broke out of jail and escaped to New Hampshire. He was recaptured and brought to the gallows at Thomaston State Prison.
The BEST book available is Lyman Rutledge's Moonlight Murder at Smuttynose which is available online from the book store on Star Island.


text:
\section{\label{sec:introduction}INTRODUCTION}

Photon-number-resolving detectors are devices capable of accurately measuring photon numbers in input wave-packets. Having the ability to resolve photon numbers is applicable and highly desirable in imaging applications \cite{Holland1999}, investigation of exceptional points in $\mathcal{PT}$-symmetric systems \cite{QuirozJuarez2019}, quantum key distribution \cite{Cattaneo2018HybridDetectors}, decoy states for quantum key distribution security \cite{Hwang2003QuantumCommunication, Lo2005DecoyDistribution}, measurements in the number basis \cite{Kovalenko2018}, and photon-counting laser-radars \cite{Huang2014}.

PNR detectors can be divided into two categories, inherent PNR detectors which utilize some process where the quantity used as the output has a well-defined dependence on the incident photon number, and multiplexed PNR detectors which split the input over multiple single-photon detectors. Examples of detectors in the former category are transition edge sensors (TES) \cite{Lita2010SuperconductingDetectors, Fukuda2011, Konno2020DevelopmentArray}, complementary metal–oxide–semiconductor (CMOS) detectors \cite{Ma2017Photon-number-resolvingGain} and charge coupled device (CCD) detectors. The latter category consists of various schemes to distribute the incoming photons over multiple single-photon detectors \cite{Paul1996PhotonLight, Kok2001DetectionPreparation, Sperling2012TrueDetectors}. Some examples of these schemes are spatial arrays \cite{Eraerds2007, Jiang2007, Divochiy2008}, loop arrays \cite{Banaszek2003, Rehacek2003Multiple-photonDetector, Haderka2004}, and temporal arrays \cite{Achilles2003, Fitch2003, Natarajan2013}.

The spatial array design consists of a symmetric $N$-port coupler that takes the input and ideally distribute the input photons uniformly over the $N$ single-photon detectors. However, the requirements on the number of single-photon detectors grow quickly \cite{Jonsson2020Photon-countingDetectors}, and the limited quantum efficiency of the detectors makes the design difficult to scale. An alternative design that uses fewer resources is the loop-based array \cite{Banaszek2003, Rehacek2003Multiple-photonDetector, Haderka2004}, which only requires one single-photon detector. However, this design results in a non-uniform distribution of photons over the time-bins and most of the light exits in the first few time-bins, which is undesirable.

The temporal multiplexed PNR detector is a device that splits input pulses over time-bins rather than over different detectors. Given that the time-bins are appropriately spaced and that the mean photon number per time-bin is much smaller than unity, it is possible to use single-photon detectors to measure the output. This results in a device that has an output distribution equivalent to a spatial array, but the required number of single-photon detectors is only two. However, by increasing the size of the temporal array, the time required per measurement increase, which reduces the detection rate.

The ability to resolve photon numbers can be quantified in multiple ways, which differ due to the application considered. In this work we mainly consider two different types of PNR capabilities. The first is the ability to find the correct photon number in a single wave-packet, single-shot measurement. This capability is useful in photon number encoded communication or quantum cryptography, where each signal is only sent once. The second is the ability to reconstruct the number distribution of the input source is useful in characterization of low intensity light sources or imaging at low intensities. A detector capable of single-shot PNR measurements is also capable of reconstructing the number distribution, but the former is much more difficult to attain and it is also difficult to quantify it directly.

The work is organized as follows. In Sec. \ref{sec:setup} we present the experimental setup for a $16$ element temporal array with two superconducting nanowire single-photon detectors as detector elements. In Sec. \ref{sec:parameter-estimation} a framework for estimating the mean photon numbers in incident laser light is presented and the limits in resolvable mean photon numbers are predicted using our model. In Sec. \ref{sec:results} the results of the experiments are presented. This includes results from measurements of losses in the multiplexer and in the single-photon detector (in Sec. \ref{sec:detector-losses}), measurements and estimations of PNR capability for multiple input data and for single input data (in Sec. \ref{sec:pnr-capability}), and the bandwidth of the detector (in Sec. \ref{sec:bandwidth}). Finally, in Sec. \ref{sec:summary}, we summarize our findings and draw conclusions.

\section{\label{sec:setup}SETUP}
\begin{figure*}[t]
    \centering
    \includegraphics[width=\linewidth]{setup/setup.pdf}
    \caption{Schematic figure of our setup. The signal generator triggers the laser with a frequency of $\SI{100}{kHZ}$. The laser light is filtered through a $\SI{980}{nm}$ narrow band-pass filter and attenuated with a variable attenuator. The attenuated light is coupled into the multiplexer which splits the light into $8$ time-bins in two output fibers. The polarization of the output is adjusted with polarization controllers and the light is detected with SNSPDs.}
    \label{fig:setup}
\end{figure*}

A schematic image of the setup is presented in Fig. \ref{fig:setup}. A $\SI{980}{nm}$ wavelength semiconductor laser is electrically driven at a repetition rate of $\SI{100}{kHz}$ with a pulse length of $\SI{16.3}{ns}$ using a signal generator. The laser output is filtered through a narrow band-pass filter and subsequently attenuated with a variable, neutral density filter to the few photon regime. These photons are focused with a lens and coupled into a polarization maintaining PANDA fiber PM980-XP. The fiber is connected to a temporal multiplexer made of a series of nominally $50/50$, polarization maintaining fiber couplers, which have been connected with PANDA fiber-loops of lengths between nominally zero to 120 m.
When splicing the input fiber and the loops to the couplers, great care was taken to align the PANDA fibers ends both transversely and rotationally as to minimize the splicing losses. The multiplexer splits every input laser pulse into $8$ non-overlapping time-bin pulses, which results in an effective detector array size of $16$ elements.

The (linear) polarization state of the output from the multiplexer is rotated using polarization controllers prior to the detectors to maximize the respective count rate from the two superconducting, nanowire single-photon detectors (SNSPD). There are two main reasons for chosing PANDA fibers instead of ordinary single mode (SM) fibers for the multiplexer in spite of the higher losses for the former. The first is that we can consistently make sure that the setup operates in the polarization regime where the SNDPD detection efficiency is the highest. The second is that also fiber couplers typically are polarization sensitive, so that the splitting ratio between the 16 output pulses would depend on the polarization state of the input if SM fibers would have been used. Moreover, mechanical stress and temperature changes may shift the polarization in SM fibers, which makes it difficult to control the polarization.

The SNSPDs are set to operate at a bias current of $\SI{13.00}{\mu A}$, $\SI{10.10}{\mu A}$ and a trigger voltage of $\SI{150}{mV}$, $\SI{50}{mV}$, respectively. The electrical outputs of the SNSPDs are recorded using a quTools quTag time-tagger with picosecond resolution is synchronized with the laser pulses. The outputs from the SNSPDs are time-binned with $\SI{30}{ns}$ windows around the expected arrival times relative to the trigger signal.

\section{\label{sec:parameter-estimation}Parameter estimation}
One potential application for the temporal array detector is to use it to estimate the mean photon number for values where neither single-photon detectors nor power meters are capable of making an accurate measurement. Here we consider the scenario when a Poissonian laser pulse of mean photon number $\mu$ is used as input for the device. The goal is to find an estimator $\hat{\mu}$ and estimate the error $\epsilon = \abs{\hat{\mu} - \mu}$.

We use the maximum likelihood method to find an estimator $\hat{\mu}_{\text{MLE}}$ for the mean photon number $\mu$. To compute this estimator we use that the probability to get $x$ clicks from the SNSPDs after the laser pulse passes through the multiplexer with $n$ effective detector elements is given by \cite{Dauler2009Photon-number-resolutionDetectors, Zhu2019ResolvingDetector, Jonsson2020Photon-countingDetectors}
\begin{equation}
    \Pr(x \mid \mu) = \binom{n}{x} e^{- \mu \eta} (1 - p_d)^n \qty(\frac{e^{\mu \eta / n}}{1 - p_d} - 1)^x,
    \label{eq:poisson-pcd}
\end{equation}
where $\eta$ and $p_d$ are the quantum efficiency and probability for a dark count per time-bin, respectively. The maximum likelihood estimator for the measurement data $\{x_i\}_{i = 0}^{N - 1}$ from $N$ laser pulses is then given by
\begin{equation}
    \hat{\mu}_{\text{MLE}}(\ev{x}) = - \frac{n}{\eta} \ln\qty(\frac{n - \ev{x}}{(1 - p_d) n}),
    \label{eq:mle-poisson-pcd}
\end{equation}
where the expectation value over the experimental data $\ev{x}$ is given by
\begin{equation}
    \ev{x} = \frac{1}{N} \sum_{i = 0}^{N - 1} x_i.
\end{equation}

The error of the maximum likelihood estimator can be divided into two parts, the variance and the resolution error caused by finite number of data points. The variance can be estimated by propagating the sample variance for $\ev{x}$ to $\hat{\mu}_{\text{MLE}}$ and a lower bound can be computed using the Cramer-Rao bound, which gives that
\begin{equation}
    \variance\qty(\hat{\mu}_{\text{MLE}}) \geq \frac{n \qty[(1 - p_d)^{-1} e^{\mu \eta / n} - 1]}{\eta^2 N}.
\end{equation}
Hence, the variance grows at least exponentially with $\mu \eta / n$, which implies that the number of required measurements to reach a certain set variance grows exponentially with the mean photon number and it therefore becomes unfeasible to measure too strong input signals. We also note that an array with size $n$ can measure around $n$ times more photons with the same variance as a single-photon detector.

The resolution error arises from the fact that the quantity $N \ev{x}$ is a non-negative integer less than or equal to $n N$. This implies that the there are in total $nN$ possible finite values $S = \{\hat{\mu}_{\text{MLE}}(0), \hat{\mu}_{\text{MLE}}(1/N), \dots, \hat{\mu}_{\text{MLE}}(n - 1/N)\}$ that the estimator $\hat{\mu}_{\text{MLE}}$ can take and the estimator has a resolution error if mean photon number $\mu \not\in S$. Let us quantify this error by the spacing between the estimator and the next possible value
\begin{equation}
\begin{split}
    \Delta \hat{\mu}_{\text{MLE}}(\ev{x}) &= \hat{\mu}_{\text{MLE}}(\ev{x} + 1/N) - \hat{\mu}_{\text{MLE}}(\ev{x})\\
    &= - \frac{n}{\eta} \ln\qty(\frac{n - \ev{x} - 1 / N}{n - \ev{x}}).
\end{split}
\end{equation}
In the weak light limit when $\ev{x} \ll n$ we notice that the spacing between adjacent estimator values is approximately
\begin{equation}
    \Delta \hat{\mu}_{\text{MLE}}(\ev{x}) \approx \frac{1}{\eta N},
\end{equation}
which is small for sufficiently many measurements. However, for strong light the spacing is not small and the resolution error diverges as $\ev{x} \to n$.

The resolution error and the limited number of outputs effectively limits how large $\mu$ that can be resolved with the detector. The largest value that can be recorded with finite resolution error and a finite estimator is
\begin{equation}
    \hat{\mu}_{\text{MLE}}(n - 2/N) = \frac{n}{\eta} \ln\qty(\frac{n N (1 - p_d)}{2}).
\end{equation}
For this experiment when $N \approx 10^6$ and the dark counts are negligible we get that the largest resolvable input is $\mu \eta_{\textrm{Max}} \approx 250$ when $n = 16$, while for a (non-multiplexed) single-photon detector $\mu \eta_{\textrm{Max}} \approx 13$.

\section{\label{sec:results}RESULTS}

\subsection{\label{sec:detector-losses}Detector characterization}

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{pulse/pulse.pdf}
    \caption{Counts per time-bin is displayed for the detectors. The multiplexer splits the input into $16$, $\SI{30}{ns}$ long, time-bins with close to uniform splitting ratio. Spacing between two following time-bins is approximately of $\SI{150}{ns}$, which sets the requirement that the single-photon detectors must be able to resolve signals of at least $\SI{7}{MHz}$.}
    \label{fig:pulse}
\end{figure}

Losses in the PNR detector can be divided into two types, losses in the temporal multiplexer and limited efficiency of the SNSPD. The former types include fiber losses, losses in splices, and excess loss in the couplers, while losses due to non-unity quantum efficiency belong to the latter type. To estimate the total loss in the multiplexer an optical power meter is used to compare the power at the detectors with the output before the first coupler. These power measurements are conducted without the neutral density filters and the power measurement before the first coupler is done by cutting the fiber. The total loss in the multiplexer is measured to be $\SI{0.63}{dB}$.

The measured loss is well in line with the average expected transmission loss of $\SI{0.66}{dB}$ (the average transmission is 0.86), based on the manufacturers' specifications, where it is assumed that each coupler has a loss of $\SI{0.1}{dB}$ and the fiber has a loss of $\SI{2.5}{dB/km}$ with an average travel distance of $\SI{105}{m}$. Each pulse also passes through 8 fiber splices. However, the splice losses are negligible compared to the other losses.

To measure the quantum efficiencies of the SNSPDs, including the fiber pigtail, the count rate of a laser at $\SI{980}{nm}$ attenuated with a calibrated attenuator is measured. The unattenuated laser power is measured using a power meter and is compared with the measurements conducted with the SNSPDs. The resulting nominal quantum efficiencies are $\SI{50}{\percent}$ and $\SI{64}{\percent}$, respectively, chosen so that the dark count rates are $\SI{0.11}{Hz} \pm \SI{0.04}{Hz}$, $\SI{0.22}{Hz} \pm \SI{0.05}{Hz}$ respectively. Hence, the average, overall quantum efficiency of the whole setup becomes $0.86 \times (0.50 + 0.64)/2 = \SI{49}{\percent}$.

 The probability for a dark count to occur within any of the $\SI{30}{ns}$ time-bins during one input pulse is $\SI{2.6e-9}{}$, $\SI{5.3e-8}{}$, which is negligible for any signal resolvable with the array. Hence it is justified to neglect dark counts in the data analysis.

A histogram displaying recorded detections in one time-bin is presented in Fig. \ref{fig:pulse} and a histogram of all time-bins in one of the output fibers is presented in the inset. The laser pulses have a $\approx \SI{4}{ns}$ FWHM which is about one order of magnitude smaller than the time-bin length. The multiplexer time-bins have an approximate spacing of $\SI{150}{ns}$ which requires the single-photon detectors to have a count rate of at least $\SI{7}{MHz}$ in order not to saturate. The used SNSPDs have a maximal count rate that exceeds $\SI{10}{MHz}$ which suggest that the spacing between the time-bins could be reduced slightly before saturation occurs. However, the SNSPDs experience a quantum efficiency drop when the count rate approaches the maximum count rate. The time-bin spacing was therefore chosen with a margin to guarantee that no extra loss is introduced.

\subsection{\label{sec:pnr-capability}PNR capability}

\begin{figure*}
    \centering
    \subfloat[\label{fig:distribution-od2}]{%
       \includegraphics[width=.33\linewidth]{distribution/20200519_PNR_780HPFiber_OD2timetag.pdf}
    }
    \subfloat[\label{fig:distribution-od3}]{%
       \includegraphics[width=.33\linewidth]{distribution/20200519_PNR_780HPFiber_OD3timetag.pdf}
    }
    \subfloat[\label{fig:distribution-od4}]{%
       \includegraphics[width=.33\linewidth]{distribution/20200519_PNR_780HPFiber_OD4timetag.pdf}
    }

    \subfloat[\label{fig:distribution-od45}]{%
       \includegraphics[width=.33\linewidth]{distribution/20200519_PNR_780HPFiber_OD45timetag.pdf}
    }
    \subfloat[\label{fig:distribution-od5}]{%
       \includegraphics[width=.33\linewidth]{distribution/20200519_PNR_780HPFiber_OD5timetag.pdf}
    }
    \subfloat[\label{fig:distribution-od7}]{%
       \includegraphics[width=.33\linewidth]{distribution/20200519_PNR_780HPFiber_OD7timetag.pdf}
    }
    \caption{Output count distribution (blue, circles) for different attenuation strengths. Each distribution is fitted to Eq. \eqref{eq:poisson-pcd} using the maximum likelihood estimator under the assumption that dark counts are negligible (orange, squares). The mean photon number estimator is presented in the legends with a one standard deviation estimated uncertainty. The resolution error is at least two orders of magnitude smaller than the standard deviation for these mean photon numbers and is excluded from the presented error. (a) Attenuation is $\text{OD} = 5.0$. The detector is close to saturation and most pulses result in detections in all $16$ time-bins. The existence of pulses resulting in less than $16$ detection events allows for estimation of the mean photon number. The blue curve is completely hidden behind the orange. (b) Attenuation is $\text{OD} = 5.6$. (c) Attenuation is $\text{OD} = 5.9$ (d) Attenuation is $\text{OD} = 6.3$. (e) Attenuation is $\text{OD} = 7.0$ (f) Attenuation is $\text{OD} = 7.9$. For sufficiently small mean photon numbers the distribution approaches a Poisson distribution $\text{Po}(\ev{x})$.}
    \label{fig:distribution}
\end{figure*}

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{linearity/2020-05-19_no_overflow_fit.pdf}
    \caption{The mean photon-number estimation as a function of the attenuation with an exponential fit. The detector displays good linearity in the measured interval, with a small deviation in the region OD 7 to OD 8. This allows the detector to measure mean photon numbers over five orders of magnitude from $10^{-3}$ to $10^2$.}
    \label{fig:linearity}
\end{figure}

We consider two different types of PNR capabilities in this experiment: The ability to measure the photon number of a single wave-packet and measuring the photon number after many repeated measurements. The former is useful in communication schemes where the data is encoded in photon numbers and the latter is useful when characterizing few photon sources. Here we show the latter experimentally, while the former capability is investigated theoretically using data from the experiments.

The output distribution for different attenuation strengths with corresponding fits are presented in Fig. \ref{fig:distribution}. The fits display good agreement with experimental data suggesting that the array detector setup is well described by equation \eqref{eq:poisson-pcd}. A small deviation between the experimental data and the fit is apparent in Figs. \ref{fig:distribution-od4} and \ref{fig:distribution-od45} which is possibly explained by the fact that the two SNSPDs do not have identical quantum efficiencies as is assumed in the derivation of equation \eqref{eq:poisson-pcd}. The reason for the deviation to be largest for this mean photon number is likely due to the variance of $x$ being maximal for $\mu \eta = n \ln{2} \approx 11.1$ which is close to the value in the experiment. The variance approaches zero for both higher and lower mean photon numbers.

The fitted mean photon numbers as a function of the applied attenuation is presented in Fig. \ref{fig:linearity}. The fit is in good agreement with the experimental data and the fitted exponent is in good agreement with theory. A small deviation from the fit is present in the interval $\text{OD}$ 7 to $\text{OD}8$ which is possibly caused by uncertainty in attenuation or due to instabilities in laser power over time. Overall, the results suggest that the device is capable of accurately estimating the mean, detected photon numbers up to $\hat{\mu}_{\text{MLE}} \eta = 150$, but as predicted in section \ref{sec:parameter-estimation} the upper limit is likely closer to $250$. The estimated number can then be used to find the absolute mean photon numbers by carefully calibrating the quantum efficiency of the detectors.

The experimental setup is not very suitable to investigate the multiplexed detector's ability to resolve photon numbers in a single wave-packet. In order to do this, the laser should be replaced by a deterministic few photon source. However, the present setup makes it is possible to estimate the probability for a correct classification using the detector parameters. We assume that the detector is well described by \eqref{eq:poisson-pcd} which is supported by the preceding results in this subsection. It then holds that the probability to get $x$ clicks when the input is a Fock state of $m$ photons is given by \cite{Fitch2003, Jonsson2020Photon-countingDetectors}
\begin{equation}
\begin{split}
    \Pr(x \mid m) &= \frac{1}{n^m} \binom{n}{x} \sum_{l = 0}^x (-1)^l (1 - p_d)^{n - x + l} \binom{x}{l}\\
    &\times [n - (n - x + l)\eta]^m.
\end{split}
\end{equation}
For this setup, the total loss (loss due to non-unity quantum efficiency combined with the loss in the multiplexer) is $\eta = 0.49$ and $p_d = 0$ since dark counts can be neglected. Given these parameters it is only possible to resolve almost one photon with at least $\SI{50}{\percent}$ probability for a correct classification. This shows, that in spite of a rather elaborate choice of components and a working wavelength on the short side of the near-infrared wavelength range, it is still difficult to achieve true, single-shot PNR performance. Increasing the SNSPDs' quantum efficiencies to $\eta > 0.9$ would still only allow single-shot PNR performance with up to two photons using this 16-array multiplexer.

Next, we analyze the upper limits of our multiplexer to its PNR performance by assuming idealized components. First we consider the scenario when the single-photon detectors are ideal and have unit quantum efficiency. The effective quantum efficiency for our setup in this scenario is $\eta = 0.86$, all due to the loss in the multiplexer. In this case it is possible to measure up to $3$ photons with more than $\SI{50}{\percent}$ probability for a correct classification for the photon numbers 0 to 3 \cite{Jonsson2020Photon-countingDetectors}.

Finally, we consider the (unrealistic) scenario when both the single-photon detectors and the multiplexer are ideal with no losses. The effective quantum efficiency is in this scenario $\eta = 1$ and the detector is only limited by the probability that more than one photon ends up in the same time-bin. In this scenario it is possible to resolve up $5$ photons with $> \SI{50}{\percent}$ success \cite{Jonsson2020Photon-countingDetectors}. This demonstrates the difficulty to resolve photon numbers in a single-shot measurement scenario.

\subsection{\label{sec:bandwidth}Bandwidth}
\begin{figure*}[t]
    \centering
    \includegraphics[width=\linewidth]{bandwidth/bandwidth.pdf}
    \caption{Power in the 8 time-bins for each single photon detector normalized against the total power as a function of the wavelength. Both detectors show a linear relationship between normalized power and wavelength. (Left) The normalized power in detector 1. (Right) The normalized power in detector 2.}
    \label{fig:bandwidth-detector}
\end{figure*}

The single-photon detectors used to build the temporal array are wavelength dependent and the temporal array is therefore also wavelength dependent. In addition, a temporal-array detector built using fiber couplers also experiences a wavelength dependence due to the wavelength dependence of the coupling ratio. The latter dependence effectively reduces the array size by changing the splitting ratio for the couplers. This in turn, reduces the dynamic range and PNR capability of the array detector.

In Fig. \ref{fig:bandwidth-detector} the normalized power in each time-bin is presented as a function of the wavelength.  The figure shows that commercial fiber couplers still exhibit quite a bit of individual behavior. The measurements show that the relation between the coupling ratio and the wavelength is linear, at least within a few percent's deviation from the design wavelength. Given the linear relationship, the transmission from the lower input port to the lower output port or the upper input port to the upper output port can be modelled as $T_{ll} = T_{uu} = 1/2 + a \Delta \lambda$. Here $a$ is a constant determining how fast the transmission changes per unit wavelength, $\Delta \lambda$ is the offset from the operational wavelength and by assumption $\abs{a \Delta \lambda} \ll 1/2$ in the interval considered. Unitarity of the coupler implies that the transmission from lower to upper or vice versa is given by $T_{ul} = T_{lu} = 1/2 - a \Delta \lambda$.

The average power of the different time-bins can be computed using the transmissions and the total input power. To linear order in $a \Delta \lambda$ we get that the normalized power in the time-bins in the first detector are given by
\begin{equation}
    \frac{P_{1, \pm}}{P_{\text{tot}}} = \frac{1}{16} \pm \frac{1}{4} a \Delta \lambda + \order{(a \Delta \lambda)^2}
\end{equation}
where half of the time-bins correspond to $P_{1, +}$ and the other half to $P_{1, -}$. The normalized powers in the second detector are given by
\begin{align}
    \frac{P_{2, 0}}{P_{\text{tot}}} &= \frac{1}{16} + \order{(a \Delta \lambda)^2},\\
    \frac{P_{2, \pm}}{P_{\text{tot}}} &= \frac{1}{16} \pm \frac{1}{2} a \Delta \lambda + \order{(a \Delta \lambda)^2},
\end{align}
where six time-bins correspond to $P_{2, 0}$ and the $P_{2, +}$, $P_{2, -}$ have one time-bin each. The non-equal behavior in the first and second detector are due to the symmetry being broken when one input port of the first 50/50 coupler is chosen over the other. If equal power is incident in both input ports in the temporal multiplexer (but in a manner so that no interference between different pulses ensue) the output is also expected to be symmetric.

In the wavelength interval considered the array size of our detector effectively decreases to $11$ elements as $\abs{\Delta \lambda}$ grows sufficiently large. These $11$ elements sustain the same power or increase in power as the wavelength changes and they can therefore still be used to measure the input signal. The remaining $5$ time-bins have a power that is quickly reduced with wavelength and their contribution to the measurements therefore decrease as the wavelength differs sufficiently from $\SI{980}{nm}$.

\section{\label{sec:summary}SUMMARY}

In this paper we present an experimental realisation of a temporal detector array with $16$ time bins, and SNSPDs as detectors. We investigate the theoretical PNR capabilities of this detector and determine an upper bound for the mean photon number that can be resolved by the detector when multiple measurements are allowed. We conclude that our detector is able to resolve one order of magnitude larger mean photon numbers than the SNSPD detector alone is able to do with the same number of measurements. In numbers, our detector can measure the average photon number in pulses between $10^{-3}$ to $10^{2}$ photons if data is collected from a sample of around $10^6$ identical pulses. The precision improves with the sample size, while the accuracy is mainly determined by how well the quantum efficiency of the used SNSPDs can be determined. In our case the imprecision after $10^6$ samples is estimated to be $\SI{1}{\percent}$, while the accuracy is difficult to estimate since there are many factors that add uncertainty to the overall loss estimations.

Our experimental investigation of the PNR capabilities shows that the agreement with the theoretical model is excellent and that the detector is linear over $5$ orders of magnitude when estimating the mean photon number over multiple measurements. This suggests that the theoretical predictions of the maximal number of photons is reasonable. Furthermore, error estimations show that the predictions made with the detector have low variance and a statistical bias caused by discretized readout is negligible.

While we are using a pulsed input, the detector could in principle be used to measure the power in a (slowly varying) cw signal. To do this, one would use a periodic gating and use the gating frequency, the detector on-off ratio, and the overall quantum efficiency to translate the photo counts to optical power at a given wavelength. Used this way, our detector could measure powers in the range 0.1 pW to 10 nW. However, by increasing the gating frequency or the on/off-ratio (without coming close to unity), even lower powers could be detectable. The gating frequency sets a upper limit to how rapid changes in the signal can be detected. The detector's dark count rate sets the fundamental lower limit to the detectable optical power. At a dark count rate of 1 Hz and a SNR of unity the detection limit becomes around $10^{-19}$ W. However, such extremely weak signals carry non-negligible intrinsic fluctuations (due to shot-noise), so from a practical viewpoint one would probably not want to operate the detector below $10^{-17}$ W.

On the negative side, in spite of substantial efforts to maximize the overall quantum efficiency of the detector, our detector is not capable of single-shot photon-number resolution of pulses since its average, overall efficiency is $0.49$. To distinguish between zero, one, and two photons with a minimum success rate of 0.5, the quantum efficiency of the detector needs to be $\geq 0.85$. At telecom wavelengths fiber propagation losses are lower than at $\SI{980}{nm}$. However, this lower loss is offset by a decreasing quantum efficiency of the detectors as the wavelength increases \cite{Natarajan2013}.

We also investigate the bandwidth of the detector and show that the time-bins have different sensitivities for changes in wavelength. This effect is due to the couplers' changing splitting ratios when the wavelength changes and some time-bins correspond to paths where this effect accumulate, while other time-bins correspond to paths where the effect cancel. This shows that the power in $11$ of the $16$ time-bins is unchanged or increases as the wavelength changes with $\SI{20}{nm}$ around $\SI{980}{nm}$.

To improve array-based PNR detectors further, one would need to use detectors with very high quantum efficiency. It would also help to space the time-bins denser, resulting in shorter loops with lower propagation loss. However, the length of the loops in the multiplexer needs to be long enough to give the single-photon detectors time to recover without reducing the quantum efficiency. Ideally, one would like to make the whole device monolitically integrated as a photonic circuit. However, for this to happen, the detector recovery-time needs to be shortened to the sub-ns range in addition to solving the problem of integration between photonic and superconducting circuits.

\section*{\label{sec:acknowledgments}Acknowledgments}
M.J. and G.B. acknowledges funding from Knut and Alice Wallenberg Foundation through the grant Quantum Sensing and from the Swedish Research Council (VR) through grant 621-2014-5410.

M.S. acknowledges funding from the Swedish Research Council (VR) through grant 2018-03457.

S.G. and V.Z. acknowledges support of the ATTRACT project funded by the EC under Grant Agreement 777222, funding from the Knut and Alice Wallenberg Foundation Grant "Quantum Sensors" and support from the Swedish Research Council (VR) through the VR Grant for International Recruitment of Leading Researchers (ref. 2013-7152) and Research Environment Grant (ref. 2016-06122).

Figure \ref{fig:setup} was created with ComponentLibrary \cite{componentlibrary}.