\documentclass{article}

\usepackage{amsmath, amsthm, amssymb}
\usepackage{fullpage}

\begin{document}
\title{SIGVISA Internal Design}
\author{Dave Moore}
\date{}

This document describes the data structures used in SIGVISA's internal representations. This includes representations of events, stations, event arrivals at a station, waveform traces, aggregate data segments from a station potentially consisting of many waveform traces (corresponding to multiple channels and/or frequency bands), etc.

Because SIGVISA has portions written in Python and in C, some structures have versions in both languages. When this is the case, both versions will be described, along with (when available) functions to inferface between the two.

\section{Waveform Traces}

The {\em trace} is the most basic form of signal in SIGVISA. It is a simple time series giving the activity over some period of time at a station, along with certain metadata.

\subsection{Python}

In Python, a trace is represented by the {\tt obspy.signal.Trace} class, which contains two members: {\tt data} is a Numpy array with the raw data, and {\tt stats} is a Python dict containing metadata in the form of key-value pairs. Standard entries are

\begin{tabular}{lp{4in}}
{\bf Key} & {\bf Value}\\
{\tt starttime\_unix} & Start time in Unix timestamp format (seconds since midnight on 1/1/1970). \\
{\tt srate} & Sample rate in Hz.\\
{\tt npts} & Number of data points (should be equal to {\tt len(data)}). \\
{\tt siteid} & The station's site id, as an integer (corresponding to {\tt static\_siteid.id} in the database).\\
{\tt chan} & String representing the channel (usually ``BHZ'', ``BHN'', or ``BHE''; also possibly ``horiz\_avg'').\\
{\tt chan} & String representing the frequency band (e.g. ``broadband'', ``broadband\_envelope'', etc., see examples below).\\
{\tt noise\_floor} (optional) & The level of ambient background noise around the time of the signal. \\
{\tt p\_time} (optional) & The time of the first P-type phase arrival during the trace.\\
{\tt s\_time} (optional) & The time of the first S-type phase arrival during the trace.\\
{\tt p\_phaseid} (optional) & The phaseid (from {\tt static\_phaseid.id} of the first P-type phase arrival during the trace.\\
{\tt s\_phaseid} (optional) & The phaseid of the first S-type -phase arrival during the trace.
\end{tabular}

\subsection{C}

A trace in C is defined by the {\tt Trace\_t} structure in {\tt sigvisa.h}. The data are represented by a C array of doubles, with fields for other metadata.

\section{Signal Segments}

A {\tt signal segment} bundles together many traces that correspond to the same time period at the same station.

\subsection{Python}

A segment in Python is a nested structure, consisting of a dictionary of dictionaries of Traces. The keys in the first-level dictionary are channels (e.g. ``BHZ'' or ``horiz\_avg''), with the values being a different second-level dictionary for each channel. The second-level dictionaries have keys corresponding to different processed versions of the signal, e.g. the envelope, log envelope, or different frequency bands, and Traces as values. Example keys include:

\begin{itemize}
\item {\tt broadband} (raw signal data)
\item {\tt broadband\_envelope} (envelope of raw signal)
\item {\tt narrow\_logenvelope\_2.00\_3.00} (apply a 2-3Hz bandpass filter to raw signal, then take the log envelope)
\end{itemize}

\subsubsection{Loading}
\begin{itemize}
\item {\tt sigvisa\_util.load\_traces} returns a list of segments for a station between a given start and end time. These segments include {\tt broadband} but no other processing is performed.
\item {\tt sigvisa\_util.load\_and\_process\_traces} calls {\tt load\_traces} but also applies a processing function to each trace. Currently this is just computing the envelope, which is stored as {\tt broadband\_envelope}.
\item {\tt sigvisa\_util.compute\_narrowband\_envelopes} takes a list of segments, and for each one computes the logenvelope in a range of frequency bands.
\item {\tt sigvisa\_util.extract\_timeslice\_at\_station} takes a list of segments, and trims/concatenates them as necessary to form a single segment corresponding to the given time window at the given station.
\item {\tt priors.coda\_decay.coda\_decay\_common.load\_signal\_slice} loads the window of time associated with an event at a station (using {\tt extract\_timeslice\_at\_station}), as well as a window of noise preceding the signal itself. It computes the narrowband envelopes, finds the noise floor, and sets {\tt p\_time, s\_time, p\_phaseid, s\_phaseid, noise\_floor} for each trace in the segment.
\end{itemize}

\subsection{C}

A segment in C is represented by the {\tt Segment\_t} structure, defined in {\tt sigvisa.h}. This stores metadata as well as an array of (pointers to) {\tt Channel\_t} structures, which each store the same metadata as the parent segment (for convenience), and contain an array of pointers to of {\tt Trace\_t} structures corresponding to the individual traces, just as in the Python case. Since these traces will primarily correspond to different frequency bands, the array has size {\tt NUM\_BANDS} of them with the indices being given by frequency band constants defined in {\tt sigvisa.h}.

\subsubsection{Converting}

Ordinarily segments are loaded in Python, then converted to C as needed.

\begin{itemize}
\item {\tt trace\_to\_signal} in {\tt sigvisa.c} converts a Python trace into a C trace.
\item {\tt py\_segment\_to\_c\_segment} in {\tt sigvisa.c} converts a Python segment to an equivalent C segment.
\item {\tt py\_segment\_to\_c\_segments} in {\tt sigvisa.c} converts a list of Python segments to a an array of C segments.
\item {\tt py\_set\_signals} in {\tt sigvisa.c} (callable from Python as {\tt sigmodel.set\_signals(segments)}) takes a list of Python segments, converts them to C segments, and saves them as part of the C sigmodel structure.
\end{itemize}

\end{document}


\chapter{Bayesian Experiments}
This chapter is under \work.

\section{A Bayesian Quincunx}

\section{Conjugate Families}

\section{Bayesian Model Selection}


\declareIM{pg}{pre-G}{2021-05-21}{AOS using ComCam, commanding MT}{AOS MT}
\completeIM{\thisIM}{2021-08-27 \JIRA{DM}{31357}{}}

Executive Summary:

Process phosim Intra/Extra simulated ComCam images with the MTAOS.
Assuming that the images where acquired and ingested previously, drive the MTAOS through the data processing steps and produce corrections.

\textbf{Does not map to P6}

\subsection{Goals of IM}

Demonstrate ability to:

\begin{itemize}
\item Have MTAOS process images and publish
  \texttt{\{\{camera,m2\}Hexapod,\{m1m3,m2\}\}Correction} events
\end{itemize}

\subsection{Prerequisites}
\begin{itemize}
\item \IM{ppg}
\item Working version of MTAOS with access to a butler repo containing simulated ingested data.
\end{itemize}

\subsection{Procedure}
\begin{itemize}
\item Generate and ingest simulated images with desired pistons (e.g. $[0, -10, 0, 10, 0]$) and appropriate
  metadata
\item As for \IM{ppg} get data from the butler and generate Zernikes, using OCPS to run the pipeline
\item Trigger MTAOS to compute updates for Camera Hexapod, M2 Hexapod, M1M3, M2,
\item Check that desired events are in the EFD
\end{itemize}

\subsection{Acceptance Criteria}
\begin{itemize}
\item A member of SITCom must be able to carry out these operations at NCSA without installing any
  software for themselves.  This need not be done using RSP; a login shell on \eg \texttt{lsst-devl3}
  would be acceptable.
\item Confirm that the results are as expected.  This should be carried out using a notebook on the RSP,
  in which case it \textit{is} acceptable to require the tester to install the notebook from \texttt{git}
  themselves due to restrictions on the packages available on the RSP.
\end{itemize}


\chapter{{\bf Distributions of input variables for the global BDT}}
\label{sec:appendix1}
Comparison of the signal and background distributions of input variables used in the global BDT for 2011, 2012, 2015 and 2016 data taking conditions. Signal distributions are from simulated \bsmumu decays for each year that have passed the selection cuts in Table~\ref{tab:BDTpresel}. The background distributions are from \bbbarmumux decays in 2011, 2012, 2015 and 2016 data with $m_{\mu \mu} > 5447$ \mevcc and passing the selection cuts in Table~\ref{tab:BDTpresel}.
The input variables used for the global BDT are:
\begin{itemize}
\item long track isolation criteria;
\item VELO track isolation criteria;
\item $\sqrt{\Delta \phi^{2} + \Delta \eta^{2}}$, where $\Delta \phi$ is the difference in azimuthal angles of the muons and $\Delta \eta$ the difference in the pseudo-rapidity of the muons;
\item the smallest \chiIP with respect to the primary vertex of the \bsmumu of the muons;
\item \chivtx of the \bs;
\item \chiIP of the \bs with respect to the primary vertex; and
\item the angle, $\theta$, between the momentum vector of the \bs and the vector connecting the production and decay vertices of the \bs.
\end{itemize}

\begin{figure}
    \centering
    \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[trim={12cm 0 0 0},clip, width=\textwidth]{./Figs/Appendix1/signal_DeltaR.pdf}
%        \caption{ }
%        \label{fig:BDTsig}
    \end{subfigure}
    ~ %add desired spacing between images, e. g. ~, \quad, \qquad, \hfill etc.
      %(or a blank line to force the subfigure onto a new line)
    \begin{subfigure}[b]{0.48\textwidth}
       \includegraphics[width=\textwidth]{./Figs/Appendix1/signal_DIRA.pdf}
%        \caption{ }
%        \label{fig:BDTbkg}
    \end{subfigure}

 \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{./Figs/Appendix1/signal_muIPS.pdf}
%        \caption{ }
%        \label{fig:BDTsig}
    \end{subfigure}
    ~ %add desired spacing between images, e. g. ~, \quad, \qquad, \hfill etc.
      %(or a blank line to force the subfigure onto a new line)
    \begin{subfigure}[b]{0.48\textwidth}
       \includegraphics[width=\textwidth]{./Figs/Appendix1/signal_IPS.pdf}
%        \caption{ }
%        \label{fig:BDTbkg}
    \end{subfigure}

 \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{./Figs/Appendix1/signal_iso_velo.pdf}
 %       \caption{ }
 %       \label{fig:BDTsig}
    \end{subfigure}
    ~ %add desired spacing between images, e. g. ~, \quad, \qquad, \hfill etc.
      %(or a blank line to force the subfigure onto a new line)
    \begin{subfigure}[b]{0.48\textwidth}
       \includegraphics[width=\textwidth]{./Figs/Appendix1/signal_long_iso.pdf}
  %      \caption{ }
  %      \label{fig:BDTbkg}
    \end{subfigure}

 \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{./Figs/Appendix1/signal_end_vertex.pdf}
   %     \caption{ }
   %     \label{fig:BDTsig}
    \end{subfigure}
    ~ %add desired spacing between images, e. g. ~, \quad, \qquad, \hfill etc.
      %(or a blank line to force the subfigure onto a new line)

    \caption{Signal distributions for input variables for the global BDT for \bsmumu simulated decays in 2011, 2012, 2015 and 2016.}
    \label{fig:signalvars}
\end{figure}

\begin{figure}
    \centering
    \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[trim={12cm 0 0 0},clip,width=\textwidth]{./Figs/Appendix1/bkgnd_DeltaR.pdf}
     %   \caption{ }
     %   \label{fig:BDTsig}
    \end{subfigure}
    ~ %add desired spacing between images, e. g. ~, \quad, \qquad, \hfill etc.
      %(or a blank line to force the subfigure onto a new line)
    \begin{subfigure}[b]{0.48\textwidth}
       \includegraphics[width=\textwidth]{./Figs/Appendix1/bkgnd_DIRA.pdf}
    %    \caption{ }
    %    \label{fig:BDTbkg}
    \end{subfigure}

 \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{./Figs/Appendix1/bkgnd_muIPS.pdf}
      %  \caption{ }
      %  \label{fig:BDTsig}
    \end{subfigure}
    ~ %add desired spacing between images, e. g. ~, \quad, \qquad, \hfill etc.
      %(or a blank line to force the subfigure onto a new line)
    \begin{subfigure}[b]{0.48\textwidth}
       \includegraphics[width=\textwidth]{./Figs/Appendix1/bkgnd_IPS.pdf}
       % \caption{ }
       % \label{fig:BDTbkg}
    \end{subfigure}

 \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{./Figs/Appendix1/bkgnd_iso_velo.pdf}
        %\caption{ }
        %\label{fig:BDTsig}
    \end{subfigure}
    ~ %add desired spacing between images, e. g. ~, \quad, \qquad, \hfill etc.
      %(or a blank line to force the subfigure onto a new line)
    \begin{subfigure}[b]{0.48\textwidth}
       \includegraphics[width=\textwidth]{./Figs/Appendix1/bkgnd_long_iso.pdf}
        %\caption{ }
        %\label{fig:BDTbkg}
    \end{subfigure}

 \begin{subfigure}[b]{0.48\textwidth}
        \includegraphics[width=\textwidth]{./Figs/Appendix1/bkgnd_vertex.pdf}
        %\caption{ }
        %\label{fig:BDTsig}
    \end{subfigure}
    ~ %add desired spacing between images, e. g. ~, \quad, \qquad, \hfill etc.
      %(or a blank line to force the subfigure onto a new line)

    \caption{Background distributions for input variables from \bbbarmumux decays in 2011, 2012, 2015 and 2016 data with $m_{\mu \mu} > 5447$ \mevcc.}
    \label{fig:signalvars}
\end{figure}


% $Id$

\subsection{Design}

Regrid is designed to be called with Field or FieldBundle
arguments in order to utilize information embedded in
these objects.  For example, Regrid requires knowledge
of underlying grid information (both PhysGrid and DistGrid)
and of the relative location (staggering) of Fields on
the Grid.  In addition, Regrid uses any mask information
that may be associated with a Field.

Regrid is separated into a Create function and a
Run function. The Create function computes
interpolation weights and initializes communication
requirements for performing a regridding of a Field
from one Grid to another.  The Run function uses
a created Regrid object to perform the actual regridding
of Fields or FieldBundles.  The reason for the separation
is that in many cases, the initial creation is
expensive and re-used often throughout an application.

Because many methods are supported for regridding,
the main Create function branches to a specific
creation function based on the regrid method requested
(e.g. bilinear, conservative, spectral).  Each of
these regrid methods are in a separate module to
prevent the main Regrid module from becoming too
large.  The user is unaware of this hierarchy as the
top-level module provides a unified API.

The Regrid object created by the ... wait for it...
Regrid Create function contains a set of ``links''
which identify how a field at a point on the
destination grid is related to a field at a
point on the source grid.  As such, a ``link''
consists of a source address, a destination address
and a weight.  Because the Grids are generally
distributed very differently, the Regrid object
also contains communication information
for any data motion required for the regridding.


\chapter{Setting Up}
 \index{Setting Up}

To play Dungeon World, you'll need to gather yourself and 2–5 friends. A group of 4 to 5, including you, is best. Choose one person to be the GameMaster (hereafter: GM). Everyone else will be players, taking the role of the characters in the game. The players get to say what their characters say, think, and do. The GM describes everything else in the world.

You can play a single session or string together multiple sessions into a campaign. Plan accordingly. Each session will usually be a few hours and you'll be able to start playing right away within the first session.

You'll need to print some materials. Before you start a new game, print off at least:

\startitemize[1,packed]

\item A few copies of the basic and special moves

\item One copy of each class sheet

\item One copy each of the cleric and wizard spell sheets

\item One copy of the adventure sheet and GM moves

\stopitemize

Everyone at the table will need something to write with and some six-sided dice. Two dice is the minimum but two dice per player is a good idea.

You'll also need some specialized dice: four-sided, eight-sided, and ten-sided. One of each is enough but more is better; you won't have to pass them around so much.

\section{Using Dice}  \index{Using Dice} \index{Dice}

When talking about dice, {\bf dX}  means a dice with X sides (d4, d6, d8, etc.). A number before the dice size means "roll this many and add them together," so 2d6 means "roll two six-sided dice and add them together." Sometimes you'll add something to the roll, maybe a static number or the result of another roll, add that to the total. If there's a "·b" at the end instead of adding them together you take the best one ('b' for 'best'), so 2d6·b+1 means "roll two six-sided dice and take the best one and add one to it." A "·w" at the end means you take the worst result, so 2d6·w means "roll two six-sided dice and take the worst one."


\section{Arcane Armor}\label{spell:arcaneArmor}
\textbf{Cost:} 100 CP\\
\textbf{Requirements:}~\nameref{rune:arcana} I,~\nameref{rune:earth} I\\
\textbf{Spell, Active, Repeatable, Source(100 Gold)}\\
You target one willing creature in a range of \passus{6} that you can see.
For 8 AP and 5 Mana, you create a visible, magical force field around that creature's body that increases its Armor Reduction by 2 against Cutting, Piercing and Blunt Damage.
The force field can be active for up to your Intellect in hours.
Every time the creature would take Cutting, Piercing or Blunt damage, even if it is reduced to 0 by their total Armor Reduction, you lose 2 Mana in order to keep up the spell.
One creature can benefit from only one Arcane Armor at a time.\\
\\
Level Progression:\\
\\
II: 350 CP, 10 Mana for 4 Armor Reduction\\
III: 750 CP, 15 Mana for 6 Armor Reduction\\
IV: 1500 CP, 20 Mana for 8 Armor Reduction\\
V: 5000 CP, 25 Mana for 10 Armor Reduction\\

\subsection{Variation: Elemental Armor}\label{spell:elementalArmor}
\textbf{Cost:} 50 CP\\
\textbf{Requirements:} -\\
When casting Elemental Armor instead of Arcane Armor, the armor instead grants Armor Reduction against elemental damage types.
Pay the initial casting cost for each damage type that the Elemental Armor should defend against.\\
For example, a level II Elemental Armor that defends against Heat and Cold, the initial casting cost would be 20 Mana, and it would grant Heat and Cold Resistance 4.\\
A creature can benefit from multiple different Elemental and Arcane Armors at a time, but the effects don't stack.
Instead, only the highest gained Damage Reduction per Damage Type applies.


\section{EOS Calls}

\subsection{Initialization}

\subsection{{\tt advance\_timestep}}

\begin{description}
\item[Step 1.] {\em Define the average expansion at time $t^\nph$ and the new $w_0.$}\\ \\
if \runparam{dpdt\_factor} $>$ 0 then
%
\begin{itemize}
\item In {\tt makePfromRhoH}, we compute a thermodynamic $p^n$ for the volume discrepancy
term using $(\rho,h,X)^n$.
\end{itemize}
end if
\item[Step 2.] {\em Construct the provisional edge-based advective velocity, $\uadvone.$}
\item[Step 3.] {\em React the full state and base state through the first time interval
of $\dt/2.$}
\begin{itemize}
\item In {\tt react\_state} $\rightarrow$ {\tt burner}, we compute $c_p$ and $\xi_k$
for inputs to {\tt VODE} using $(\rho,T,X)^n$.
\item In {\tt react\_state}, we compute $T^{(1)}$ using $(\rho,h,X)^{(1)}$
(if \runparam{use\_tfromp} {\tt = F}) or $(\rho,X)^{(1)}$ and $p_0^n$ (if {\tt use\_tfromp = T})
\end{itemize}
if \runparam{evolve\_base\_state} {\tt = T} then
\begin{itemize}
\item In {\tt make\_gamma}, we compute $\Gamma_1$ using $(\rho,X)^{(1)}$ and $p_0^n$.
\end{itemize}
end if
\item[Step 4.] {\em Advect the base state, then the full state, through a time interval
of $\dt.$}\\ \\
if \runparam{use\_thermal\_diffusion} {\tt = T} then
\begin{itemize}
\item In {\tt advance} before {\tt make\_explicit\_thermal}, we compute the coefficients for
$\nabla\cdot(\kth/c_p)\nabla h + \cdots$ using $(\rho,T,X)^{(1)}$.
\item In {\tt enthalpy\_advance} $\rightarrow$ {\tt update\_scal}, we compute $h$ above
the \runparam{base\_cutoff\_density} using $(\rho,X)^{(2),*}$ and $p_0^{n+1,*}$.
\item In {\tt thermal\_conduct}, we compute $T^{(2),*}$ using $(\rho,h,X)^{(2),*}$
(if {\tt use\_tfromp = F}) or $(\rho,X)^{(2),*}$ and $p_0^{n+1,*}$ (if {\tt use\_tfromp = T}).
\end{itemize}
else
\begin{itemize}
\item In {\tt enthalpy\_advance} $\rightarrow$ {\tt update\_scal}, we compute $h$ above
the \runparam{base\_cutoff\_density} using $(\rho,X)^{(2),*}$ and $p_0^{n+1,*}$.
\item In {\tt enthalpy\_advance}, we compute $T^{(2),*}$ using $(\rho,h,X)^{(2),*}$
(if {\tt use\_tfromp = F}) or $(\rho,X)^{(2),*}$ and $p_0^{n+1,*}$ (if {\tt use\_tfromp = T}).
\end{itemize}
end if
\item[Step 5.] {\em React the full state through a second time interval of $\dt/2.$}
\begin{itemize}
\item In {\tt react\_state} $\rightarrow$ {\tt burner}, we compute $c_p$ and $\xi_k$
for inputs to {\tt VODE} using $(\rho,T,X)^{(2),*}$.
\item In {\tt react\_state}, we compute $T^{n+1,*}$ using $(\rho,h,X)^{n+1,*}$
(if {\tt use\_tfromp = F}) or $(\rho,X)^{n+1,*}$ and $p_0^{n+1,*}$ (if {\tt use\_tfromp = T}).
\end{itemize}
if {\tt evolve\_base\_state} then
\begin{itemize}
\item In {\tt make\_gamma}, we compute $\Gamma_1$ using $(\rho,X)^{n+1,*}$ and $p_0^{n+1,*}$.
\end{itemize}
end if
\item[Step 6.] {\em Define a new average expansion rate at time $t^\nph.$}\\ \\
if \runparam{use\_thermal\_diffusion} then
\begin{itemize}
\item In {\tt advance} before {\tt make\_explicit\_thermal}, we compute the coefficients for
$\nabla\cdot(\kth/c_p)\nabla h + \cdots$ using $(\rho,T,X)^{n+1,*}$.
\end{itemize}
end if
\begin{itemize}
\item In {\tt make\_S}, we compute thermodynamic variables using $(\rho,T,X)^{n+1,*}$.
\end{itemize}
if \runparam{dpdt\_factor} $>$ 0 then
\begin{itemize}
\item In {\tt makePfromRhoH}, we compute a thermodynamic $p^{n+1,*}$ for the volume
discrepancy term using $(\rho,h,X)^{n+1,*}$.
\end{itemize}
end if
\item[Step 7.] {\em Construct the final edge-based advective velocity, $\uadvtwo.$}
\item[Step 8.] {\em Advect the base state, then the full state, through a time interval
of $\dt.$}\\ \\
if \runparam{use\_thermal\_diffusion} = T then
\begin{itemize}
\item In {\tt enthalpy\_advance} $\rightarrow$ {\tt update\_scal}, we compute $h$ above
the \runparam{base\_cutoff\_density} using $(\rho,X)^{(2)}$ and $p_0^{n+1}$.
\item In {\tt advance} before {\tt thermal\_conduct}, we compute the coefficients for
$\nabla\cdot(\kth/c_p)\nabla h + \cdots$ using $(\rho,T,X)^{(2),*}$.
\item In {\tt thermal\_conduct}, we compute $T^{(2)}$ using $(\rho,h,X)^{(2)}$
(if \runparam{use\_tfromp} {\tt = F}) or $(\rho,X)^{(2)}$ and $p_0^{n+1}$ (if {\tt use\_tfromp} {\tt = T}).
\end{itemize}
else
\begin{itemize}
\item In {\tt enthalpy\_advance} $\rightarrow$ {\tt update\_scal}, we compute $h$ above
the {\tt base\_cutoff\_density} using $(\rho,X)^{(2)}$ and $p_0^{n+1}$.
\item In {\tt enthalpy\_advance}, we compute $T^{(2)}$ using $(\rho,h,X)^{(2)}$
(if {\tt use\_tfromp} {\tt = F}) or $(\rho,X)^{(2)}$ and $p_0^{n+1}$ (if {\tt use\_tfromp = T}).
\end{itemize}
end if
\item[Step 9.] {\em React the full state and base state through a second time interval
of $\dt/2.$}
\begin{itemize}
\item In {\tt react\_state} $\rightarrow$ {\tt burner}, we compute $c_p$ and $\xi_k$
for inputs to {\tt VODE} using $(\rho,T,X)^{(2)}$.
\item In {\tt react\_state}, we compute $T^{n+1}$ using $(\rho,h,X)^{n+1}$
(if {\tt use\_tfromp = F}) or $(\rho,X)^{n+1}$ and $p_0^{n+1}$ (if {\tt use\_tfromp = T}).
\end{itemize}
if \runparam{evolve\_base\_state} {\tt = T} then
\begin{itemize}
\item In {\tt make\_gamma}, we compute $\Gamma_1$ using $(\rho,X)^{n+1}$ and $p_0^{n+1}$.
\end{itemize}
end if
\item[Step 10.] {\em Compute $S^{n+1}$ for the final projection.}\\ \\
if {\tt make\_explicit\_thermal} then
\begin{itemize}
\item In {\tt advance} before {\tt make\_explicit\_thermal}, we compute the coefficients for
$\nabla\cdot(\kth/c_p)\nabla h + \cdots$ using $(\rho,T,X)^{n+1}$.
\end{itemize}
end if
\begin{itemize}
\item In {\tt make\_S}, we compute thermodynamic variables using $(\rho,T,X)^{n+1}$.
\end{itemize}
\item[Step 11.] {\em Update the velocity.}\\ \\
if \runparam{dpdt\_factor} $>$ 0 then
\begin{itemize}
\item In {\tt makePfromRhoH}, we compute a thermodynamic $p^{n+1}$ for the volume
discrepancy term using $(\rho,h,X)^{n+1}$.
\end{itemize}
end if
\item[Step 12.] {\em Compute a new $\dt.$}
\end{description}

\subsection{{\tt make\_plotfile}}

\section{Temperature Usage}

\subsection{{\tt advance\_timestep}}
\begin{description}
\item[Step 1.] {\em Define the average expansion at time $t^\nph$ and the new $w_0.$}
\item[Step 2.] {\em Construct the provisional edge-based advective velocity, $\uadvone.$}
\item[Step 3.] {\em React the full state and base state through the first time interval
of $\dt/2.$}
\begin{itemize}
\item In {\tt react\_state} $\rightarrow$ {\tt burner}, we compute $c_p$ and $\xi_k$
for inputs to {\tt VODE} using $(\rho,T,X)^n$.
\item In {\tt react\_state}, we compute $T^{(1)}$ using $(\rho,h,X)^{(1)}$
(if \runparam{use\_tfromp} {\tt = F}) or $(\rho,X)^{(1)}$ and $p_0^n$ (if {\tt use\_tfromp = T}).
\end{itemize}
\item[Step 4.] {\em Advect the base state, then the full state, through a time interval
of $\dt.$}\\ \\
if \runparam{use\_thermal\_diffusion} {\tt = T} then
\begin{itemize}
\item In {\tt advance} before {\tt make\_explicit\_thermal}, we compute the coefficients for
$\nabla\cdot(\kth/c_p)\nabla h + \cdots$ using $(\rho,T,X)^{(1)}$.
\item In {\tt thermal\_conduct}, we compute $T^{(2),*}$ using $(\rho,h,X)^{(2),*}$
(if {\tt use\_tfromp = F}) or $(\rho,X)^{(2),*}$ and $p_0^{n+1,*}$ (if {\tt use\_tfromp = T}).
\end{itemize}
else
\begin{itemize}
\item In {\tt enthalpy\_advance}, we compute $T^{(2),*}$ using $(\rho,h,X)^{(2),*}$
(if {\tt use\_tfromp = F}) or $(\rho,X)^{(2),*}$ and $p_0^{n+1,*}$ (if {\tt use\_tfromp = T}).
\end{itemize}
end if
\item[Step 5.] {\em React the full state through a second time interval of $\dt/2.$}
\begin{itemize}
\item In {\tt react\_state} $\rightarrow$ {\tt burner}, we compute $c_p$ and $\xi_k$
for inputs to {\tt VODE} using $(\rho,T,X)^{(2),*}$.
\item In {\tt react\_state}, we compute $T^{n+1,*}$ using $(\rho,h,X)^{n+1,*}$
(if {\tt use\_tfromp = F}) or $(\rho,X)^{n+1,*}$ and $p_0^{n+1,*}$ (if {\tt use\_tfromp = T}).
\end{itemize}
\item[Step 6.] {\em Define a new average expansion rate at time $t^\nph.$}\\ \\
if \runparam{use\_thermal\_diffusion} {\tt = T} then
\begin{itemize}
\item In {\tt advance} before {\tt make\_explicit\_thermal}, we compute the coefficients for
$\nabla\cdot(\kth/c_p)\nabla h + \cdots$ using $(\rho,T,X)^{n+1,*}$.
\end{itemize}
end if
\begin{itemize}
\item In {\tt make\_S}, we compute thermodynamic variables using $(\rho,T,X)^{n+1,*}$.
\end{itemize}
\item[Step 7.] {\em Construct the final edge-based advective velocity, $\uadvtwo.$}
\item[Step 8.] {\em Advect the base state, then the full state, through a time interval
of $\dt.$}\\ \\
if {\tt use\_thermal\_diffusion = T} then
%
\begin{itemize}
\item In {\tt advance} before {\tt thermal\_conduct}, we compute the coefficients for
$\nabla\cdot(\kth/c_p)\nabla h + \cdots$ using $(\rho,T,X)^{(2),*}$.
\item In {\tt thermal\_conduct}, we compute $T^{(2)}$ using $(\rho,h,X)^{(2)}$
(if {\tt use\_tfromp = F}) or $(\rho,X)^{(2)}$ and $p_0^{n+1}$ (if {\tt use\_tfromp = T}).
\end{itemize}
else
\begin{itemize}
\item In {\tt enthalpy\_advance}, we compute $T^{(2)}$ using $(\rho,h,X)^{(2)}$
(if {\tt use\_tfromp = F}) or $(\rho,X)^{(2)}$ and $p_0^{n+1}$ (if {\tt use\_tfromp = T}).
\end{itemize}
end if
\item[Step 9.] {\em React the full state and base state through a second time interval
of $\dt/2.$}
\begin{itemize}
\item In {\tt react\_state} $\rightarrow$ {\tt burner}, we compute $c_p$ and $\xi_k$
for inputs to {\tt VODE} using $(\rho,T,X)^{(2)}$.
\item In {\tt react\_state}, we compute $T^{n+1}$ using $(\rho,h,X)^{n+1}$
(if {\tt use\_tfromp = F}) or $(\rho,X)^{n+1}$ and $p_0^{n+1}$ (if {\tt use\_tfromp = T}).
\end{itemize}
\item[Step 10.] {\em Compute $S^{n+1}$ for the final projection.}\\ \\
if {\tt make\_explicit\_thermal} then
\begin{itemize}
\item In {\tt advance} before {\tt make\_explicit\_thermal}, we compute the coefficients for
$\nabla\cdot(\kth/c_p)\nabla h + \cdots$ using $(\rho,T,X)^{n+1}$.
\end{itemize}
end if
\begin{itemize}
\item In {\tt make\_S}, we compute thermodynamic variables using $(\rho,T,X)^{n+1}$.
\end{itemize}
\item[Step 11.] {\em Update the velocity.}
\item[Step 12.] {\em Compute a new $\dt.$}
\end{description}


\section{Introduction}
\label{sec:intro}

With the discovery of a new scalar particle the Higgs boson~\cite{HIGG-2012-27,CMS-HIG-12-028} at the LHC~\cite{Evans:2008zzb} by the ATLAS and CMS collaborations with a mass of 125 GeV~\cite{HIGG-2016-33,HIGG-2013-12,HIGG-2014-14,CMS-HIG-14-009}, the whole content of the Standard Model (SM) of particle physics became complete. A priority of the ATLAS and CMS collaborations has been to better understand its properties and couplings.
After the EWSB and the higgs field acquires the vev, the higgs potential obtained in Equation.. can be obtained as following:
\begin{align}
	V(\phi) \rightarrow V(\phi)_{\text{EWSB}} =  -\lambda \nu^2 h^2 - \lambda \nu h^3 - \frac{1}{4} \lambda h^4 + \text{const.}
	\label{eq:higgs_pot}
\end{align}
The first term of the above equation is the Higgs mass term, and the remaining are the trilinear and quadri-linear Higgs-self couplings,
\begin{align}
	\underbrace{m_{h} = \sqrt{ - 2 \mu^2} = \sqrt{ 2 \lambda v^2 }}_\text{Higgs boson mass} \hspace{1cm} \underbrace{\lambda_{hhh} \propto \frac{m_h^2}{v} \hspace{1cm} \lambda_{hhhh} \propto \frac{m_h^2}{v^2}}_\text{Trilinear and quadri-linear Higgs self-couplings}.
	\label{eq:higgs_self_couplings}
\end{align}
A measurements of this couplings would therefore give a hint about the actual structure of the potential, whose shape can have  theoretical consequences. The quartic Higgs coupling, $\lambda_{hhhh}$, can not be measured at LHC since the cross-section of triple Higgs production is small~\cite{PhysRevD.74.113008}~\cite{PhysRevD.93.035026}, while the trilinear coupling can be probed directly in Higgs pair production.

The trilinear coupling leads to non-resonant pair production of Higgs bosons, where an off-shell
Higgs decays to a pair of Higgs bosons, the leading production mechanism being $gg$-fusion.
Direct observation of Higgs pair production would lead to measurements directly sensitive to
$\lambda$ but in the SM there are competing diagrams, proceeding via quark (re: top quark) loops
that are instead senstive to the Yukawa coupling of the Higgs rather than the trilinear coupling
$\lambda$. These \dihiggs production mechanisms are illustrated in the Feynman diagrams presented
in Figure \ref{fig:sm_nonres_hh}. Not only does the quark-loop induced process present
itself as an irreducible background to the process senstive to the Higgs self-coupling, but
it interfereces \textit{destructively} with the latter, making the observation of this
type of Higgs pair production more challenging.

% Due to its lower cross section, diHiggs production is not expected to be observed in the current data taking setup, however it is possible to define limits on the measurement to constrain the BSM physics theories.
\begin{figure}[!htb]
    \centering
    \includegraphics[width=0.9\textwidth]{figures/sm_nonres_hh_production}
    \caption{Representative diagrams that contribute to non-resonant \hh production.
    \textit{Left}: Diagram that is sensitive to the trilinear coupling, $\lambda$.
    \textit{Right}: Box diagram that interferese destructively with the $\lambda$-sensitive
    process.}
    \label{fig:sm_nonres_hh}
\end{figure}

As a result of the destructive interference, and the already relatively large Higgs mass of
125 \GeV, the SM \dihiggs production has a total cross-section of $\sim 33.4$ fb ~\cite{CERN-YR-4}
at a \pp center-of-mass collision energy of 13 \TeV. The inclusive cross-section for the pair-production
of top quarks, which will be one of the dominant SM backgrounds in the present
analysis, is nearly 1000 pb, or $1\times10^6$ fb ~\cite{TOPQ-2015-09, ATLAS-CONF-2015-049}. That
of \textit{single} Higgs production is $\sim 50$ pb, or $5 \times 10^4$ fb ~\cite{ATLAS-CONF-2015-060}.

Furthermore, enhancements to the di-Higgs production rate, either non-resonantly or through a resonance, may be observable with the full Run2 dataset and would point to new physics beyond the Standard Model, making such analyses interesting now. The wide class of two Higgs double models (2HDM) predict an altered and enlarged
Higgs sector from which the currently Higgs is built.
The Minimal Supersymmetric Standard Model
(MSSM) is a  class of 2HDM. For the latter set, one such model is a Randall-Sundrum
type graviton or the lightest Kaluza-Klein excitation which have masses of at least
$2\times$ the mass of the SM Higgs boson. The presence of such BSM scenarios would act to alter
the measured value of $\lambda$ with respect to that of the SM, potentially enlarging it.
As a result, early evidence for the pair-production of Higgs bosons within the current
Run-2 dataset may indicate the presence of new physics without having to resort to
precision measurements of $\lambda$. Examples of such decay
scenarios are illustrated in example Feynman diagrams in Figure \ref{fig:2hdm_feynman_diagrams}.

\begin{figure}[!htb]
    \centering
    \includegraphics[width=0.8\textwidth]{figures/mg5_hh_heavy_scalar.png}
    \includegraphics[width=0.8\textwidth]{figures/mg5_hh_2hdm_cp_even.png}
    \caption
    {
        Diagrams contributing to enhanced $hh$ production scenarios.
        \textit{Top}: A heavy scalar, $H$, that couples to the Standard Model Higgs
        boson, $h$, contributes to the Standard Model processes (left two diagrams).
        \textit{Bottom}: CP-even diagrams in 2HDM scenario that contribute to enhanced
        non-resonant production of Standard Model Higgs bosons as well as resonant
        channels with the heavy CP-even Higgs, $H^0$, decaying to the Standard Model
        low-mass CP-even Higgs, $h$.
    }
    \label{fig:2hdm_feynman_diagrams}
\end{figure}

In addition to the hh production mechanisms described above, the searches for evidence of an dditional extended Higgs sector model that introduces two new heavy Higgs bosons, X and S~\cite{vonBuddenbrock:2016rmr}. In this model, $X$ couples strongly to both $S$ and $h$. $S$ couples weakly to SM particles, suppressing direct production, but has the same mass-dependent branching ratios as $h$.

Searches for resonant Higgs pair production have been performed in a number of final states, $b\bar{b}b\bar{b}$, $b\bar{b}\tau^{\pm}\tau^{\mp}$, $b\bar{b}\gamma\gamma$, $W^{\pm}W^{\mp *}\gamma\gamma$, $b\bar{b}VV$ (With $V$ either $Z$ or $W$) and $W^+W^-W^+W^-$ at $\sqrt{s}=8\,\TeV$ and $\sqrt{s}=13\,\TeV$ by ATLAS~\cite{HIGG-2013-33,Aad:2019uzh} and CMS collaborations~\cite{CMS-HIG-13-032,CMS-HIG-15-013,PhysRevLett.122.121803} including the combination of multiple final states.

In this note, the searches for the \dihiggs non-resonant production in multilepton final states is described. Typically the decay modes of \dihiggs of $W^+W^-W^+W^-$, $ZZ^*bb$, $VV\tauh\tauh$, $\tauh\tauh\tauh\tauh$, $ZZZZ$ are the dominant ones which corresponds to 5.7\% of all \dihiggs decays. Additionally, $\gamma\gamma + X$ final states are studied which corresponds to 0.13\% of \dihiggs decays. Simple extension of the SM by introducing two new scalars $gg \rightarrow X \rightarrow SH$ signature is investigated.

This note is organised as follows: Section~\ref{sec:dataset} describes the Monte Carlo (MC) samples as well as the
recorded dataset used in this analysis. The object definition and event preselection are detailed in Section~\ref{sec:objselec}.
The signal region definitions and the multivariate analysis discriminants are described in Section~\ref{sec:eventselec}.
The reducible background estimates using data-driven method are explained in Section~\ref{sec:nonpromptbkg}.
%Section~\ref{sec:bkgval} shows the data and MC agreement of the prompt lepton background contributions in
various validation regions.
Theoretical and experimental systematic uncertainties are described in Section~\ref{sec:systematics}.
An overview of the signal regions before the fit can be found in Section~\ref{sec:signalregion}.
Section~\ref{sec:results} describes the expected fit results for each individual channel and the
expected and observed fit results for the combination.
Finally, the conclusions are summarised in Section~\ref{sec:conclusion}.


\section{Introduction}\label{sec:intro}

Fish population (aka "stock")  assessment models determine the impact of past fishing on the historical and current abundance of the population, evaluate sustainable rates of removals (catch), and project future levels of catch reflecting one or more risk-averse catch rules.  These catch rules are codified in regional Fishery Management Plans according to requirements of the Sustainable Fisheries Act.  In the U.S., approximately 500 federally managed fish and shellfish populations are managed under approximately 50 Fishery Management Plans.  About 200 of these populations are assessed each year, based on a prioritized schedule for their current status. Despite this, many minor species have never been quantitatively assessed.  Although the pace is slower than that for weather forecasting, fish stock assessments are operational models for fisheries management.

Assessment models typically assimilate annual catches, data on fish abundance from diverse surveys and fishery sources, and biological information regarding fish body size and proportions at age.  A suite of models is available depending on the degree of data availability and unique characteristics of the fish population or its fishery.  Where feasible, environmental time series are used as indicators of changes in population or observation processes, especially to improve the accuracy of the projections of abundance and sustainable catch into the future.  Such linkages are based principally on correlations given the challenge of conducting field observations on an appropriate scale.  The frontier of model development is in the rapid estimation of parameters to include random temporal effects, in the simultaneous modeling of a suite of interacting species, and in the explicit treatment of the spatial distribution of the population.

Assessment models are loosely coupled to other models. For example, an ocean-temperature or circulation model or benthic-habitat map may be directly included in the pre-processing of the fish abundance survey.  A time series of a derived ocean factor, like the North Atlantic Oscillation, can be included as an indicator of a change in a population process.  Output of a multi-decadal time series of derived fish abundance can be an input to ecosystem and economic models to better understand cumulative impacts and benefits.

Stock Synthesis is an age- and size-structured assessment model in the class of models termed integrated analysis models. Stock Synthesis has evolved since its initial inception in order to model a wide range of fish populations and dynamics.  The most recent major revision to Stock Synthesis occurred in 2016, when version 3.30 was introduced. This new version of Stock Synthesis required major revisions to the input files relative to earlier versions (see the \hypertarget{ConvIssues}{Converting Files} section for more information). The acronym for Stock Synthesis has evolved over time with earlier versions being referred to as SS2 (SS v.2.+), SS3 (SS v.3.+), or just simply SS.  The current official abbreviation of Stock Synthesis is SS3 for all versions greater than SS v.3.+.

SS3 has a population sub-model that simulates a stock's growth, maturity, fecundity, recruitment, movement, and mortality processes, an observation sub-model estimates expected values for various types of data, a statistical sub-model characterizes the data’s goodness of fit and obtains best-fitting parameters with associated variance, and a forecast sub-model projects needed management quantities.  SS3 outputs the quantities, with confidence intervals, needed to implement risk-averse fishery control rules. The model is coded in C++ with parameter estimation enabled by automatic differentiation (\href{http://www.admb-project.org}{admb}).  Windows, Linux, and iOS versions are available.  Output processing and associated tools are in R, and a graphical interface is in QT.  SS is available from NOAA’s VLAB. The rich feature set in SS allows it to be configured for a wide range of situations.  SS3 has become the basis for a large fraction of U.S. assessments and many other assessments around the world.

This manual provides a guide for using SS. The guide contains a description of the input and output files and usage instructions. An overview and technical description of the model itself is in \citet{methotstock2013}. However, SS3 has continued to evolve and grow since the publication in 2013, with this manual reflecting the most up to date information regarding SS.  The model and a graphical user interface are available from the NOAA \href{https://vlab.noaa.gov/group/stock-synthesis/home}{VLAB}. The VLAB site also provides a user forum for posting questions and for accessing various additional materials.  An output processor package, r4ss, in R is available for download from CRAN or \href{https://github.com/r4ss/r4ss}{GitHub}.

Additional guidance for new users can be found in the SS3 \href{https://vlab.noaa.gov/group/stock-synthesis/document-library}{document library} within the NOAA VLAB website.  The "Begin Here - Introduction to Stock Synthesis" folder located in the Document Library contains step-by-step guidance for running Stock Synthesis.

\pagebreak

\section{File Organization}\label{FileOrganization}
	\subsection{Input Files}
	\begin{enumerate}
		\item starter.ss: required file containing filenames of the data file and the control file plus other run controls (required).
		\item datafile: file containing model dimensions and the data (required)
		\item control file: file containing set-up for the parameters (required)
		\item forecast.ss: file containing specifications for reference points and forecasts (required)
		\item ss.par: previously created parameter file that can be read to overwrite the initial parameter values in the control file (optional)
		\item wtatage.ss: file containing empirical input of body weight by fleet and population and empirical fecundity-at-age (optional)
		\item runnumber.ss: file containing a single number used as run number in output to CumReport.sso and in the processing of profilevalues.ss (optional)
		\item profilevalues.ss: file contain special conditions for batch file processing (optional)
	\end{enumerate}

	\subsection{Output Files}
	\begin{enumerate}
		\item data.ss\textunderscore new: Contains a user-specified number of datafiles, generated through a parametric bootstrap procedure, and written sequentially to this file.
		\item control.ss\textunderscore new: Updated version of the control file with final parameter values replacing the initial parameter values.
		\item starter.ss\textunderscore new: New version of the starter file with annotations.
		\item Forecast.ss\textunderscore new: New version of the forecast file with annotations.
		\item warning.sso: This file contains a list of warnings generated during program execution.
		\item echoinput.sso: This file is produced while reading the input files and includes an annotated echo of the input. The sole purpose of this output file is debugging input errors.
		\item Report.sso: This file is the primary report file.
		\item ss\_summary.sso: Output file that contains all the likelihood components, parameters, derived quantities, total biomass, summary biomass, and catch. This file offers an abridged version of the report file that is useful for quick model evaluation. This file is only available in SS3 v.3.30.08.03 and greater.
		\item CompReport.sso: Observed and expected composition data in a list-based format.
		\item Forecast-report.sso: Output of management quantities and for forecasts.
		\item CumReport.sso: This file contains a brief version of the run output, output is appended to current content of file so results of several runs can be collected together.  This is useful when a batch of runs is being processed.
		\item Covar.sso: This file replaces the standard ADMB ss.cor with an output of the parameter and derived quantity correlations in database format.
		\item ss.par: This file contains all estimated and fixed parameters from the model run.
		\item ss.std, ss.rep, ss.cor etc.:  Standard ADMB output files.
		\item checkup.sso: Contains details of selectivity parameters and resulting vectors.  This is written during the first call of the objective function.
		\item Gradient.dat: New for SS3 v.3.30, this file shows parameter gradients at the end of the run.
		\item rebuild.dat: Output formatted for direct input to Andre Punt's rebuilding analysis package.  Cumulative output is output to REBUILD.SS (useful when doing MCMC or profiles).
		\item SIS\_table.sso: Output formatted for reading into the NMFS Species Information System.
		\item Parmtrace.sso: Parameter values at each iteration.
		\item posteriors.sso, derived\_posteriors.sso, posterior\_vectors.sso: Files associated with MCMC.
	\end{enumerate}

	\subsection{Auxiliary Files}
	These files are additional files (e.g. excel files) which allow for exploration or understanding of specific parameterization which can assist in selecting appropriate starting values.  These files are available for download from the Vlab website.
	\begin{enumerate}
		\item SS3-OUTPUT.xls: Excel file with macros to read report.sso and display results.
		\item SS\_330\_Control\_Setup.xls:
		\begin{enumerate}
			\item Shows how to setup an example control file for SS.
		\end{enumerate}
		\item SS\_330\_Data\_Input.xls:
		\begin{enumerate}
			\item Shows how to setup an example data input file for SS.
		\end{enumerate}
		\item SS\_330\_Starter\&Forecast.xls:
		\begin{enumerate}
			\item Shows how to setup an example starter and forecast input files for SS.
		\end{enumerate}

	\end{enumerate}

\pagebreak

\section{Starting Stock Synthesis}
SS3 is typically run through the command line interface, although it can also be called from another program, R, the SS-GUI or a script file (such as a DOS batch file). SS3 is compiled for Windows, Mac, and Linux operating systems. The memory requirements depend on the complexity of the model you run, but in general, SS3 will run much slower on computers with inadequate memory. See the Running Stock Synthesis section on page \pageref{sec:RunningSS} for additional notes on methods of running SS3.

Communication with the program is through text files.  When the program first starts, it reads the file starter.ss, which typically must be located in the same directory from which SS3 is being run.  The file starter.ss contains required input information plus references to other required input files, as described in the File Organization section on page \pageref{FileOrganization}.  The names of the control and data files must match the names specified in the starter.ss file.  File names, including starter.ss, are case-sensitive on Linux and Mac systems but not on Windows. The echoinput.sso file outputs how the executable read each input file and can be used for troubleshooting when trying to setup a model correctly.  Output from SS3 consists of text files containing specific keywords.  Output processing programs, such as the SS3 GUI, Excel, or R can search for these keywords and parse the specific information located below that keyword in the text file.

\pagebreak


\section{On-demand evaluation for KBP}
\label{sec:application}
Applying the on-demand evaluation framework to a task requires us to answer three questions:
\begin{enumerate}
    \itemsep0pt
  \item What is the desired distribution over system predictions $p_i$?
  \item How do we label an instance $x$, i.e.\ check if $x \in \sY$?
  \item How do we sample from the unknown set of true instances $x \sim p_0$?
\end{enumerate}
In this section, we present practical implementations for knowledge base population.

\subsection{Sampling from system predictions}
% 1. what is the point of this distribution?
% - prevent over sampling instances.
%\ac{@PL:\@ I think that the role of $p_i$ is really very subtle, but very important. I hope the below paragraph highlights the paradigm shift in how one has to think about on-demand evaluation.}
Both the official TAC-KBP evaluation and the on-demand evaluation we propose use micro-averaged precision and recall as metrics. However, in the official evaluation, these metrics are computed over a fixed set of evaluation entities chosen by LDC annotators, resulting in two problems: (a) defining evaluation entities requires human intervention and (b) typically a large source of variability in evaluation scores comes from not having enough evaluation entities (see e.g.\ \citep{webber2010measurement}). In our methodology, we replace manually chosen evaluation entities by sampling entities from each system’s output according $p_i$. In effect, $p_i$ makes explicit the decision process of the annotator who chooses evaluation entities.

%\ac{@PL:\@ This paragraph goes into details, but I felt that it was useful to illustrate how we tailored a distribution to meet our evaluation goals.}
Identifying a reasonable distribution $p_i$ is an important implementation decision that depends on what one wishes to evaluate.
Our goal for the on-demand evaluation service we have implemented is to ensure that KBP systems are fairly evaluated on diverse subjects and predicates, while at the same time, ensuring that entities with multiple relations are represented to measure completeness of knowledge base entries.
As a result, we propose a distribution that is inversely proportional to the frequency of the subject and predicate and is proportional to the number of unique relations identified for an entity (to measure knowledge base completeness).
See \refapp{implementation} in the supplementary material for an analysis of this distribution and a study of other potential choices.

%Recall that a KBP system predicts a set of relation instances of the form (\entity{subject}, \relation{predicate}, \entity{object}, \textsc{provenance}).
%Simply choosing $p_i$ to be the uniform distribution over this set causes our evaluation metrics to be be dominated by a few common predicates (e.g.\ professional titles) or subject entities (e.g.\ ``United States of America'').
%Instead, we propose the following distribution which we reasonably approximates our goals:\footnote{
%See \refapp{implementation} for an analysis of this distribution and a study of other potential choices.}
%\begin{align*}
%  p_i(x) &\eqdef \frac{\operatorname{\#uniq-relns}_i(x)}{\operatorname{\#pred-insts}_i(x) \operatorname{\#subj-insts}_i(x)},
%\end{align*}
%where $\operatorname{\#uniq-relns}_i(x)$ measures the number of unique relations (\entity{subject}, \relation{predicate}, \entity{object} triples) with the same \entity{subject} as $x$ in the system's output $X_i$,
%$\operatorname{\#pred-insts}_i(x)$ measures the number of instances with the same \relation{predicate} as $x$ in $X_i$ and
%$\operatorname{\#subj-insts}_i(x)$ measures the number of instances with the same \relation{subject} as $x$ in $X_i$.

%%The choice of distribution over this set determines how much we put weight on rare predicates and subject entities in our evaluation metric. % in our precision score. % PL: for recall
%We therefore propose two instance distributions that evaluate precision and recall scores macro-averaged over predicates and subject entities, respectively:
%\begin{align*}
%  p_i^\text{(pred)}(x) &\eqdef \frac{1}{\operatorname{\#preds}(X_i)} \frac{1}{\operatorname{\#insts}(X_i, \relation{pred}(x))} \\
%  p_i^\text{(subj)}(x) &\eqdef \frac{1}{\operatorname{\#subjs}(X_i)} \frac{1}{\operatorname{\#insts}(X_i, \relation{subj}(x))}
%\end{align*}
%where $\operatorname{\#preds}$, $\operatorname{\#nsubjs}$ specifies the number of predicates, subject entities in $X_i$ respectively and $\operatorname{\#insts}$ the number of instances with the given predicate or subject entity.
%See \refapp{implementation} for more details.

\subsection{Labeling predicted instances}
We label predicted relation instances by presenting the instance's provenance to crowdworkers
  and asking them to identify if a relation holds between the identified subject and object mentions (\reffig{relation-interface}).
  Crowdworkers are also asked to link the subject and object mentions to their canonical mentions within the document and to pages on Wikipedia, if possible, for entity linking.
On average, we find that crowdworkers are able to perform this task in about 20 seconds, corresponding to about \$0.05 per instance.
We requested 5 crowdworkers to annotate a small set of 200 relation instances from the 2015 TAC-KBP corpus
and measured a substantial inter-annotator agreement with a Fleiss' kappa of 0.61 with 3 crowdworkers and 0.62 with 5. % \pl{why does it go up with more annotators?}.

%\pl{what does it mean to annotate a relation - get all the relation instances for that relation or just one?}.
\begin{figure*}
  \centering

  \begin{subfigure}{0.49\textwidth}
  \centering
    \includegraphics[width=\textwidth]{figures/interface/relation-interface}
    \caption{\label{fig:relation-interface}}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.49\textwidth}
  \centering
    \includegraphics[width=\textwidth]{figures/interface/extraction-interface}
    \caption{\label{fig:entity-interface}}
  \end{subfigure} \\

  \begin{subfigure}{0.49\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/simulation/simulation-p}
    \caption{}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.49\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/simulation/simulation-r}
    \caption{}
  \end{subfigure} \\

  \begin{subfigure}{0.49\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/simulation/simulation-n}
    \caption{}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.49\textwidth}
    \centering
    \input{evaluation.table}
    %\includegraphics[width=\columnwidth]{figures/kbp2016/kbp2016_f1}
    \vfill
    \caption{\label{fig:evaluation-results}}
  \end{subfigure}

  \caption{\label{fig:simulation}
  \textbf{(a, b):} Interfaces for annotating relations and entities respectively.
  \textbf{(c, d):}
  A comparison of bias for the pooling, simple and joint estimators on the TAC KBP 2015 challenge.
  Each point in the figure is a mean of 500 repeated trials; dotted lines show the 90\% quartile.
  %The pooling based method uses between 5,000 and 6,000 labeled instances, while the sampling based methods use
  %approximately 150 samples from each system.
  %Dashed trend lines indicate the mean bias of the estimation method:
  %Unbiased estimates lie on the $y = x$ line.
  Both the simple and joint estimators are unbiased, and the joint estimator is able to significantly reduce variance.
  \textbf{(e):}
  A comparison of the number of samples used to estimate scores under the fixed and adaptive sample selection scheme.
  %In the simulation, the top 40 systems were evaluated in randomized order to achieve a target variance equal to that obtained with 500 samples for a single system.
  Each faint line shows the number of samples used during a single trial, while solid lines show the mean over 100 trials.
  The dashed line shows a square-root relationship between the number of systems evaluated and the number of samples required.
  %\ac{Note: fitting with a cubic gives $x = 0.01 y^3 -0.1y^2 + 1.8 y - 1.1$ with $R=1.0$.}
  Thus joint estimation combined with adaptive sample selection can reduce the number of labeled annotations required by an order of magnitude.
  \textbf{(f):}
Precision ($P$), recall ($R$) and \fone{} scores from a pilot run of our evaluation service for ensembles of a rule-based system (R), a logistic classifier (L) and a neural network classifier (N) run on the TAC KBP 2016 document corpus.
  }
\end{figure*}

\subsection{Sampling true instances}
Sampling from the set of true instances $\sY$ is difficult because we can't even enumerate the elements of $\sY$.
As a proxy, we assume that relations are identically distributed across documents and have crowdworkers annotate a random subset of documents for relations using an interface we developed (\reffig{entity-interface}).
Crowdworkers begin by identifying every mention span in a document.
  For each mention, they are asked to identify its type, canonical mention within the document
  and associated Wikipedia page if possible.
They are then presented with a separate interface to label predicates between pairs of mentions within a sentence that were identified earlier.

We compare crowdsourced annotations against those of expert annotators using data from the TAC KBP 2015 EDL task on 10 randomly chosen documents.
We find that 3 crowdworkers together identify 92\% of the entity spans identified by expert annotators, while 7 crowdworkers together identify 96\%.
When using a token-level majority vote to identify entities, 3 crowdworkers identify about 78\% of the entity spans; this number does not change significantly with additional crowdworkers.
We also measure substantial token-level inter-annotator agreement using Fleiss' kappa for identifying typed mention spans ($\kappa = 0.83$), canonical mentions ($\kappa = 0.75$) and entity links ($\kappa = 0.75$) with just three workers.
Based on this analysis, we use token-level majority over 3 workers in subsequent experiments.
We also manually labeled 200 relation instances that from the exhaustively annotated documents and observed a precision of about 75\%.
While this precision is slightly less than the 80\% obtained by annotators at LDC \citep{ellis2016overview}, we believe it could be easily improved with appropriate changes to the annotation interface.
Consequently, we take a majority vote over 3 workers in subsequent experiments,
leading to a total cost of \$0.15 per relation instance.

The entity annotation interface is far more involved and takes on average about 13 minutes per document, corresponding to about \$2.60 per document, while the relation annotation interface takes on average about \$2.25 per document.
Because documents vary significantly in length and complexity, we set rewards for each document based on the number of tokens (.75c per token) and mention pairs (5c per pair) respectively.
With 3 workers per document, we paid about \$15 per document on average.
Each document contained an average 9.2 relations, resulting in a cost of about \$1.61 per relation instance.
We note that this is about ten times as much as labeling a relation instance.

We defer details regarding how documents themselves should be weighted to capture diverse entities that span documents to \refapp{implementation}.
%We provide details regarding our sampling scheme and its distribution over entities in \refapp{implementation} of the supplementary material.
%When considering uniformly sampled documents, we found that a majority of the relations extracted correspond to very rare entities and result in very few entities with more than one relation (\reffig{entity-distribution}).
%In contrast, the TAC KBP query are almost evenly split between rare and semi-frequent entities.
%As a heuristic, we adopt the following two-stage sampling procedure:
%First, 20\% of our exhaustive document collection is sampled uniformly and annotated.
%We then uniformly sample the entities annotated to create a collection of ``evaluation entities''.
%Finally, we construct the remaining 80\% of our document collection by searching for documents that contain the evaluation entities according to an exact string match. This process results in far more entities of medium frequency.


 \documentclass [12pt]{article}

\usepackage {amsmath}
\usepackage {amsthm}
\usepackage {amssymb}
\usepackage {graphicx}
\usepackage {float}
\usepackage {multirow}
\usepackage {xcolor}
\usepackage [ruled,vlined,commentsnumbered,titlenotnumbered]{algorithm2e} \usepackage {array}
\usepackage {booktabs}
\usepackage {url}
\usepackage {parskip}
\usepackage [margin=1in]{geometry}
\usepackage [T1]{fontenc}
\usepackage {cmbright}
\usepackage [many]{tcolorbox}
\usepackage [colorlinks = true,
            linkcolor = blue,
            urlcolor  = blue,
            citecolor = blue,
            anchorcolor = blue]{hyperref}
\usepackage {enumitem}
\usepackage {xparse}
\usepackage {verbatim}
\usepackage{algpseudocode}
\usepackage{listings}
\usepackage{xcolor}
\lstset { %
    language=C++,
    backgroundcolor=\color{black!5}, % set backgroundcolor
    basicstyle=\footnotesize,% basic font setting
}
\newtheorem{theorem}{Theorem}
\newtheorem{remark}{Remark}
\newtheorem{lemma}[theorem]{Lemma}
\theoremstyle{definition}
\newtheorem{definition}{Definition}[section]
\newtheorem{claim}{Claim}

\DeclareTColorBox {Solution}{}{breakable, title={Solution}} \DeclareTColorBox {Solution*}{}{breakable, title={Solution (provided)}} \DeclareTColorBox {Instruction}{}{boxrule=0pt, boxsep=0pt, left=0.5em, right=0.5em, top=0.5em, bottom=0.5em, arc=0pt, toprule=1pt, bottomrule=1pt} \DeclareDocumentCommand {\Expecting }{+m}{\textbf {[We are expecting:} #1\textbf {]}} \DeclareDocumentCommand {\Points }{m}{\textbf {(#1 pt.)}}

\begin {document}

\vspace {1em}
\begin {Instruction}
Adapted From Virginia Williams' lecture notes.
\end {Instruction}

{\LARGE \textbf {COMP 285 (NC A\&T, Spr `22)}\hfill \textbf {Lecture 27} }

\begin{centering}
\section*{Dynamic Programmig V - 0/1 Knapsack and Maximal Independent Set}
\end{centering}

\section{The Knapsack Problem}

This is a classic problem, defined as the following:

We have $n$ items, each with a value and a positive weight. The $i$-th item has weight $w_i$ and value $v_i$. We have a knapsack that holds a maximum weight of $W$. Which items do we put in our knapsack to maximize the value of the items in our knapsack? For example, let’s say that $W = 10$; that is, the knapsack holds a weight of at most 10. Also suppose that we have four items, with weight and value:

\begin{table}[h!]
\centering
\begin{tabular}{c|c|c}
Item & Weight & Value \\
\hline
A & 6 &  25 \\
B & 3 & 13 \\
C & 4 & 15 \\
D & 2 & 8
\end{tabular}
\end{table}

We will talk about two variations of this problem, one where you have infinite copies of each item (commonly known as Unbounded Knapsack), and one where you have only one of each item (commonly known as 0-1 Knapsack).

What are some useful subproblems? Perhaps it’s having knapsacks of smaller capacities, or maybe it’s having fewer items to choose from. In fact, both of these ideas for subproblems are useful. As we saw last lecture, the first idea is useful for the Unbounded Knapsack problem, and a combination of the two ideas is useful for the 0-1 Knapsack problem.

\subsection{The 0-1 Knapsack Problem}

We consider what happens when we can take at most one of each item. Going back to the initial example, we would pick item $A$ and item $C$, having a total weight of $10$ and a total value of $40$.

The subproblems that we need must keep track of the knapsack size as well as which items are allowed to be used in the knapsack. Because we need to keep track of more information in our state, we add another parameter to the recurrence (and therefore, another dimension to the DP table). Let $K(x, j)$ be the maximum value that we can get with a knapsack of capacity $x$ considering only items at indices from $1, \cdots , j$. Consider the optimal solution for $K(x, j)$. There are two cases:

\begin{enumerate}
  \item Item $j$ is used in $K(x, j)$. Then, the remaining items that we choose to put in the knapsack must be the optimum solution for $K(x - w_j , j - 1)$. In this case, $K(x, j) = K(x - w_j , j - 1) + v_j$ .
  \item Item $j$ is not used in $K(x, j)$. Then, $K(x, j)$ is the optimum solution for K(x, j - 1). In this case, $K(x, j) = K(x, j - 1)$.
\end{enumerate}

So, our recurrence relation is: $K(x, j) = \max\{K(x -w_j , j -1) +v_j , K(x, j -1)\}$. Now, we’re done: we simply calculate each entry up to $K(W, n)$, which gives us our final answer. Note that this also runs in $O(nW)$ time despite the additional dimension in the DP table. This is because at each entry of the DP table, we do $O(1)$ work.

\begin{algorithm}
\caption{ZeroOneKnapsack(W, n, w, v)}
\label{alg:zerooneknapsack}
\begin{algorithmic}
\State \For{$j = 1, \cdots, n$}{
  $K[0,j] \gets 0$
}
\State \For{$x = 0, \cdots, W$}{
    $K[x, 0] \gets 0$
}
\State \For{$x = 1, \cdots, W$} {
  \State \For{$j = 1, \cdots, n$} {
    \State $K[x, j] \gets K[x, j-1]$
    \State \If {$w_j <= x$} {
        $K[x, j] = \max\{K[x - w_j, j-1] + v_j, K[x, j] \}$
    }
  }
}
\State \Return $K[W, n]$
\end{algorithmic}
\end{algorithm}

\section{The Independent Set Problem}

This problem is as follows:

Say that we have an undirected graph $G = (V, E)$. We call a subset $S \subseteq V$ of vertices ``independent'' if there are no edges between vertices in $S$. Let vertex $i$ have weight $w_i$ , and denote $w(S)$ as the sum of weights of vertices in $S$. Given $G$, find an independent set of maximum weight $arg\max_{S\subseteq V} w(S)$.

Actually, this problem is NP-hard for a general graph $G$. However, if our graph is a tree, then we can solve this problem in linear time. In the following figure, the maximum weight independent set is highlighted in blue.

\begin{remark}
Dynamic programming is especially useful to keep in mind when you are solving a problem that involves trees. The tree structure often lends itself to dynamic programming solutions.
\end{remark}

\begin{figure}[h!]
\centering
\includegraphics[scale=0.5]{max_is.png}
\end{figure}

As usual, the key question to ask is, ``What should our subproblem(s) be?'' Intuitively, if the problem has to do with trees, then subtrees often play an important role in identifying our subproblems. Let's pick any vertex $r$ and designate it as the root. Denoting the subtree rooted at $u$ as $T_u$, we define $A(u)$ to be the weight of the maximum weight independent set in $T_u$. How can we express $A(u)$ recursively? Letting $S_u$ be the maximum weight independent set of $T_u$, there are two cases:

\begin{enumerate}
    \item If $u \notin S_u$, then $A(u) = \sum_{v} A(v )$ for all children $v$ of $u$.
    \item If $u \in S_u$, then $A(u) = w_u + \sum_{v} A(v )$ for all grandchildren $v$ of $u$.
\end{enumerate}

To avoid solving the subproblem for trees rooted at grandchildren, we introduce $B(u)$ as the weight of the maximum weight independent set in $T_u \setminus \{u\}$. That is, $B(u) = \sum_{v} A(v )$ for all children $v$ of $u$. Equivalently, we have the following cases:

\begin{enumerate}
    \item If $u \notin S_u$, then $A(u) = \sum_{v} A(v )$ for all children $v$ of $u$.
    \item If $u \in S_u$, then $A(u) = w_u + \sum_{v} B(v )$ for all children $v$ of $u$.
\end{enumerate}

So, we can calculate the weight of the maximum weight independent set:
$$
A(u) = \max\left\{w(u) + \sum_{v \in Children(u)} B(u), \sum_{v \in Children(u)} A(v)  \right\}
$$

To create an algorithm out of this recurrence, we can compute the $A(u)$ and $B(u)$ values in a bottom-up manner (a post-order traversal on the tree), arriving at the answer, $A(r)$. This
takes $O(|V|)$ time.

\begin{algorithm}
\caption{MaxWeightIndependentSet(G)}
\label{alg:MaxWeightIndependentSet}
\begin{algorithmic}
\State \texttt{// G is a tree}
\State $r \gets \text{ArbitraryVertex}(G)$
\State $T \gets \text{RootTreeAt}(r)$
\Procedure{SolveSubtreeAt}{$u$}
    \State \eIf{Children$(T, u) = \emptyset$} {
        \State $A(u) \gets w_u$
        \State $B(u) \gets 0$
    }{
        \State \For{$v \in \text{Children}(u)$} {
            \State \texttt{SolveSubTreeAt}(v)
        }
        \State $A(u) \gets \max\left\{\sum_{v \in \text{Children}(u)} A(v), w_u + \sum_{v \in \text{Children}(u)} B(v) \right\}$
        \State $B(u) \gets \sum_{v \in \text{Children}(u)} A(v)$
    }
\EndProcedure
\State \texttt{SolveSubtreeAt}$(r)$
\State \Return $A(r)$
\end{algorithmic}
\end{algorithm}

\end{document}