% *********** Document name and reference:
% Title of document
\renewcommand{\ndoctitle}{NuPyCEE}
% Document category acronym
\renewcommand{\ndocname}{NuPyCEE}
% svn dir
\renewcommand{\svndir}{svn://forum.astro.keele.ac.uk/utils/GCE/NuPyCEE}
% Contributors to this document
\renewcommand{\ndoccontribs}{Christian Ritter, Benoit C\^ot\'e}

%\input{chap}

Document name: \ndocname \\
SVN directory: \svndir\\
Contributors: \ndoccontribs\\
Please send any feedback to critter@uvic.ca or bcote@uvic.ca. Thank you.

{  \textbf{Abstract:} %\slshape{}
%\begin{abstract}
\noindent The Nugrid-Python-Chemical-Evolution Environment, short NuPyCEE, provides
a framework to simulate the ejecta of simple stellar populations with the 1-zone code
Stellar Yields for Galactic Modeling Applications as well as galaxies
with the One-zone Model for the Evolution of Galaxies.
Various chemical evolution input parameters can be specified through a python-based interface.
Both tools provide extraction and plotting methods.
We also provide a web interface with limited capabilities which allows to download
yield tables containing the ejecta of SSPs.
To compare the results to observational data, the user can utilize the STELLAB module
which provides various spectroscopic abundance ratios of stars observed in the
Milky Way and local dwarf spheroidal galaxies.
%\end{abstract}
}

%##############################################################
%# Section: Introduction
%##############################################################

\section{Introduction}
\index{\ndocname}
Welcome to the \ndocname\ user Guide. The purpose of \ndocname\ is to
provide the user with a framework to perform chemical evolution calculations.
The focus lies on simplification and user-friendliness.
The user can choose between the \textit{Stellar Yields for Galactic Modeling Applications} (SYGMA) module
and the \textit{One-zone Model for the Evolution of Galaxies} (OMEGA) module.
SYGMA allows to follow the ejecta of simple stellar populations in a closed box while OMEGA deals with
the simulation of galaxies with inflows and outflows.
Both tools can be used in a similar manner through the ipython command line or
ipython notebooks.
%However included are low-mass, intermediate and massive star contributions, including
%SN2 and SN1a.

%This section will provide the user with a
%tutorial and walk through of the tools contained within \ndocname\ and
%how a typical user would go about using and working with this python module.
%Note also that each SYGMA version provides a python-based function documentation (Sphinx)
%in Sphinx\textunderscore DOC which can be accessed with the web brower through build/html/index.hmtl.

\subsection{Fastest possible instruction}

To download the code
you need to have git installed on your local machine.
Then download the code from github at

\url{https://github.com/NuGrid/NuPyCEE}

into your local directory via

\begin{verbatim}
git clone  https://github.com/NuGrid/NuPyCEE.git .
\end{verbatim}

Install ipython with numpy and matplotlib and
nugridpy via

\begin{verbatim}
pip install nugridpy
\end{verbatim}

Start ipython in the NuPyCEE directory with

\begin{verbatim}
ipython -pylab -p numpy
\end{verbatim}

To run a SYGMA calculation:

\begin{verbatim}
import sygma as s
s1=s.sygma()
\end{verbatim}

That's it! To plot the total  mass ejected:

\begin{verbatim}
s1.plot_totmasses(source='all')
\end{verbatim}

%More plot functions are available: s1.plot$<$tab$>$.
%Use a question mark after the plotting function
%for more infos.

To run OMEGA:

\begin{verbatim}
import omega as o
o1=o.omega()
\end{verbatim}

\begin{verbatim}
o1.plot_spectro(xaxis='[Fe/H]',yaxis='[O/Fe]')
\end{verbatim}

Now you might want to compare the prediction with
observations through the STELLAB module.

\begin{verbatim}
import stellab as st
st1=st.stellab()
st1.plot_spectro(xaxis='[Fe/H]', yaxis='[O/Fe]')
\end{verbatim}

%Check out the documentation in Section \ref{sect_omega}.

\subsection{Preparing the environment}%Tutorial and Walkthough}

To take advantage of the interactive ipython environment
SYGMA and OMEGA are both launched after starting
ipython with

\begin{verbatim}
ipython -pylab -p numpy
\end{verbatim}

where the modules numpy and matploblib have to be installed.
Also you might need to install other python modules (e.g., h5py).

A necessary dependence is NuGrid's pylib toolbox nugridpy which
contains a number of python scripts.  It can simply be installed via
pip:

\begin{verbatim}
pip install nugridpy
\end{verbatim}

%Make sure your PYTHONPATH variable is pointing to this directory.

\subsection{SYGMA}

\subsubsection{General concepts}

In order to follow the chemical enrichment via gas particles
in hydrodynamical simulations one has to describe the ejecta of material
from such star particles, essentially simple stellar populations.
With SYGMA one can follow the ejecta of such populations via
diagrams and you can extract the composition of the ejecta via tables.

\subsubsection{Getting started}

If one wants to start SYGMA from outside the SYGMA directory
one has to specify the $SYGMADIR$ variable which
points to the SYGMA dir.

Import the module:

\begin{verbatim}
import sygma as s
\end{verbatim}

Next, we create an SSP instance, initiate a class instance $s1$ of SYGMA:

\begin{verbatim}
s1=sygma.sygma(iniZ=0.0001,tend=1e10,mgal=1e9)
\end{verbatim}

A single stellar population of mass $1\times10^{9}\msun$ and metal fraction
of $0.0001$ is created and evolved in time steps of $10^{8}$ years
to the final time of 1e10 years. Other not specified and hence
default parameter include for example the choice of the initial-mass function.
%range of $1\msun$ up to $30\msun$ is used. NuGrid tables
%and SN1a yields from Thielemann86 are taken

One can check the definition of these input parameter online via
\url{http://nugrid.github.io/NuPyCEE/SPHINX/build/html/sygma.html}.

No yield tables are defined and hence the NuGrid tables as the
default choice were selected. These are the Set1 and Set1extension
yields which are
%\begin{verbatim}
%s1=sygma.sygma(alphaimf=2.35,sfr='input',iniZ=0.0001,dt=1e8,tend=1e10,mgal=1e9,
% table='yield_tables/isotope_yield_table.txt',
% sn1a_table='yield_tables/sn1a_t86.txt',
% bb_table='yield_tables/bb_walker91.txt')
%\end{verbatim}
AGB and massive star yields for five different metallicities.
In this mode the user specifies $iniZ$
which can be either $Z=2e-2, 1e-2, 6e-3, 1e-3, 1e-4$ and $0.0$.
Scaled abundances for Z=1e-t65, 1e-6 will be available soon.
Currently for $Z=0$ the PopIII stars from Heger et al. (2010) are set
as default.

\subsubsection{Providing your own yield tables}

Yield tables are available in the NuPyCEE subdirectory
yield\textunderscore tables. Add your yield tables to
this directory and SYGMA will be able read the table
if you have specified the $table$ variable. Only
for table of Z=0 the variable $pop3\textunderscore table$ is used.
Both tables need yields specified in the SYGMA (and OMEGA)
yield input format. See for the structure the default table.
It is important to provide an initial abundance
file which has to match the number of species provided in the yield tables.
Provide the file in the iniAbu directory inside the directory yield\textunderscore tables.
The input variable with which the table file can be specified is $iniabu\textunderscore table$.
For the necessary structure see again the default choice of that variable.
%Another subdirectory,
%plotting\textunderscore scripts, contains various
%plotting scripts. The latter use SYGMA's plotting functions.

\subsubsection{What do I get out?}

The output during execution shows some initial parameters,
e.g. the isotopes being part of chemical evolution,
as well as the time and metallicity evolution.
Also shown is the contribution time of stars of
certain initial mass.

%For more examples, see the SYGMA ipython notebooks available
%at \\http://208.75.74.161:8888/tree/Publicnotebooks/intro\textunderscore SYGMA.

To analyze the run afterwards we provide a variety of plotting  functions
which are described in detail at \url{http://nugrid.github.io/NuPyCEE/SPHINX/build/html/sygma.html}.
In following we give some examples. The function plot\textunderscore mass() shows the evolution
of the yield in solar masses.

\begin{verbatim}
s1.plot_mass()
\end{verbatim}

With SYGMA you can look at different contributions
for AGB stars, massive stars, and SN~Ia.

\begin{verbatim}
s1.plot_mass(source='agb')
s1.plot_mass(source='massive')
s1.plot_mass(source='sn1a')

\end{verbatim}

Note that those lines appear all in one figure.
%The solar abundance distribution is taken from GN93 (NuGrid file).

%To get an overview over the available abundance distribution
%(in mass fraction divided by solar mass fraction) of the gas
%reservoir at a certain timestep, a function
%called plot\textunderscore abu\textunderscore distr() is available.
The function $plot\textunderscore mass\textunderscore range\textunderscore contributions()$
shows the yield contribution of stars as a function of their initial mass.
It allows to identify which mass range contributes to certain elements.
%Also plotted is to each yield (grid input) corresponding mass range
%as vertical lines. In these intervals the yields are the same (for one
%simple stellar population).
To monitor the evolution of the mass of the gas reservoir ejected
masses over time, use plot\textunderscore totmasses().
The star formation rate as a function of time can be plotted with the function $plot\textunderscore sfr()$.
In the case you want to write out detailed chemical evolution tables
of isotopes, use the function $write\textunderscore evol\textunderscore table()$.
It writes out the mass of isotopes ejected by stars as a function of time
(each line represents a timestep).

\subsection{OMEGA}

\subsubsection{General concepts}
OMEGA is a classical one-zone, open-box galaxy model with
an input star formation history where stars form and inject new elements within
the same gas reservoir, using SYGMA to create a SSP at every timestep.
At any time, each existing SSP ejects material that reflects its age, mass, and initial metallicity.
OMEGA can mimic known local galaxies, such as Sculptor, Carina, and Fornax,
by using the observed star formation history and current mass, which is taken
from the literature.  It also offers the possibility to mimic the Milky Way, but we
warn the user that a one-zone model is probably too simplistic to simulate the
evolution of massive and complex galaxies, although they are useful for testing
stellar yields.

The code offers three different prescriptions for treating gas inflows and outflows
(see the OMEGA Userguide ipython notebook on WENDI, \url{http://www.nugridstars.org/projects/wendi}).
Inflows and outflows can be independently switched off to consider a semi-open box or a closed box model.
The different prescriptions can also be found in C\^ot\'e et al. (2016, \url{http://adsabs.harvard.edu/abs/2016arXiv160407824C}).

A complete description of OMEGA's input parameters and functions can
be found in the Sphinx documentation (\url{http://nugrid.github.io/NuPyCEE/SPHINX/build/html/omega.html}).
All the SSP parameters associated with SYGMA are available with OMEGA as well.

\subsubsection{Getting Started with OMEGA}

We refer to the OMEGA Userguide ipython notebook on WENDI (\url{http://www.nugridstars.org/projects/wendi})
for more information about how to use OMEGA with command lines, and how to use plotting functions
to analyze the predicted chemical evolution of the target galaxy.

\subsection{STELLAB}

STELLAB (short for Stellar Abundances) is a module that allows to plot the stellar abundances
observed in different galaxies.  So far, data are available for the Milky Way, Sculptor, Fornax, Carina,
and the LMC.  This code uses an input collection of data and plots any abundance ratio [X/Y] as
a function of another one.  It also offers the possibility to re-normalize all data to the same solar
normalization.  It is a useful and complementary tool that enables to confront numerical predictions
with observations.

We refer to the STELLAB Userguide ipython notebook on WENDI (\url{http://www.nugridstars.org/projects/wendi})
for more information on its usage, options, and collection of data.  It should be noted that observational
data have not been curated, but have rather been blindly taken from the literature.  We are currently
working on a curation process in collaboration with JINA-CEE members (\url{http://www.jinaweb.org}).

\section{Disclaimer}

\section{History}
This document history complements the svn log.

\begin{tabular*}{\textwidth}{lll}
\hline
Authors & yymmdd & Comment \\
CR & 130926 & Update of links and details \\
\hline
CR & 160513 & creation \\
\end{tabular*}

\subsection{Contact}
If any bugs do appear or if there are any questions, please email critter@uvic.ca or bcote@uvic.ca
% --------------- latex template below ---------------------------
\begin{verbatim}

\end{verbatim}

%\begin{figure}[htbp]
%   \centering
%%   \includegraphics[width=\textwidth]{layers.jpg} %
%      \caption{}   \includegraphics[width=0.48\textwidth]{FIGURES/HRD90ms.png}
%   \includegraphics[width=0.48\textwidth]{FIGURES/HRD150ms.png}
%
%   \label{fig:one}
%\end{figure}
%
%\begin{equation}
%Y\_a = Y\_k + \sum\_{i \neq k} Y\_i
%\end{equation}
%
%{
%%\color{ForestGreen}
%\sffamily
%  {\center  --------------- \hfill {\bf START: Some special text} \hfill ---------------}\\
%$Y\_c$ does not contain ZZZ but we may assign one $Y_n$ to XYZ which is the decay product of the unstable nitrogen isotope JJHJ. %
%
%{\center ---------------  \hfill {\bf END:Some special text} \hfill ---------------}\\
%}


\subsubsection{\stid{3.01} hypre}

\paragraph{Overview}
The {\sl hypre} software library \cite{hypre:homepage,hypre_design_impl_2006} provides high performance preconditioners and solvers for the solution of large sparse linear systems on massively parallel computers, with particular focus on algebraic multigrid solvers. One of {\sl hypre}’s unique features is the provision of a (semi)-structured interface, in addition to a traditional linear-algebra based interface. The semi-structured interface is appropriate for applications whose grids are mostly structured, but with some unstructured features. Examples include block-structured grids, composite grids in structured adaptive mesh refinement (AMR) applications, and overset grids. These interfaces give application users a more natural means for describing their linear systems, and provide access to methods such as structured multigrid solvers, which can take advantage of the additional information beyond just the matrix. Since current architecture trends are favoring regular compute patterns to achieve high performance, the ability to express structure has become much more important. The {\sl hypre} library provides both unstructured and structured multigrid solvers, which have shown excellent scalability on a variety of high performance computers, e.g Blue Gene systems (unstructured solver BoomerAMG has scaled up to 1.25 million MPI cores with a total of 4.5 million hardware threads). It is used by many ECP application teams, including ExaAM, Subsurface, ExaWind, CEED, and more. It requires a C compiler and an MPI implementation, but it also runs in an OpenMP environment. It has some GPU capabilities.

\paragraph{Key  Challenges}

While {\sl hypre}'s solvers contain much parallelism, their main focus is the solution of sparse linear systems, leading to  very large demands on memory bandwidth. In addition, the use of multiple levels, while greatly aiding convergence of the solvers, leads to decreasing systems sizes, number of operations and parallel efficiencies on coarser levels. Particularly the unstructured algebraic multigrid solver BoomerAMG\cite{HeYa2002}, which is {\sl hypre}'s most often used preconditioner, suffers from increasing communication complexities on coarser levels. Coarse grid operators are generated by multiplying three matrices leading to increasing numbers of nonzeroes per row in the resulting matrices and with it increasing numbers of neighbor processes. While BoomerAMG's solve phase mainly consists of matrix vector products and smoothing operations, which are fairly straight forward to parallelize, even on a GPU, its setup phase is highly complex, including many branches, a lot of integer operations as well as some sequential passages. Current  interpolation strategies that lead to best convergence and performance on distributed memory machines are not suitable for implementation on GPUs or similar architectures requiring extreme parallelism. Since {\sl hypre} is a mature product with many solvers and interdependent features, any significant changes that affect the whole library, are tedious and require much testing to ensure that the library stays backward compatible and no features are broken.

\paragraph{Solution Strategy}

Since computer architectures continue to change rapidly, it was important to come up with strategies that will facilitate future porting of the software. Therefore we developed and implemented a new memory model that addresses the use of different memory locations.
Since the upcoming computer architectures are heterogeneous with accelerators, we focus on enabling {\sl hypre} for GPUs. We have looked into various options, such as the use of CUDA, OpenMP 4.5, as well as RAJA and Kokkos. We limited the latter two options to the structured interface and solvers which are more natural candidates for such an approach due to their use of macros, called BoxLoops, for loops. We will also investigate the use of HIP and SYCL for AMD and Intel accelerators that will be available on future exascale computers.

\paragraph{Recent Progress}

Previously, we enabled the structured solvers, SMG and PFMG\cite{AsFa1996}, both setup and solve phase, to completely run on GPUs, using both CUDA or OpenMP4.5, and to not require unified memory. In addition, options to use RAJA and Kokkos are available.
Porting the unstructured solver, BoomerAMG turned out to be far more complex. We first ported the solve phase to GPUs for select smoothers, mainly Jacobi smoothers, requiring unified memory. During the last year, we focused on enabling selected components of the setup phase to run on GPUs, specifically PMIS coarsening, direct interpolation, and the generation of the coarse grid operator, which consists of the multiplication of three sparse matrices. Figure \ref{fig:rap} shows runtime comparisons of different implementations of the coarse grid operator generation on one V-100 GPU or one Power 9, which demonstrate that the hypre GPU implementation outperforms state-of-the-art implementations such as CUSP and cuSPARSE. It is now possible to perform a complete linear system solve with BoomerAMG on GPUs using specific settings.

\begin{figure}
\centering
	\includegraphics[width=5in]{projects/2.3.3-MathLibs/2.3.3.12-SUNDIALS-hypre/hypre-rap-1gpu.png}
	\caption{\label{fig:rap} Runtimes to generate a coarse grid operator for a 7pt 3d Laplace problem matrix on 1 V-100 GPU or Power 9 CPU with up to 20 OMP threads for various implementations}
\end{figure}

We also implemented a new integer datatype called HYPRE$\_$BigInt to avoid the requirement that all integers need to be converted to 64 bit integers when solving linear systems greater than 2 billions using the unstructured solvers. The new datatype allows to only convert variables that need to be 64 bit integers and improves memory usage and performance, as illustrated in Figure \ref{fig:mixed-int}.

\begin{figure}
\centering
	\includegraphics[width=3in]{projects/2.3.3-MathLibs/2.3.3.12-SUNDIALS-hypre/hypre-mixed-int.png}
	\caption{\label{fig:mixed-int} Weak scaling study on LLNL Linux cluster Quartz: Total runtimes in seconds for AMG-PCG using 1M points/core for 2 different 3D diffusion problems.The new mixed-int capability performs about 20-25 percent better than the 64 bit integer version while using less memory and is capable to solve larger problems than the 32 bit integer version.}
\end{figure}

\paragraph{Next Steps}

We will pursue the following tasks:

\begin{itemize}
\item We will continue to add new GPU capabilities to {\sl hypre}. This includes converting various components that are currently running only on the CPU to be usable on the GPU using CUDA or OpenMP 4.5. We will particularly focus on additional setup components, such as interpolation operators, to achieve better convergence.
\item We also plan on improving the efficiency of interfacing applications with {\sl hypre}'s solvers.
\end{itemize}
In addition, we will work with ECP application teams who are using {\sl hypre}, such as ExaWind, or would like to use it, to achieve best performance by tuning the solvers for them and potentially implementing suitable algorithmic changes.


\section{Provoke}\label{maneuver:provoke}
\textbf{Cost:} 200 CP\\
\textbf{Requirements:} -\\
\textbf{Maneuver, Active, Repeatable, Source(150 Gold)}\\
You can spend 2 AP to provoke an Opponent to attack you.
Make an Empathy check, contested by this opponent's Perception check.
If you succeed, this opponent deals half damage (round up) against any creature that isn't you for their next 1D6 attacks, with an exception to attacks that deal area-based damage effects that includes you.\\
\\
\\
Level Progression:\\
\\
\rowcolors{2}{lightgray}{white}
\begin{tabular}{l | l | l | l}
	Level & CP Cost & Requires & Effect\\
	II & 500 & - & 3 AP for 2D6 attacks also possible\\
	III & 2,500 & - & 4 AP for 3D6 attacks also possible\\
	IV & 5,000 & - & 5 AP for 4D6 attacks also possible\\
	IV & 7,500 & - & 6 AP for 5D6 attacks also possible\\
\end{tabular}


\section{Experiments and Results}

We conducted a variety of experiments to audit our previously
described approach. As a running example we used a DNN which we
trained on images from a generated dataset we dubbed \emph{Picasso
  Dataset}. The foundation are images of human faces we took from the
FASSEG dataset~\cite{khan2015multi,khan2017head}.
The Picasso Dataset contains collage images of faces with the facial
features (eyes, mouth, nose) either in the correct constellation
(positive class) or in a mixed-up constellation (negative class). See
Fig.~\ref{fig:picasso} for examples.
No distinction is made between originally left and right eyes.

\begin{figure}
  \begin{center}
    \includegraphics[width=.15\textwidth]{pic_00001_pos.png}\quad
    \includegraphics[width=.15\textwidth]{pic_00009_pos.png}\qquad\qquad
    \includegraphics[width=.15\textwidth]{pic_00000_neg.png}\quad
    \includegraphics[width=.15\textwidth]{pic_00001_neg.png}
  \end{center}
  \caption{Examples from the Picasso Dataset
    (\textit{left:} positive class, \textit{right:} negative).}
  \label{fig:picasso}
\end{figure}

In order to not establish a divergence in the image space of the two
classes, the positive and negative classes contain facial features
that were cut out of a set of images from the original FASSEG
dataset. As a canvas to include the features, we took a set of
original faces and got rid of the facial features by giving the
complete facial area a similar skin-like texture. Then we included the
cut out facial features onto the original positions of the original
features in the faces.

The face images in Fig.~\ref{fig:picasso} show that
the resulting dataset is rather constructed. This however will suffice for
a proof of concept to show that our approach in fact exploits object parts
and their relations. In the future we plan on moving towards more natural
datasets.

\subsection{Analyzed DNNs}
We evaluated our method on three different architectures from the pytorch
model-zoo\footnote{\url{https://pytorch.org/docs/stable/torchvision/models.html}}:
AlexNet~\cite{krizhevsky_one_2014},
VGG16~\cite{simonyan_very_2015}, and
ResNeXt-50~\cite{xie_aggregated_2017}.
The convolutional parts of the networks were initialized with weights
pre-trained on the ImageNet dataset.
For fine-tuning the DNNs for the Picasso Dataset task,
%
the output dimension was reduced to one and the in- and output
dimension of the second to last hidden dense layer was reduced to 512
for AlexNet and VGG16.
Then the dense layers and the last two convolutional layers
(AlexNet, VGG16) respectively bottleneck blocks (ResNeXt) were fine-tuned.
%
The fine-tuning was conducted in one epoch on a training set of 18,002
generated, $224\times224$-sized picasso samples with equal
distribution of positive and negative class.
All models achieved accuracy greater than 99\%
on a test set of 999 positive and 999 negative samples.

% The fine-tuning failed for VGG16 when initialized with weights from
% the VGG face dataset, which we explain with the sub-optimal alignment
% of the objectives between VGG face and the Picasso Dataset.

\subsection{Training the Concept Models}
%%% Concept Data
In our example we determined the best ensembled detection concept vectors for
the concepts \textsc{eyes}, \textsc{mouth} and \textsc{nose} amongst the
considered layers.
%
%%% Analyzed Model Layers
We excluded layers with low receptive field, as they are assumed to
hold only very local features
(for the layers used see Fig.~\ref{fig:layerwiseresults}).
Convolutional output was only considered after the activation.
% ResNeXt: starting at bottleneck modules 4/16,
% VGG16: starting at conv layer 5/13,
% AlexNet: starting at conv layer 2/5
%
\begin{figure}%
  \centering%
  \hspace*{-0.08\textwidth}\includegraphics[width=1.16\textwidth]{layerwise_results_concept_embedding}
  \caption{The layer-wise mean set IoU results of the concept analysis runs.}
  \label{fig:layerwiseresults}
\end{figure}
%
For each concept, 452 training/validation, and 48 test picasso samples
with segmentation masks were used.
% The concept window size determining the kernel size of the concept
% model convolution was chosen as the mean boxed size of the concept.
% The latter was chosen as
% $0.15\times0.05$ for the nose,
% $0.08\times0.2$ for the mouth, and
% $0.08\times0.16$ for an eye.
The training objective was: Predict at each activation map pixel
whether the kernel window centered there lies \enquote{over}
an instance of the concept. Over meant that the fuzzy intersection of
the concept segmentation and the kernel window area exceeds a threshold
(\emph{intersection encoding}).
This fuzzy definition of a box center
% introduces some noise to the ground truth, but
tackles the problem of sub-optimal intersections in later layers
due to low resolution.
% At a threshold greater or equal 0.5 masks are smoothed and sharpened.
Too high values may lead to elimination of an instance,
% Intersection over union (IoU) encoding is especially vulnerable to this for
% small concepts, wherefore we preferred intersection encoding.
and thresholds were chosen to avoid such issues with
values 0.5/0.8/0.7 for nose/mouth/eye.
We implemented the encoding via a convolution.
As evaluation metric we use set IoU (sIoU) between the detection
masks and the intersection encoded masks as in Net2Vec.
%
%%% Optimization Method
On each dataset and each layer, 15 concept models were trained in
three 5-fold-cross-validation runs with the following settings:
Adam optimization with mean best learning rate of 0.001,
a weighting of 5:1 of Dice to bBCE loss,
batch size of 8, and
two epochs (all layers showed quick convergence).
% \begin{align*}
    %     \text{Loss}(M_\text{pred}, M)
    %     &= \lambda_\text{Dice} \text{Dice}(M_\text{pred}, M) +
            %             \lambda_\text{bBCE} \text{bBCE}(M_\text{pred}, M) \\
    %     \text{Dice}(M_\text{pred}, M)
    %           &= \frac{2\cdot MM_\text{pred}}
                  %                   {2\cdot MM_\text{pred} + (1-M)M_\text{pred} + M(1-M_\text{pred})} \\
    %     \text{bBCE}(M_\text{pred}, M)
    %           &= b M \log(M_\text{pred}) + (1-b) (1-M) \log(1-M_\text{pred})
                  %   \end{align*}
                  %                   for a ground truth intersection encoded mask $M$,
                  %                   a predicted mask $M_\text{pred}$,
                  %                   $b=0.99$ the mean proportion of background pixels in a concept mask
                  %                   (chosen for nose 0.999, else 0.995),
                  %                   and $\lambda_\text{Dice}=0.5$ and $\lambda_{bBCE}=0.1$.
                  %                   We recorded slightly worse results for higher weighting of the binary
                  %                   cross-entropy part or balancing of the Dice loss towards less false
                  %                   positives or less false negatives as described in
                  %                   \cite{salehi_tversky_2017}.
                  %

\paragraph{Results}
%%% ensemble performance pretty well coincides with mean performance, sometimes slightly better (possibly due to implicit confidence calibration)
Our normalized ensembling approach proved valuable as it yielded
mean or slightly better performance compared to the single runs.
For the considered models, meaningful embeddings of all concepts could
be found (see Tab.~\ref{tab:bestembeddings}): The layers all reached
sIoU values greater than 0.22
%%% influence of resolution
despite of the still seemingly high influence of sub-optimal
resolutions of the activation maps.
Fig.~\ref{fig:conceptsamples} shows some exemplary outputs.
%%% embeddings in early layers
The concepts were best embedded in earlier layers,
%%% embeddings in different layers
while different concepts did not necessarily share the same layer.
\begin{table}
  \centering
  \caption{Results for ensemble embeddings with
    set IoU (sIoU), mean cosine distance to the runs (Cos.d.),
    and index of conv layer or block (L) (cf. Fig.~\ref{fig:layerwiseresults}).
    \label{tab:bestembeddings}}%
  \footnotesize%
  \newcommand*{\normentry}[1]{\multicolumn{1}{c}{#1}}%
  \newcommand{\model}[1]{\multirow{4}{*}{
      \begin{tabular}{@{}c@{}}
        \rotatebox[origin=c]{90}{\strut#1}
      \end{tabular}}}%
  \newenvironment{statstable}[1]{%
    \begin{tabular}{@{}l >{\scshape}l c@{~~} S[table-auto-round] S[table-auto-round]@{}}%
      \toprule%
      \model{#1}
      &
      & \normentry{L~~}
      & \normentry{sIoU}
      & \normentry{Cos.d.}\\%
      \cmidrule[\heavyrulewidth]{2-5}
    }{\bottomrule\end{tabular}}%
  \mbox{%
    \begin{statstable}{AlexNet}
      & nose  & 2 & 0.227925 & 0.04030534426371257 \\ % 0.290909
      & mouth & 2 & 0.239023 & 0.04018515348434448 \\ % 0.290484
      & eyes  & 2 & 0.271918 & 0.0581478198369344  \\ % 0.347822
    \end{statstable}\hspace*{1em}%
    \begin{statstable}{VGG16}
      & nose  & 7 & 0.332171 & 0.10435034632682805 \\ % 0.468819
      & mouth & 6 & 0.295552 & 0.15369088053703306 \\ % 0.561507
      & eyes  & 6 & 0.349707 & 0.1969874024391174  \\ % 0.628199
    \end{statstable}\hspace*{1em}%
    \begin{statstable}{ResNeXt}
      & nose  & 6 & 0.263708 & 0.017367599999999928 \\ % 0.192076
      & mouth & 5 & 0.236788 & 0.020498533333333402 \\ % 0.208507
      & eyes  & 7 & 0.302339 & 0.019860466666666854 \\ % 0.205269
    \end{statstable}%
  }
\end{table}
\begin{figure}%
  \centering%
  \includegraphics[width=0.6\textwidth]{vis_concept_predictions_vgg16}\\
  \includegraphics[width=0.6\textwidth]{vis_concept_predictions_alexnet}%
  \caption{
    Ensemble embedding outputs of
    \textsc{nose} (green), \textsc{mouth} (blue), \textsc{eyes} (red).}
  \label{fig:conceptsamples}
\end{figure}

\subsection{Example Selection for ILP Training}
The goal of the ILP model is to approximate the behavior of the main
DNN, \idest its decision boundary.
For this, few but meaningful training samples and their DNN output
are needed: class-prototypes as well as ones that tightly frame the
DNN decision boundary.
From the 1,998 samples in the picasso test set, in total 100 samples
were chosen from the DNN test set to train the ILP model.
The DNN confidence score here was used to estimate
the proximity of a data point to the decision boundary.
For each class, we selected the 50 samples predicted to be in this
class and with confidence closest to the class boundary of 0.5.
In our setup this provided a wide range of confidence values
(including 0 and 1).

\subsection{Finding the Symbolic Explanation}
In order to find the background knowledge needed for Aleph to generate
the explanation, we need to extract the information about the facial
features and their constellations from the masks of the samples drawn in the
previous step. Abiding the procedure described in
Sec.~\ref{sec:bkextraction}, we first find contiguous clusters in the
mask layers to then infer the positional information for them. This is
straight-forward for the nose and the mouth but imposes a problem for
the eyes, since we do not want to have a single position proposal for
them in the eye that produces the biggest cluster in the mask
layer. Thus, we allow for the top two biggest clusters to infer a
position. Although we give them unique constants in the BK, we both
give them the type \ilprule{eye} $\in C$.

The next step consists of the extraction of the spatial features
between the found parts. Since the relation pair
\ilprule{left\_of}\,/ \ilprule{right\_of} as well as
\ilprule{top\_of}\,/ \ilprule{bottom\_of} can be seen as the inverses of
the respective other relation, we omit the relations
\ilprule{right\_of} and \ilprule{bottom\_of} in the BK. This is
possible, because the \emph{Closed World Assumption} holds (Everything
that is not stated explicitly is false).

Once the BK is found for all examples, we can let Aleph induce a
theory of logic rules. Consider the induced theory for the trained
VGG16 network:
\ilprule{
  \begin{align*}
    \text{face(F) :- } &\text{contains(F, A), isa(A, nose), contains(F, B), isa(B, mouth),}\\
                       &\text{top\_of(A, B), contains(F, C), top\_of(C, A).}
  \end{align*}
}
The rule explicitly names the required facial concepts \ilprule{nose}
and \ilprule{mouth} and the fact that the nose has to be above the
mouth in order for an image to be a face. Further there is another
unnamed component \ilprule{C} required which has to be placed above
the nose. By construction this has to be one of the eyes. The rule
makes sense intuitively as it describes a subset of correct
constellations of the features of a human face.

To further test the fidelity of the generated explanations to the
original black-box network, we calculated several performance metrics
for a test set of 1998 test images (999 positive and 999 negative
examples). We handled the learned explanation rules as binary
classification model for the test images in BK representation. If an
image representation is covered by the explanation rules, it is
predicted to be positive, otherwise negative. We now can handle the
binary output of the black-box model as ground truth to our
explanation predictions. The performance metrics together with the
induced explanation rules for several DNN architectures are listed in
Tab.~\ref{tab:ilp_results}. It can be seen that the explanations stay
true to the original black-box model.

\begin{table}[t]
  \newcommand*{\rulepart}[1]{\ilprule{\scriptsize #1}}
  \caption{Learned rules for different architectures and their
    fidelity scores (accuracy and F1 score wrt.\ to the original model
    predictions).
    Learned rules are of common form
    \rulepart{face(F) :- contains(F, A), isa(A, nose), contains(F, B), isa(B, mouth), distinctPart}
  }
  \label{tab:ilp_results}
  \begin{tabular*}{\textwidth}{lcc@{\extracolsep{\fill}}l}
    \toprule
    \textbf{Arch.}   & \textbf{Accuracy} & \textbf{F1} & \textbf{Distinct rule part}\\
    \midrule[\heavyrulewidth]
    VGG16   & 99.60\%  & 99.60\%  & \rulepart{top\_of(A, B), contains(F, C), top\_of(C, A)} \\
    AlexNet & 99.05\%  & 99.04\%  & \rulepart{contains(F, C), left\_of(C, A), top\_of(C, B), top\_of(C, A)} \\
    ResNext & 99.75\%  & 99.75\%  & \rulepart{top\_of(A, B), contains(F, C), top\_of(C, A)}\\
    \bottomrule
  \end{tabular*}
\end{table}

% When it comes to explaining the network behavior it is
% particularly important to look at edge cases. This can be cases
% where the network is not sure about a prediction or is even
% wrong. We expect an explanation to accurately resemble the
% decisions of the original network. We therefore investigated if
% the false positives in the test set are correctly labeled as
% positives and the false negatives are correctly labeled as
% negatives by the explanation rules. This was in fact the case all
% occurring

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "concept_embeddings_and_ilp"
%%% End:


\chapter{Height Field Files}

This appendix describes the format of the files that store data
for the height field primitive.
The format is an historical relic; a better format is needed.

Height field data is stored in binary form.
The first record in the file is a
32-bit integer giving the square root of number of data
points in the file.
We'll call this number
the size of the height field.

The size is
followed by altitude ($z$) values stored as 32-bit
floating point values.  The 0th value in the file specifies the $z$
coordinate
of the lower-left corner of the height field (0, 0).
The next
specifies the Z coordinate for $(1/(size-1), 0)$.  The last specifies the
coordinate for $(1., 1.)$.  In other words, the $i^{th}$ value
in the heightfield file specifies the $z$ coordinate for the point
whose $x$ coordinate is
$(i \% size) / (size -1)$, and whose
$y$ coordinate is $(i / size) / (size -1)$.
Non-square height fields may be rendered by specifying altitude values
less than or equal to the magic value
$-1000$.  Triangles that have any vertex less than
or equal in altitude to this value are not rendered.

While this file format is compact, it sacrifices portability for
ease of use.  While creating and handling height field files is
simple, transporting a height field from one machine to another
is problematical due to the fact that differences in byte order
and floating-point format between machines is not taken into
account.

These problems could be circumvented by writing the height field file
in a fixed-point format, taking care to write the bytes that
encode a given value in a consistent way from machine to machine.
An even better idea would be to write a set of tools for
manipulating arbitrary 2D arrays of floating-point values in a compact,
portable way, allowing for comments and the like in the file\ldots


\documentclass[12pt,a4paper]{article}
\usepackage{datetime}
\usepackage{hyperref}
\usepackage{graphicx}
\usepackage{color}
\usepackage{amsmath}
\usepackage{siunitx}

\begin{document}

\title{BOLODSP Software User's Guide}

\author{Jack Lovell\thanks{\texttt{jack.lovell@ukaea.uk}}}

\date{\today}

\maketitle

\abstract{This document serves as a guide for the end user to work with the BOLODSP software package which is bundled with D-TACQ's ACQ400 series firmware.
  A basic theory of operation is presented, and the parameters which control the operation of the BOLODSP module are discussed. General operation of
  D-TACQ's software, and the implementation details of the software and associated FPGA firmware module, are out of scope.}

\tableofcontents

\section{Introduction}
The BOLODSP software package is part of a custom module for D-TACQ ACQ400 series products used to operate the BOLO8BLF hardware module. It provides
software control of the BOLODSP FPGA module through a series of Linux commands, as well as some helper applications to simplify the fine tuning of the
signal processing capabilities of the module.

This document is \textit{not} a general user guide for D-TACQ software. For that, refer to the D-TACQ 4G user guide at \mbox{\url{http://www.d-tacq.com}}.
It is only concerned with the commands and settings provided by the package itself. Likewise, any requests for support with operating a system with this
package installed should be directed to D-TACQ directly, and not to the author, unless the query is specifically about an aspect of the BOLODSP module.

\section{Theory of Operation}
\subsection{Voltage Measurement}%
\label{sec:voltage}
\begin{figure}
  \centering
  % \includegraphics[width=0.3\textwidth]{sensor_schematic.eps}
  \def\svgwidth{0.3\columnwidth}
  \input{sensor_schematic.pdf_tex}
  \caption{Electrical schematic of bolometer sensor\label{fig:sensor}}
\end{figure}
A resistive bolometer consists of 4 resistors in a Wheatstone bridge configuration (see Figure~\ref{fig:sensor}), as described in Ref~\cite{mast-1991}.
The two resistors marked $R_m$ are ``measurement'' resistors, which are in thermal contact with a thin foil. Incident radiation heats the foil and hence
the resistors, increasing their resistance. The two resistors marked $R_r$ are ``reference'' resistors: they are not in contact with the foil, and so their
temperature (and hence resistance) remain constant.

When a voltage $V_{in}$ is applied across one diagonal of the bridge as shown, there will be zero output voltage $V_{out}$ if the bridge is balanced.
However, if the measurement resistors have increased resistance compared to the reference resistors, the bridge will be unbalanced, and $V_{out}$ will be
non-zero. By measuring $V_{out}$ it is therefore possible to calculate the resistance change of the foil, and by knowing the electrical and thermal
properties of the sensor it is possible to then infer the temperature change and the absorbed power. The details of these calculations are out of the
scope of this document.

To minimise noise in the system, AC synchronous detection is used to measure $V_{out}$. An AC excitation voltage $V_{in}$ is applied to the bridge,
typically a sinusoid with a frequency of a few 10s of kHz. The amplitude of $V_{out}$ at the same frequency is measured, and this voltage amplitude is
used in the bolometer equation to determine the absorbed power. The amplitude calculation (demodulation) is performed on the FPGA module, as follows.

The excitation voltage $V_{in}$ is of the form
\begin{equation}
  \label{equ:vin}
  V_{in} = V_0\sin(\omega t)
\end{equation}
Where $V_0$ is the amplitude of the excitation voltage and $\omega$ is the frequency. The output from the bridge due to the bridge imbalance is of the form:
\begin{equation}
  \label{equ:vout}
  V_{out} = A\sin(\omega t - \phi)
\end{equation}
Where the amplitude $A$ is the quantity we want to measure, and $\phi$ is the phase difference between the output and excitation voltages. This phase
shift is due to round trip time and parasitic capacitance in the system, and can be significant when there is a long cable between the electronics and
the sensor.

By multiplying $V_{out}$ with a normalised form of $V_{in}$ (which we shall call $V_{ref,i}$) and time averaging to remove the $2\omega$ frequency component
from the resulting signal, we can obtain the amplitude, assuming it varies slowly compared to the time averaging window. This is known as the ``in-phase''
component, since the signal is multiplied by a reference voltage of the same phase as the excitation voltage:
\begin{equation}
  \label{equ:I}
  \begin{split}
  I &= \frac{1}{n\pi}\int_{0}^{2n\pi}V_{ref,i}V_{out}\mathrm{d}\omega t \\
  &= \frac{1}{n\pi}\int_{0}^{2n\pi}\sin(\omega t) A \sin(\omega t - \phi)\mathrm{d}\omega t \\
  &= \frac{A}{2}\cos \phi
  \end{split}
\end{equation}
Equation~\ref{equ:I} still depends on the phase $\phi$. Traditionally, some phase shift has been applied to $V_{ref,i}$ such that $\phi = 0$ to remove the
phase dependence on $I$, but we employ a more general method. Consider a second reference, $V{ref,q}$ which is a quarter of a period out of phase with the
excitation voltage:
\begin{equation}
  \label{equ:Q}
  \begin{split}
  Q &= \frac{1}{n\pi}\int_{0}^{2n\pi}V_{ref,q}V_{out}\mathrm{d}\omega t \\
  &= \frac{1}{n\pi}\int_{0}^{2n\pi}\cos(\omega t) A \sin(\omega t - \phi)\mathrm{d}\omega t \\
  &= -\frac{A}{2}\sin \phi
  \end{split}
\end{equation}
We can now combine Equations~\ref{equ:I} and~\ref{equ:Q} to remove the phase dependence:
\begin{equation}
  \label{equ:A}
  A = 2\sqrt{I^2 + Q^2}
\end{equation}
Equation~\ref{equ:A} thus gives a measure of the amplitude of the output voltage independent of phase, and hence independent of the hardware used. This
method is known as ``quadrature detection''.

For completeness, the phase is given by:
\begin{equation}
  \label{equ:phi}
  \phi = -\tan^{-1}\left(\frac{Q}{I}\right)
\end{equation}

On the FPGA, the time averaging is done using a low-pass filter. The choice of filter coefficients will affect the integration time, or alternatively the
bandwidth of the output. Higher bandwidth allows higher time resolution, but comes at the cost of increased noise in the measurement, since higher
frequency noise components are less suppressed. A helper script exists to calculate the appropriate filter coefficients for a given bandwidth, and load
them onto the FPGA (see Section~\ref{sec:lpfdesign}).

The process of time averaging over multiple periods means it is no longer necessary to sample at a high enough rate to resolve the AC excitation
waveform. Instead, we only need to sample at a high enough rate to resolve amplitude (and phase) fluctuations below the cutoff frequency of the filter
used to perform the time averaging. Bolometer sensors naturally suppress high frequency signals due to the thermal properties of the sensor, so the
signal-to-noise ratio gets very low at higher frequencies. The BOLODSP module on the FPGA downsamples the output data by a factor of 100. This
means that for an input sampling rate of 1MSPS (the maximum sampling rate of the BOLO8BLF), data will be output at 10kSPS\@. This means that the FPGA
filters should be set to a maximum of 5kHz, though in practice there is little point going above about 1.5kHz.

\subsection{Voltage Offset}%
\label{sec:offset}
No sensor is perfect, in the sense that the bridge will never be perfectly balanced in the absence of any input power. Likewise, no electronics is perfect,
and there will always be some cross talk between excitation and measurement voltages. This leads to a voltage offset with no input power:
\begin{equation}
  \label{equ:voff}
  V_{off} = A_{off}e^{i\phi_{off}}
\end{equation}
Note that the offset voltage has both an amplitude \textit{and} phase, which may be different to the phase of the signal due to incident power, making the
measured voltage a complex signal. It is necessary to subtract this offset from the measured signal using a suitable complex subtraction, either by
decomposing into real ($I$) and imaginary ($Q$) parts and subtracting the real and imaginary parts of the offset respectively, or by performing a
vector-based subtraction using the cosine rule.

The FPGA firmware has the capability to perform the first of these methods on-chip, using user-loadable offset values for each channel
(Section~\ref{sec:offset_correction}). When offset subtraction has been correctly performed, the remaining phase should be constant (being only a property
of the geometry), and the remaining amplitude will be a measure only of the bridge imbalance (and hence the temperature change of the sensor foil). The
phase is therefore a useful measurement for data validation.

\subsection{Real-Time Power Output}%
\label{sec:rtpower}
The physics involved in calculating the power aborted by the foil from the resistance change of the meander resistors is complex, particularly when an AC
excitation voltage is involved. For a thorough treatment of the subject, see Ref~\cite{giannone-2002}. However, the following simplified formula gives a
good approximation of the more complex result:
\begin{equation}
  \label{equ:pbol}
  P = \frac{1}{S}\left(A + \tau \frac{\mathrm{d}A}{\mathrm{d}t}\right)
\end{equation}
Here, $S$ is the sensitivity of the sensor, and $\tau$ is the cooling time. Both these values can be calculated using an in-situ calibration procedure, as
described in Section~\ref{sec:calibration}.

The flexibility of the FPGA's digital filtering means it is possible to design a filter to both perform the time averaging described in
Section~\ref{sec:voltage}, and adding to this the time derivative as in Equation~\ref{equ:pbol}. Using this filter (known as a ``deconvolution'' filter)
will produce an estimate of the incident power straight from the bridge voltage measurement, with a latency of about 1ms, which should be quick enough to
use in a real-time control loop. The design of this filter requires an accurate calibration, and is discussed in Section~\ref{sec:calibration}.

\subsection{Calibration}%
\label{sec:calibration}
It can be seen from Equation~\ref{equ:pbol} that at least 2 quantities need to be determined from a calibration: the sensitivity $S$ and the cooling
time $\tau$. The sensitivity can be determined by measuring the change in the voltage output for a known input power, and the cooling time can be
calculated by fitting an exponential decay to the voltage curve as the sensor cools immediately after removing all input power. In the BOLO system,
the input power is provided by ohmic heating, by applying a DC bias (or offset) to both $V_{in}$ and $V_{out}$, whilst at the same time keeping the
AC excitation voltage of $V_{in}$ to simulate operation with radiation heating as closely as possible.

\begin{figure}
  \centering
  \def\svgwidth{\columnwidth}
  \input{calibration.pdf_tex}
  \caption{Electrical schematic of the calibration procedure\label{fig:calibration}}
\end{figure}

The calibration procedure is described in some detail in Ref~\cite{lovell-2015}. Figure~\ref{fig:calibration}, based on that paper, shows the calibration
procedure. The application of $V_{OS}$ is such that current flows through the two measurement resistors $R_M$, but not through the reference resistors
$R_R$. This causes only the measurement resistors to be ohmically heated, with the total heating power for both measurement resistors given by
\begin{equation}
  \label{equ:pohmic}
  P_{OH} = 2 V_{OS} I_{OS}
\end{equation}
Note that the voltages applied to the bridge are differential: the quantities $V_{AC}$ and $V_{OS}$ are those requested of the BOLO DACs, and $V_{OS}$ and
$I_{OS}$ are the quantities measured by the ADC.\@ The differential nature of the signal results in the signs and factors of 2 present in the voltages at
each point in the circuit.

When the ohmic heating is switched off, the sensor cools. The bridge voltage (amplitude and phase) measured during cooling provides a number of useful
calibration values. They must however be converted into Cartesian representation ($I$ and $Q$) first:
\begin{itemize}
\item{The offsets $A_{off}$ and $\phi_{off}$ can be calculated using Equations~\ref{equ:I} and~\ref{equ:Q}, using $I_{off} = I(t\rightarrow\infty)$ and
    $Q_{off} = Q(t\rightarrow\infty)$}
\item{The cooling time can be calculated by fitting an exponential decay of the $I$ and $Q$ cooling curves. The decay time constant is the cooling time.
    It should be the same (to within errors) for both curves.}
\item{The sensitivity is given by $S = \frac{P_{OH}}{\Delta A}$, where $P_{OH}$ is given by Equation~\ref{equ:pohmic} and $\Delta A$ is given by
    Equation~\ref{equ:A}, using $\Delta I = I(t=0) - I_{off}$, and $\Delta Q = Q(t=0) - Q_{off}$.}
\end{itemize}
During the calibration, the FPGA measures $V_{OS}$ and $I_{OS}$, in addition to $A$ and $\phi$, thus providing all the required data for a
calibration in one place. For consistency, we shall refer in the rest of this guide to $V_{OH}$ and $I_{OH}$, which are equal to $V_{OS}$ and
$I_{OS}$, to minimise confusion with the voltage offsets $I_{off}$ and $Q_{off}$ and the current measurement offset $I_{off}$ (see
Section~\ref{sec:scaling}).

\section{Data Outputs}%
\label{sec:outputs}
\subsection{Output Channel Format}%
\label{sec:channels}
For each physical channel on a BOLO8, there are 3 ``logical'' output channels. This means each BOLO8 site has 24 output channels, each of which can be read
like any other D-TACQ output (see the D-TACQ 4G user guide for details). All channels are 4-byte signed integers. The channels are arranged as follows:
\begin{enumerate}
\item{The voltage amplitude $A$ from the quadrature synchronous detection process (Section~\ref{sec:voltage}).}
\item{The phase $\phi$ from the quadrature synchronous detection process (Section~\ref{sec:voltage}).}
\item{The third channel depends on the operating mode:}
  \begin{enumerate}
  \item{During normal mode: the real-time power calculation from the deconvolution process (Section~\ref{sec:rtpower}).}
  \item{During calibration mode: a combination of the ohmic heating voltage $V_{OH}$ and current $I_{OH}$ (Section~\ref{sec:calibration}). The top 2 bytes of
      each 4-byte sample are the voltage, and the bottom 2 bytes are the current. In other words, this output is $V_{OH} \times 2^{16} + I_{OH}$. In this
      scenario, $V_{OH}$ is a signed 2-byte integer, and $I_{OH}$ is an unsigned 2-byte integer.}
  \end{enumerate}
\end{enumerate}

For a physical channel $N$ (numbered 1--8 per site), logical channel $3N-2$ is the voltage amplitude, $3N-1$ is the voltage phase and $3N$ is either the
real-time power or calibration quantities. So for example, for physical channel 3, logical channel 7 is the amplitude, 8 is the phase and 9 is
power/calibration signals.

\subsection{Data Scaling}%
\label{sec:scaling}
The following are the numerical conversion factors to convert from raw counts (the 2- or 4-byte values read from the device) into physical units. There is
a dependence on $V_{gain}$, which is the voltage gain used on the BOLO8's ADCs. The value can be read back as a knob in CSS or EPICS (or your favourite data
acquisition tool); contact D-TACQ for the details of setting and reading these values. The recommended gain is 1V2 (1.25V), since the bolometer signals are
typically small (10s to a few 100s of mV), even with strong radiation.

There is also a dependence on the CORDIC scale factor:
\begin{equation}
  \label{equ:cordic}
  \begin{split}
    Z_{30} &= \prod_{i=1}^{30}\sqrt{1 + 2^{-2i}} \\
    &= 1.1644353\ldots
  \end{split}
\end{equation}
This is due to the algorithm used to convert $I$ and $Q$ into $A$ and $\phi$ on the FPGA\@.

The scalings are as follows:
\begin{equation}
  \label{equ:avolts}
  \begin{split}
    A(V) &= A(counts) V_{gain} \frac{20}{18 \times 2^{24} \times Z_{30}} \\
    &= A(counts) V_{gain} \times 5.6875105\ldots \times 10^{-8}
  \end{split}
\end{equation}

\begin{equation}
  \label{equ:phirad}
  \begin{split}
    \phi(rad) &= \phi(counts) \times 2^{-29} \\
    &= \phi(counts) \times 1.8626451\ldots \times 10^{-9}
  \end{split}
\end{equation}

\begin{equation}
  \label{equ:pwatts}
  \begin{split}
    P(W) &= P(counts) V_{gain} \frac{20}{18 \times 2^{18} \times Z_{30}} \\
    &= P(counts) V_{gain} \times 3.6400067\ldots \times 10^{-6}
  \end{split}
\end{equation}

\begin{equation}
  \label{equ:vdcvolts}
  \begin{split}
    V_{OH}(V) &= V_{OH}(counts) V_{gain} \times 2^{-15} \\
    &= V_{OH} V_{gain} \times 3.0517578\ldots \times 10^{-5}
  \end{split}
\end{equation}

\begin{equation}
  \label{equ:curramps}
  \begin{split}
    I_{OH}(A) &= I_{OH}(counts) \times \frac{128}{100} \times \frac{25}{3 \times 2^{12} \times 1000} - I_{off} \\
    &= I_{OH}(counts) \times 2.6041666\ldots \times 10^{-6} - I_{off}
  \end{split}
\end{equation}

$I_{off}$ in Equation~\ref{equ:curramps} is the current reading when $V_{OH} = 0$, i.e.\ no bias voltage is applied. In theory it should be around
$20.83 \times 10^{-3} \mathrm{A}$ but will vary slightly between channels. The most reliable way to measure $I_{off}$ is to take the mean value of
the current during the cooling part of the calibration.

\section{Control Knobs}%
\label{sec:knobs}
This section describes the control knobs available for the BOLODSP module. They are read and set in the same way as standard D-TACQ knobs using the remote
command interface, with the BOLODSP module treated as Site 14. See the D-TACQ 4G user guide for details.

\subsection{Low-level knobs}%
\label{sec:knobsll}
These control knobs are a fairly low level interface to the underlying FPGA module. For the most part they will be set by utility scripts. The ones which
may require the user to manually set them (or at least write a script to do so during the shot cycle) are documented here.

\subsubsection{DSP{\_}RESET}
Set to 1 to reset the BOLODSP firmware module. Set to 0 to remove from reset. To ensure correct channel ordering and an accurate time base for the data,
the module should be reset before each acquisition (transient or streaming). To perform a reset, run the \mbox{\path{/usr/local/bin/reset.dsp}} command.

\subsubsection{STEP{\_}SIZE}
This is used to generate the AC excitation voltage to the sensor. The excitation voltage is a sine wave of 18V amplitude. Note that the electronics is
capable of $\pm 20\mathrm{V}$ output; the 2V headroom is left to enable DC biasing during calibration. Due to implementation details in the BOLODSP
module, the STEP{\textunderscore}SIZE value $S$ required to produce an excitation voltage of $f$ Hz is given by:
\begin{equation}
  \label{equ:step_size}
  S = \frac{f}{f_{clk}} \times 2^{27}
\end{equation}
Here, $f_{clk}$ is the system clock frequency. Is is typically 100 MHz on older D-TACQ systems, and 125 MHz on newer systems. Contact D-TACQ if you are
unsure which frequency to use (or pick one and check the output on a scope to see if it's the correct frequency).

There also exists a script, \mbox{\path{/usr/local/bin/set.fdrive}~\texttt{<freq>}}, which takes a frequency in Hz and automatically sets the register
correctly. By default it assumes 100 MHz system clock, but this can be changed by modifying the script contents (contact D-TACQ or the author for
details).

\subsubsection{CALIBRATION}
Set to 1 to put the device in calibration mode, 0 to put it in normal mode. This changes the data output in the third logical channel for each physical
channel (Section~\ref{sec:channels}). This is done automatically by the calibration script in Section~\ref{sec:run_calibration}.

\subsubsection{FILTER{\_}STATUS}
This is a read-only knob, and indicates the readiness of the filters on the FPGA to be reloaded. It reports a hexadecimal value, and should be
\texttt{0x33} before attempting any reload. If it reads anything else, ensure you have read back any previous capture data and then run %chktex 29
\mbox{\path{/usr/local/bin/wait_for_filters}}, which will do a short capture to pass data through the filters so they are ready for the next reload.

\subsection{High-level knobs}%
\label{sec:knobshl}
These control knobs interface with the BOLODSP module at a higher level. Their inputs are in physical units, rather than register settings, and the
utility scripts read these knobs in order to set the lower-level knobs. All high-level knobs presented by the BOLODSP package are described here, and
should be set by the user.

\subsubsection{DIODE{\_}DROP{\_}V}
This value, in volts, sets the extra voltage to apply to one diagonal of the bridge to compensate for the voltage drop across diodes in the electronics.

The electronics features some diodes for noise reduction. There will be a voltage drop across these diodes, meaning that the applied bias voltage $V_{OH}$
in calibration will be less than the requested voltage on one of the bridge diagonals. Set this knob (in volts) to apply an extra voltage across that
diagonal to compensate for the voltage drop. The value should be tweaked until the measured ohmic heating voltage $V_{OH}$ (Section~\ref{sec:channels}) is
close to that set by the VBIAS knob. As a guide, when VBIAS is set to 1V, the diode voltage drop is about 0.5V for a $1.2\mathrm{k}\Omega$ bolometer
sensor.

\subsubsection{FILTER{\_}BANDWIDTH}
As mentioned in Section~\ref{sec:voltage}, the choice of filter affects the trade-off between noise levels and time resolution. The filters themselves are
designed using helper scripts (Section~\ref{sec:lpfdesign}). Those scripts take the value of this knob to design a filter with a given bandwidth. The
bandwidth should be specified in Hz. The maximum supported bandwidth is 1950Hz for use with the filter helper scripts.

\subsubsection{TWAIT}
This value, in seconds, specifies the length of time to wait at the start of the calibration before applying the bias voltage to heat the sensor.
This can be useful for checking the offset calculated from the calibration cooling curve.
A value of \SI{0.1}{s} should be sufficient for most applications.

\subsubsection{THEAT}
This value, in seconds, specifies the length of time spent ohmically heating the sensor during calibration. It should be long enough to ensure the sensor
has reached equilibrium, i.e.~several cooling times.

\subsubsection{TCOOL}
This value, in seconds, specifies the length of time to continue measuring the calibration curve after ohmic heating is turned off. It thus controls the
amount of the cooling curve measured. Like with THEAT, it should be set to multiple cooling times such that the sensor is nearly back in thermal
equilibrium at the end of the measurement.

\subsubsection{VBIAS}
This value, in volts, specifies the bias voltage $V_{OH}$ to be used to ohmically heat the bridge during calibration. It must be $<1.2\mathrm{V}$. The
required setting for DIODE{\_}DROP{\_}V will typically depend upon this knob.

\subsubsection{CAL{\_}EN}
Set this to 1 to enable a calibration before a shot. You should ensure that the data acquisition system of which the unit is a part is set up to run 2
acquisitions during the shot cycle: one to acquire the calibration data and a second to acquire the shot data.

This knob is ignored by the BOLODSP module and software, but may be helpful for user-written shot-cycle scripts to examine and work out what they need to
do.

\subsubsection{CAL{\_}DELAY}
This value, in seconds, specifies the length of time to wait between the start of a shot cycle (i.e.~prepare/arm) and the start of a calibration
procedure. It allows the user to cause the calibration procedure to run at a known time in the shot initialisation sequence. This could for example mean
performing the calibration as close as possible to the start of the shot, for conditions as similar as possible during the calibration and shot, or
performing the calibration earlier before sources of electrical noise such as coil power supplies are switched on.

Like the CAL{\_}EN knob, this is ignored by the BOLODSP module and software, but exists to allow user-written shot-cycle scripts to examine it and work
out when they need to call the calibration script.

\section{Operation}%
\label{sec:operation}
This section documents the usage of the various helper scripts throughout a shot cycle. Before a shot, a calibration should be performed (possible before
every shot, but not strictly necessary), and the results of the calibration should be used to remove offsets from the signal and design the filters for
the voltage and power output. After every capture (shot or calibration run) the BOLODSP module should be reset, using the
\mbox{\path{/usr/local/bin/reset.dsp}} script, to ensure the correct channel ordering and to clean out any stale data from the signal processing
module.

Refer also to the data capture section of the D-TACQ 4G user guide for details of how to set up a transient capture.

\subsection{Calibration}%
\label{sec:run_calibration}
A calibration run proceeds as follows. These steps should be performed by a user-written script, perhaps as part of a shot cycle script.
\begin{enumerate}
\item{Set the high-level knobs described in Section~\ref{sec:knobshl}, and set the STEP{\_}SIZE low-level knob to give the desired excitation
    frequency.}
\item{Reset the DSP system, by running the \mbox{\path{/usr/local/bin/reset.dsp}} script. This ensures the channel ordering and time bases will be
    correct.}
\item{Set the offsets for all voltage channels to 0 (Section~\ref{sec:offset_correction})}.
\item{Set up a transient data capture, set to wait until the system receives soft trigger, to acquire at least $(0.1 + T_{HEAT} + T_{COOL})\times 10000$ samples. For example, if $T_{HEAT} = T_{COOL} = \SI{1}{s}$:\\
    \verb!set.site 0 `transient POST=21000 SOFT_TRIGGER=0; set_arm'!\\
    The system is now ready to perform the calibration.}
\item{Run the \mbox{\path{/usr/local/bin/bolo_calibration}} script, which controls the hardware to perform a calibration procedure, as described in
    Section~\ref{sec:calibration}. To run it, a user or script must have SSH access to the device. It reads the voltage and timing high-level knobs, and
    proceeds as follows:
\begin{enumerate}
\item{Put the device in calibration mode, so that $V_{OH}$ and $I_{OH}$ are recorded, instead of real-time power.}
\item{Trigger the data collection.}
\item{Wait 0.1 seconds to measure the voltages and current with no heating applied (this can be helpful in post-processing).}
\item{Turn the ohmic heating on, and wait for THEAT seconds}.
\item{Turn the ohmic heating off, and wait until data collection finishes.}
\item{Put the device in normal mode, so that the real-time power will be output on the next capture.}
\end{enumerate}
}
\item{Reset the DSP system again before the shot, to ensure the correct channel ordering and time base.}
\end{enumerate}

The capture data should be read back before the shot capture is started, as the latter will overwrite the calibration data with shot data. The procedure
takes THEAT + TCOOL + a few seconds to complete, plus the time to read back the calibration data. It is advisable to allow at least 30 seconds from
starting a calibration to the expected shot trigger, just in case, but the procedure is quick enough to be performed before every shot if needed. Of
course, a calibration can also be performed separately, by following this procedure manually.

\subsubsection{The calibfit utility}%
\label{sec:calibfit}
Newer versions of the BOLODSP package (those built after 16th December 2016) include the \mbox{\path{/usr/local/bin/calibfit}} command. This can be
called after a calibration run has been performed and the data captured with a transient capture, and will automatically perform curve fitting on the
data to calculate the sensitivity, cooling time and voltage offsets for a given channel. These can be used to load the offset registers
(Section~\ref{sec:offset_correction}) and design the deconvolution filters for the real-time power output (Section~\ref{sec:lpfdesign}) without having
to read the data from the device, calculate the fit parameters offline and then send the correct parameters back to the device.

The \texttt{calibfit} utility works on a single channel, supplied via a command line argument. Using $V_{OH}$ it determines what part of the time trace
is heating, and which part is cooling: if $V_{OH}$ is above a threshold value then the sensor is being ohmically heated at that time, and if it is
below a threshold value after a given time (to allow for the short pause for offset measurement at the start of the calibration) then it assumes
cooling is occurring. The utility thus requires that the calibration has been performed in the manner described in Section~\ref{sec:run_calibration},
preferably using the \texttt{bolo\_calibration} utility. The wait time and heating/cooling thresholds are configurable as command line parameters.

From the heating phase, \texttt{calibfit} calculates the applied ohmic heating power using $P_{OH} = 2 V_{OH} I_{OH}$, which is used in the sensitivity
calculation. From the cooling phase, it calculates $I$ and $Q$ using Equations~\ref{equ:I} and~\ref{equ:Q} and then fits the following curve (same
for $Q$):
\begin{equation}
  \label{equ:Icool}
  I_{cool} = \Delta I e^{-t / \tau} + I_{off}
\end{equation}
Here, $t$ is the time since cooling started, and $\Delta I$, $\tau$ and $I_{off}$ are the fit parameters (for performance reasons $1/\tau$ is fitted,
but the program outputs $\tau$). To generate the time vector $t$ for the fit, the program assumes a sample rate of 10kSPS, which is the expected output
sample rate if setting the BOLO8s to sample at 1MSPS\@.

The sensitivity is calculated from $\Delta I$, $\Delta Q$ and $P_{OH}$ as described in Section~\ref{sec:calibration}. The cooling time is the average
of the $I$ and $Q$ fits, since the cooling time is independent of the phase (and therefore should be identical for both $I$ and $Q$ fits if the fits
were perfect and free of noise). The offsets $A_{off}$ and $\phi_{off}$ are calculated from $I_{off}$ and $Q_{off}$ using Equations~\ref{equ:I}
and~\ref{equ:Q}, though both polar and Cartesian offsets are output by the utility, so offset correction can be done either on-chip
(Section~\ref{sec:offset_correction}) or in post-processing using the user's preferred method (see Section~\ref{sec:offset}).

The \texttt{calibfit} utility takes the following command line options:
\begin{verbatim}
calibfit [-c channel] [-C cooling_theshold]
         [-H heating_threshold] [-t t_wait] [-T tau_guess]
\end{verbatim}
The options are as follows:
\begin{itemize}
\item{\texttt{channel}: The physical channel to use. Default=1}
\item{\texttt{cooling\_threshold}: Assume cooling if the voltage is below this threshold (in V), and after t\_wait. Default=0.001}
\item{\texttt{heating\_threshold}: Assume heating if the voltage is above this threshold (in V). Default=0.95}
\item{\texttt{t\_wait}: Assume no cooling occurs before this time (in seconds). Default=0.2}
\item{\texttt{tau\_guess}: Initial guess at the cooling time (in seconds), to aid the fit. Default=0.2}
\end{itemize}

A sample output of the program is below, done for a calibration of a sensor in air:
\begin{verbatim}
acq2106_006> calibfit -c13
Successfully read 21000 samples
Fit parameters for i: 0.003383145	16.24632	-0.02794252
Fit parameters for q: -0.003408019	16.20701	0.02941226
Fitting complete. Fit parameters:
sens = 6.098115
tau = 0.06162704
a0 = 0.08113854
phi0 = -2.330575
i0 = -0.02794252
q0 = 0.02941225
\end{verbatim}
The output is printed to \texttt{stdout}. It is the user's responsibility to redirect the output to a file to record for posterity, and/or parse the
output to have the device automatically load the offset registers (Section~\ref{sec:offset_correction}) and the real-time power filter
(Section~\ref{sec:lpfdesign}).

\subsubsection{Example calibration data}%
\label{sec:example-calibr-data}

We have provided screen grabs of the 3 calibration traces, shown in CSS using the plot transient tab, to give an example of what a successful calibration of a bolometer sensor should look like.
Note that the precise values will vary significantly with different hardware.
In particular, the phase values will change with cable length, and if the phase is positive then the amplitude signal will look like it is inverted compared to these examples.
Successful offset subtraction (see Section~\ref{sec:offset} and Section~\ref{sec:offset_correction}) will correct for this.

Figure~\ref{fig:example-acal} shows the voltage amplitude, without any offset correction applied.
Figure~\ref{fig:example-phical} shows the phase, again without any offset correction.
Figure~\ref{fig:example-dc} shows the DC channel, which contains both the DC bias voltage and the current applied to the bridge in the form $V_{DC} \cdot 2^{16} + I_{DC}$.

\begin{figure}
  \centering
  \includegraphics[width=\textwidth]{acal_CSS.png}
  \caption[Example trace for the bridge amplitude for a calibration]{Example trace for the bridge amplitude for a calibration.
    There are 3 regions: the first 1.5E3 samples with no heating indicates the amplitude offset.
    Then there are approximately 1E4 samples of heating, where the bridge heats up (shown an an exponential curve) as the DC bias voltage is applied.
    Finally, the last 1E4 samples show the cooling curve: the exponential decay of the bridge amplitude as the measurement resistors return to equilibrium.
    Note that the signal appears to be inverted here (an increase in the bridge temperature reduces the amplitude), but this is simply due to the phase being negative.
    After performing offset correction this particular trace will be inverted, and the first 1.5E3 samples will be approximately 0.
    Different cabling setups affect the phase, so this trace may or may not be inverted for other users (though whether or not a given channel is inverted should be the same across successive calibrations if the hardware arrangement remains the same).\label{fig:example-acal}}
\end{figure}

\begin{figure}
  \centering
  \includegraphics[width=\textwidth]{phical_CSS.png}
  \caption[Example trace for the phase for a calibration]{Example trace for the phase for a calibration.
    The phase changes relatively little, though there is a noticable different between heating and cooling due to the internal hardware of the BOLO8 board.
    Note that the phase in this case is negative, which causes the amplitude  trace to be inverted: it may not be in other setups with different cabling.\label{fig:example-phical}}
\end{figure}

\begin{figure}
  \centering
  \includegraphics[width=\textwidth]{VDC_CSS.png}
  \caption[Example trace for the DC bias in a calibration]{Example trace for the DC bias in a calibration.
    This includes both the DC bias voltage and the current flowing through the resistors.
    The initial 1.5E3 samples with no heating, the 1E4 samples of heating and the 1e4 samples of cooling can clearly be seen.
    Since this quantity is actually $V_{DC} \cdot 2^{16} + I_{DC}$, even a small negative voltage during the cooling will appear to produce a large negative quantity in CSS.\@
    In this case, the heating voltage was \SI{1}{V}, and during cooling \SI{0.01}{V} was measured.
    The current measurement, being $2^{16}$ times smaller than the voltage measurement, is impossible to see.\label{fig:example-dc}}
\end{figure}

\clearpage % Place these figures before the next section

\subsection{Offset Correction}%
\label{sec:offset_correction}
As described in Section~\ref{sec:offset}, the voltage measured by the BOLO8 will include an offset due to intrinsic bridge imbalance and cross talk in the
electronics hardware and cabling. The offset can be measured from a calibration, either from the initial 0.1s equilibrium method or the asymptotic voltage
calculated from the cooling curve --- both measurements should be equal. It is possible to remove this offset in post processing, but the BOLODSP FPGA
module also has the ability to remove it during on-chip processing. This is particularly necessary for the real-time power signals if they are to be used
in a control loop, since no phase information is kept.

The on-chip offset correction consists of a series of registers which contain the $I_{off}$ and $Q_{off}$ values for each channel, followed by a second
series of registers which contain $P_{I,off}$ and $P_{Q,off}$ values, where $P_{I,off} = I_{off}/S$ and $P_{Q,off} = Q_{off}/S$ are the offsets in the
real-time power signals. The script \mbox{\path{/usr/local/bin/load_offset_channel.tcl}} can be used to load these registers for a particular channel. It
takes 4 mandatory arguments:
\begin{enumerate}
\item{\texttt{channel}: the channel to load, numbered from 1 to 48 (8 channels per site).}
\item{\texttt{I0}: the offset value $I_{off}$ in V.}
\item{\texttt{Q0}: the offset value $Q_{off}$ in V.}
\item{\texttt{sens}: the sensitivity, as described in Section~\ref{sec:calibration}, in V/W.}
\end{enumerate}

The offsets can be loaded at any time, including just before a shot. This is important, as the offset measured in a calibration long before a shot may not
be exactly equal to the offset when the machine is running with a full complement of power supplies and magnetic fields potentially affecting the sensor
and cabling. The neutral gas pressure may also affect the offset. Accurately predicting the offset drift throughout a shot cycle is an interesting area of
research~\cite{zhang-2012}.

\subsection{Filter Design}%
\label{sec:lpfdesign}
As described in Section~\ref{sec:voltage}, the calculation of the voltage amplitude and phase involves time averaging, which is performed on the D-TACQ
unit with a low-pass filter. In addition, the real-time power calculation uses a deconvolution filter, as described in Section~\ref{sec:rtpower}. The
bandwidth for both of these filters is set using the FILTER{\_}BANDWIDTH control knob (Section~\ref{sec:knobshl}).

The filter used to calculate the voltage is designed and loaded onto the FPGA by the \mbox{\path{/usr/local/bin/lpfdesign.tcl}} script. It takes no
arguments, and loads a single filter which is used by all channels.

The deconvolution filter used to calculate the power is designed and loaded by the \mbox{\path{/usr/local/bin/power_filter_design.tcl}} script. It takes 3
arguments:
\begin{enumerate}
\item{\texttt{channel}: the channel to load, numbered from 1 to 48 (8 channels per site).}
\item{\texttt{sens}: the sensitivity of the channel as calculated from a calibration, in V/W.}
\item{\texttt{tau}: the cooling time for the channel as calculated from a calibration, in seconds.}
\end{enumerate}
Since each channel will have different calibration factors, a different deconvolution filter must be loaded for every channel being used. Including the
sensitivity in the filter design allows the same scaling factor (Equation~\ref{equ:pwatts}) to be applied to every channel to convert to physical units
afterwards. When loading a new deconvolution filter, you should also update the offsets (Section~\ref{sec:offset_correction}) for that channel to
match. Like with the voltage filter, all deconvolution filters will have the same bandwidth, given by the control knob.

\textbf{Important}: between each deconvolution filter channel reload you \textbf{must} run \mbox{\path{/usr/local/bin/wait_for_filters}}, as described in
the FILTER{\_}STATUS knob in Section~\ref{sec:knobsll}. Failure to do so will lock up the system and require a hard reset.

\section{Conclusion}
This document describes the theory behind the BOLODSP module, which is intended to complement the D-TACQ BOLO8 hardware module. It also describes the
various settings and scripts used to operate the unit as a useful bolometer electronics back-end. For more detailed information about the system, consult
the references.

Finally, this is a very new product. There are likely to be bugs or undesirable ``features'', as in every new product (and indeed some very established
products). The source code for the software (and soon the FPGA firmware module) is freely available, and offers of collaboration for fixes or enhancements
will be warmly received.

\bibliographystyle{ieeetr}
\bibliography{refs}

\end{document}


%\vspace{-5pt}
\section{Experiments and Results}
\label{sec:results}

In this section, we evaluate the methods outlined in Section~\ref{sec:methods},
on Linux kernel and Twisted library. We first present the accuracy of
predictions on the test set, and then present some interesting predictions that
come out from recurring patterns. We implemented the matrix vector model in
Python, and the feed-forward and recurrent models in Keras~\cite{ref:keras}.
Our code is available on github~\cite{ref:codecompletion}.

\subsection{Accuracy}
\label{sec:accuracy}

\begin{table}[h]
  \centering
%  \small {
  \begin{tabular}{l l l r r r r r}
    \hline
    Method & Known & Abs & Top 3 & Key & Pos \\
    Win, \#Keys & acc. & acc. & acc.  & acc. & acc. \\
    \hline
    NL-3, 40, 2000 & 64.4 & 53.4 & 81.7 & 78.0 & 43.7\\
    NL-4, 64, 2000 & 63.2 & 50.6 & 82.0 & 72.8 & 41.8\\
    Attn, 40, 1000 & 67.5 & 56.0 & 83.6 & 79.9 & 48.4\\
    BestKW-RandPos & 49.8 & 41.3 & -- & 79.9 & 7.0\\
    \hline
  \end{tabular}
%  }
  \caption{Test accuracy (\%) of predictions for Linux project}
  \label{tab:linux}
\end{table}

\begin{table}[h]
  \centering
%  \normal {
  \begin{tabular}{l l l r r r r r}
    \hline
    Method & Known & Abs & Top 3 & Key & Pos \\
    Win, \#Keys & acc. & acc. & acc.  & acc. & acc. \\
    \hline
    NL-3, 40, 500 & 46.3 & 39.2 & 64.8 & 64.1 & 14.2\\
    NL-4, 40, 500 & 44.8 & 37.7 & 62.2 & 63.9 & 10.8\\
    Attn, 40, 500 & 47.9 & 40.6 & 66.5 & 64.6 & 17.6\\
    GRU, 40, 500 & 41.6 & 35.2 & 59.1 & 64.4 & 4.9\\
    BestKW-RandPos & 43.6 & 37.5 & -- & 64.4 & 5.5\\
    \hline
  \end{tabular}
%  }
  \caption{Test accuracy (\%) of predictions for Twisted project}
  \label{tab:twisted}
\end{table}

\noindent
Table~\ref{tab:linux} shows results for Linux source, a C project, and
Table~\ref{tab:twisted} for Twisted, a Python networking library. As mentioned
in Section~\ref{sec:dataset}, we use half the files for training and half for
testing.
Column 1
lists the learning method, window size and number of key words.
NL-3 is a feed forward model with 3 non-linearity layers, and NL-4 is a feed
forward model with 4 non-linearity layers. Attn. computes a weighing
parameter for each input token in the window, and uses 3 non-linearity layers.
GRU is a recurrent model. We do not report numbers for GRU in the Linux codebase
since it gave poor results in the smaller Twisted codebase.
Additionally, we report numbers for a fictional random
predictor that achieves the accuracy of the best predictor in case of keywords,
but in case of a non-keyword token predicts one randomly from the non-keyword
tokens in the window, to show the significance of positional tokens.
We name this BestKW-RandPos.

Known acc. is the
accuracy for predictions for the cases where the next token is a key token, or
a positional token from the window being considered. Abs acc. is the accuracy
for predictions for all cases, including cases where the next token is
neither a key word nor a seen positional token. These are often new function
names or variable names, or variables outside of our window. Top 3 is the
percentage of cases where the next token is within top 3 predictions, ignoring
the unknown cases (unseen variable and function names).
We report this number, because generally, we are interested in the top few
suggestions as opposed to the top suggestion with a code recommendation system.
Key acc. is the
accuracy with which we predict key tokens correctly, given the next token is a
key token,
and pos. acc. is the accuracy with which we predict positional tokens
correctly, given the next token is a positional token.
These give us insight into how well our methods predict tokens that are not key
tokens.

We found that attention gives the best results among all the methods we tried.
This makes sense because it is also the most general of all models --- it
directly connects all inputs to the output, (unlike GRU where computed states
are chained), and it has an additional attention parameter, that is just a
constant in the simple feed-forward case. We also found that the feed forward
model with 4 non-linearity layers tends to overfit the training data, giving a slightly
lower accuracy on test data. Increasing the word vector dimensions also tends
to result in overfitting. Though regularization can help with overfitting, we
found that reducing vector dimensions and number of layers was also sufficient
to reduce high variance from overfitting.

The accuracy of predicting non-key words, that is, positional tokens is quite
high for Linux kernel, a C project, indicating there is a lot of redundancy/
recurring patterns. The positional token accuracy is lower for Python.
We think this is because Python is a dynamically typed, high level
language with less redundancy, and more terse syntax.
However, in both cases, the top 3 prediction
accuracy is still much higher than a random predictor, indicating that the
model does extract some patterns out of the code.

A plot of the word vectors reveals a significant clustering of similar
words --- {\texttt {uint8\_t}} and {\texttt {uint16\_t}}, different locking and
unlocking calls, and integers. We have omitted these plots from the report for
space. Unlike natural language texts, however,
we did not observe any significant vector relationships between different
tokens.


\section{Model Functions}
The code performs the following functions:
\begin{itemize}
	\item \textbf{Reset}: Resets the rwMotorVoltage module to original settings.
	\item \textbf{Closed loop}: Evaluates commanded torque based on closed loop controls.
	\item \textbf{Torque to Voltage}: Uses gains to convert a given torque to a voltage output.
	\item \textbf{Saturation}:  Checks that calculated output is within min/max bounds.
\end{itemize}

\section{Model Assumptions and Limitations}
This code makes the following assumptions:
\begin{itemize}
	\item \textbf{Linear}: This code assumes that there is a linear relationship between the torque desired and the voltage required to create that torque.
\end{itemize}