\documentclass{article}

\usepackage{geometry}[a4paper, margin=2in]
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{hyperref}
\hypersetup{urlcolor=blue, colorlinks=true}

\newcommand{\f}[2]{\frac{#1}{#2}}
\newcommand{\D}[2]{\frac{d#1}{d#2}}
\newcommand{\DD}[2]{\frac{d^2 #1}{d {#2}^2}}
\title{Vibrations and Harmonic Oscillations}
\author{Aayush Arya}
\date{(Last Updated: \today)}

\begin{document}

\maketitle

{
	\hypersetup{linkcolor=black}
	\tableofcontents
}

\section{Introduction}
The study of vibrations and waves is so central to physics because wave phenomena can arise in virtually every physical scenario, from ripples in water to vibrations in a quartz crystal to gravitational waves. All these waves have one characteric in common \textemdash periodicity.

We will confine our discussion to primarily one type of periodic motion: harmonic motion, in which the disturbance in the concerned physical quantity makes it oscillate sinusoidally. This may apparently sound like constraining our study to a very narrow range of phenomena but it isn't so due to two reasons:

\begin{itemize}
	\item Many systems that otherwise aren't simple harmonic behave so in the special case when the displacement in the physical quantity in context is small
	\item According to a theorem by Joseph Fourier (1807), all periodic waves can be written as a converging summation series of infinite harmonic terms.
\end{itemize}

We begin with one of the most trivial examples in the study of harmonic motion: the simple case of a single block attached to a spring fixed to a wall.

	\subsection{Block on a spring}
	This type of system can be described by the differential equation of motion
	$$ m\DD{x}{t} = -(k_1x + k_2x^2 + ... + k_nx^n) $$
	For small displacements $x$, the system can be assumed to behave linearly. It should be emphasized that the assumption of linearity is only an approximation. The equation thus reduces to a second-order ordinary linear differential equation

	$$ m\DD{x}{t} + k_1x = 0 $$

	Solving this equation is rather trivial and it can be easily verified that it satsfies the solution

	$$ x = Asin(\omega t + \phi)$$
	where $ \omega = \sqrt{\f{k_1}{m}}$ .
	This equation corresponds to a sine curve with amplitude $A$ in the $x-t$ plane ranging from $ -\infty < x < \infty $. However, any \textit{physical} vibration would be recorded in a given time interval. If the oscillation begun at $t=t_1$ and ended at $t=t_2$ then we must impose the following initial and boundary conditions

	$$ x = \begin{cases}
	0 & -\infty < x < t_1\\
	Asin(\omega t + \phi) & t_1 \leq x \leq t_2\\
	0 & t_2 < x < \infty
	\end{cases}$$

	The description of a \textit{physical} harmonic oscillator requires such conditions to be imposed on the mathematical solution. It should be noted that any real vibration appears to be simple harmonic only in the case when it occurs for a large number of time periods. This is again a problem of Fourier analysis and will be discussed later. Still, it should be noted that the simple harmonic picture can be used to describe the system only in the \textit{steady state}.
\end{document}


\subsection{Ellipticity as an Estimator of Shear}
% Dodelson chap 10.3 page 315
% barSch page 337


\section{Mathematical Methods}
\label{sec:math}

A small number of mathematical methods which may not be familiar to the reader are used within this thesis. They are outlined below.

\subsection{K-means Clustering}

K-means clustering is a method for cluster analysis, whereby unsorted data is sorted into $k$ distinct groups based on proximity to evolving anchors. Figure \ref{fig:KM1lr} shows the results of a k-means clustering on unsorted data, with colours indicating output cluster designations. It can be seen that the chosen clusters align well with what may have been chosen by a human observer. In this case the groups are relatively well separated, and so the task to the k-means algorithm was relatively undemanding.

\begin{figure}[htbp]
 \includegraphics[max width=\textwidth]{figs/comp/KMeansMarkDemo/1.pdf}
 \caption{A reproduction of Figure \ref{fig:KM1} from Page \pageref{fig:KM1}, which shows the results of a k-means clustering of the data from Figure \ref{fig:corrected} from Page \pageref{fig:corrected} (which shows the \emph{actual} groups.}
 \label{fig:KM1lr}
\end{figure}

The standard algorithm proceeds as follows:

\begin{enumerate}
    \item $k$ random starting positions are assigned.
    \item Each data-point is assigned a group based on which starting position is closest.
    \item The mean location of the data-points in each group is computed.
    \item Steps 2 and 3 are repeated, using the computed means instead of the original starting positions, until a stage where an iteration of these two steps results in no data-point changing groups. Once this occurs, the algorithm is said to have converged, and the resulting groupings are output.
\end{enumerate}

\subsection{Principal Component Analysis}

\textit{A valuable primer on \acrshort{PCA} in relation to colour technology is available from \citet{tzeng_review_2005}.}

\bigskip

\glsreset{PCA}
\Gls{PCA} is a dimensionality reduction method, used to to reduce the number of variables within a dataset whilst retaining as much of the variance as possible, and often used to identify the correlated roots of variance.

\begin{figure}[htbp]
 \includegraphics[max width=\textwidth]{figs/LitRev/PCA.png}
 \caption{Two-dimesional data with the first and second principal components added. Source: \url{https://commons.wikimedia.org/wiki/File:GaussianScatterPCA.svg}}
 \label{fig:PCA}
\end{figure}

Along with similar techniques (such as single value decomposition), it is used extensively within the study of daylight \glspl{SPD} \citep{hernandez-andres_color_2001,ojeda_influence_2012,pant_estimating_2009,bui_group_2004,judd_spectral_1964,maloney_computational_1984,spitschan_variation_2016} and natural \glspl{SRF} \citep{maloney_computational_1984,dzmura_color_1992,maloney_evaluation_1986,maloney_color_1986,cohen_dependency_1964,ferrero_principal_2011,zhang_reconstructing_2008,kwon_surface_2007,agahian_reconstruction_2008,harifi_recovery_2008,parkkinen_characteristic_1989,vrhel_color_1992,fairman_principal_2004,ayala_use_2006,eem_reconstruction_1994-2,connah_multispectral_2006,shi_using_2002,morovic_metamer-set-based_2006}.

%A key goal for many of the researchers listed above... - how many dimensions/sensors do we need

In eras where the transmission of large datasets was troublesome, dimensionality reduction methods such as \gls{PCA} held value as a method to summarise a dataset, with the understanding that a reader could reconstruct a pseudo-dataset with minimal data-loss from the provided principal components. One such example is the work of \citet{judd_spectral_1964} where only the mean and first four characteristic vectors are provided\footnote{This is not technically \gls{PCA} but the related technique of \citet{morris_objective_1954}.}. The data from this study is replotted in Figure \ref{fig:Judd} and it can be seen that for their dataset this description does seem to offer a sensible summary of the data: the shape of the mean can be seen, and it can be seen that further variation principally occurs as a result of $V_{2}$ which is a broad and relatively monotonic function, indicating that changes are likely to be a skewing of the spectral shape.

\begin{figure}[htbp]
 \includegraphics[max width=\textwidth]{figs/LitRev/Judd.pdf}
 \caption{The mean and first four characteristic vectors of \citet{judd_spectral_1964}.}
 \label{fig:Judd}
\end{figure}

\clearpage


\section{Experiment 1}
The given capacitive circuit is a high pass filter.
For the first order LTI system time constant is numerically denoted by tau = R*C.
Time constant helps us to find out how long it will take a capacitor to charge to a certain voltage level
It is time taken by capacitor to charge till 63.2 percent of applied Voltage.
High pass filter allows the signals of high frequency to pass and block the low frequency signals. High pass filter can be constructed by joining the capacitor and resistor in series.

Series RL circuit - \\
In this circuit resistor and the capacitor are connected in series.
The time constant of an RL circuit is defined as the time taken by the current to reach its maximum value that had maintained during its initial rate of rise.
The time constant of a series RL circuit equal to the ratio of value of inductor to the value of resistance.
Since, reactance is Xl = 2*pi*f*L, for low frequency(f) the inductive reactance will be low and
so there will not be any voltage drop across inductor, the output voltage across resistor
will be same as input voltage. Therefore series RL circuit will act as inductor circuit.
Transfer function of RL circuit is V0(s)/Vi(s) = 1/(S+1).
%By taking inverse laplace we will get
%V0 = Vi * exp(-t). When we apply impulse input (Vi(t)), we will get a decaying exp curve. However,
%if we apply step input, the value of ouput will reach at a saturation point.


\documentclass[10pt]{paper}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{xcolor}
\usepackage{booktabs}
\usepackage[hyphens]{url}

\newcommand{\po}{\mathsf{po}}
\newcommand{\poloc}{\mathsf{po\text{-}loc}}
\newcommand{\co}{\mathsf{co}}
\newcommand{\rf}{\mathsf{rf}}
\newcommand{\fr}{\mathsf{fr}}
\newcommand{\com}{\mathsf{com}}
\newcommand{\loc}{\mathsf{loc}}
\newcommand{\axiom}[1]{\textsc{#1}}
\newcommand{\litmus}[1]{\texttt{#1}}
\newcommand{\state}[1]{\texttt{#1}}

\newcommand{\match}{\;\mathsf{match}\;}
\newcommand{\isEndFor}{\;\mathsf{isEndFor}\;}

\newcommand{\JAComment}[1]{\textcolor{red}{(Jade: #1)}}
\newcommand{\NCComment}[1]{\textcolor{green}{(Nathan: #1)}}
\newcommand{\DSComment}[1]{\textcolor{blue}{(Daryl: #1)}}

\begin{document}
\title{Linking Architecture and Microarchitecture with Cats and Dogs}
\author{Jade Alglave\and Nathan Chong\and Daryl Stewart}
\maketitle

\begin{abstract}
We report on work to address the problem of formally linking a specification to an implementation in hardware design.
%
More specifically, the memory consistency model (architecture) with respect to the design of a L1 memory system (microarchitecture).
\end{abstract}

Determining if a design meets or is within the envelope of a specification is a key problem in design integrity.
%
This is a particularly difficult problem if the specification is large or subtle, as in the case of the ARM architecture (which is both).
%
Common and valuable approaches to this problem are code reviews, testing and block-level assertions.
%
In this report, we look at linking the work of Alglave et al.~\cite{cats}, for reasoning about memory models (i.e., architecture), with the work of Stewart et al.~\cite{dogs}, for defining and checking end-to-end properties of a processor memory system (i.e., microarchitecture).

\section{Background}

\subsection{Cats}

Cats is a generic framework for defining and exploring (axiomatic\footnote{Operational semantics is an alternative approach}) formalisations of memory models.
%
In this work a memory model is a set of axioms, expressed as relations and orders over the set of memory events, that can be `run', using the herd tool, over a given litmus test to automatically explore possible execution behaviours.

A litmus test is a multiprocessor program giving a sequence of instructions (e.g., \texttt{LDR} and \texttt{STR}) per thread.
%
A litmus test gives rise to a set of behaviours or \emph{candidate executions} (containing events such as reads and writes) that can be checked with respect to the memory model axioms.
%
If any axiom is violated then the execution is forbidden (i.e, an implementation must never exhibit this behaviour); otherwise, the execution is allowed by the architecture (i.e., an implementation may exhibit this behaviour).

For example, the ARM model in herd defines the axiom
  \axiom{SC-PER-LOC} (sequential consistency per location)
as
  $\texttt{acyclic} (\poloc \cup \com)$,
where
  $\poloc = \po \cap \loc$
and
  $\com = \co \cup \rf \cup \fr$.

The relation $\po$ (program order) is defined between events arising from instructions executing on the same thread;
%
$\co$ (coherence order) is a total order over writes to the same location;
%
$\rf$ (reads-from) is a relation that connects a write to a read of the same location (where the value read is the same as the value written);
%
and, finally, the $\fr$ (from-reads) relation is induced by $\co$ and $\rf$: if a write $w$ is $\co$-before a write $w'$ and $r$ reads-from $w$ then $r$ from-reads $w'$.

Now, consider the following litmus test, known as \litmus{coRR}:
%
\begin{verbatim}
  // [x] = 0 initially
  // P0         |   P1
  STR [x], #1   |   LDR r0, [x]
                |   LDR r1, [x]
\end{verbatim}

The outcome $\texttt{r0} = 1$ with $\texttt{r1} = 0$ is forbidden due to the \axiom{SC-PER-LOC} axiom.
%
To see this, consider the candidate execution with this result, which will consist of the following memory events:
%
\begin{verbatim}
  a: W x 1        b: R x 1
                  c: R x 0
\end{verbatim}
%
Then there is a cycle $a \xrightarrow{\rf} b \xrightarrow{\poloc} c \xrightarrow{\fr} a$, which violates the \texttt{SC-PER-LOC} axiom.

\subsection{Dogs}

DOGRel is a language for expressing implementation-independent end-to-end properties.
%
DOGs are properties given as multiple finite state machines (FSMs) (at least one FSM per interface) where each FSM describes an observable behaviour.
%
A DOG can be seen as a graph structure where vertices are states and edges are \emph{event expressions} that must be observed to trigger a state change.
%
A subtlety about DOG transitions is that events that do not match the outgoing edge are ignored (rather than transitioning to a non-accepting state).
%
In this work, we concentrate on DOGs that express properties over the interface of the L1MS (L1 memory system):
\begin{itemize}
%
\item The \emph{load-store domain} between the DPU (datapath unit) and LSU (load-store unit)
%
\item The \emph{read-write domain} between the BIU (bus interface unit) and wider memory system
%
\end{itemize}

Figure~\ref{fig:load-load-dog} gives a DOG that describes behaviour of two successive (in \emph{clock-time order}) loads to the same address.
%
A rough interpretation of this DOG is that loads to the same location in clock-time order are preserved by the L1MS by \emph{star order}, which is a partial order over when an event takes effect\footnote{Our current thinking that this relates to what is called `externally-visible order' or `globally-observed order' in the ARMARM}.
%
The path
  $\state{L0} \rightarrow \state{L1}
              \rightarrow \state{L2}
              \rightarrow \state{L3}
              \rightarrow \state{L4}$
gives a trace for the load-store domain with two loads to the same location \texttt{A0} with different data\footnote{There is an implicit assumption that $\texttt{D0} \neq \texttt{D1}$} \texttt{D0} and \texttt{D1} in clock-time order (implicitly given by the order that the events are observed).
%
The path
  $\state{M0} \rightarrow \state{M1}
              \rightarrow \state{M2}
              \rightarrow \state{M4}$
describes a trace for the read-write domain with two reads (which match the two loads) where the first read (\texttt{D0}) is star ordered (explicitly given by the star notation attached to each event) before the second read (\texttt{D1}).
%
The implication $\state{L4} \mapsto \state{M4}$ ensures that if the behaviour of the load-store domain is observed then the behaviour of the read-write domain \emph{must} have been observed.

\begin{figure}[t]
\centering
  \includegraphics[width=.75\textwidth]{figures/loadload.pdf}
\caption{Load Load DOG (source: difts14 talk)}
\label{fig:load-load-dog}
\end{figure}

\section{DOGs as First Order Logic Formulas}

A DOG implicitly structures events using different orders such as clock-time order and star order.
%
We can make this explicit by consider a first-order formula that represents the semantics of the DOG.

Give a DOG we consider the set of implications specified by the graph.
%
Each implication is transformed into a formula and the first-order formula of the whole DOG is the \emph{conjunction} of these formulas.

We assume that each implication is of the form $(S_1 \wedge S_2 \wedge \cdots) \mapsto (T_1 \vee T_2 \vee \cdots)$.
%
By substituting each state with an equivalent formula we transform the implication into a first-order implication.
%
We call each $S_i$ a \emph{triggering state} and each $T_i$ an \emph{accepting state}.
%
For each (triggering or accepting) state $U$ we consider all paths $P_0, P_1, \dots$ from the initial state of the FSM (containing $U$) to $U$.
%
Each path $P_i$ is transformed into a path-formula $f_i$ so that the formula for $U$ is the \emph{disjunction} of these path-formulas (i.e., $\bigvee f_i$).

In the following we assume the path is a sequence of states $U_0, \dots, U_{final}$.
%
Consider the expressions that transition the states of the path.
%
If the expression contains events then we first generate a fresh (event) variable that corresponds to these events.
%
This is the concrete event from a trace that matches an event of the expression, allowing the edge to transition.
%
For example, the expression \texttt{selectBE \&\& BE \&\& start = 1} between states \state{L0} and \state{L1} in Figure~\ref{fig:load-load-dog} uses the events $E = \{\textsf{reset}, \textsf{dsb@s}, \textsf{ST(A0,!D\#)@s}\}$.
%
This is denoted $(e \match E)$ and we call $e$ the match variable.
%
\subsection{Clock-time order}
%
For each pair of edges $U \xrightarrow{expr} U' \xrightarrow{expr'} U''$ in the path where $e$ and $e'$ are the corresponding match variables for the expressions $expr$ and $expr'$ we generate the following \emph{path constraint}:
%
\[
e \leq_{ct} e'
\]
%
For example, in Figure~\ref{fig:load-load-dog}, the path $\state{L0} \xrightarrow{expr_0} \state{L1} \xrightarrow{expr_1} \state{L2} \xrightarrow{expr_2} \state{L3} \xrightarrow{expr_3} \state{L4}$ yields the path constraint $e_0 \leq_{ct} e_1 \leq_{ct} e_2 \leq_{ct} e_3$ where $e_i$ is the corresponding match variable for $expr_i$.

\subsection{Vacuous escapes}
%
In Figure~\ref{fig:load-load-dog}, the transitions to state \state{LT} are \emph{vacuous escapes} because if any of these transitions is taken then the triggering state \state{L4} is unreachable.
%
We encode this as part of the \emph{path constraint} by generating a fresh match variable for each vacuous escape transition.
%
For example, if $x_0$ is the corresponding match variable for the vacuous escape transition $\state{L1} \rightarrow \state{LT}$ then we encode:
\[
\neg (\exists x_0 : e_0 \leq_{ct} x_0 \leq_{ct} e_1)
\]
%
and similarly, for the other escape transitions.

\subsection{Star order}
%
If the path contains a pair of expressions containing \emph{star} events then we generate constraints using Table~\ref{table:star} where $e$ (respectively $f$) is the match variable corresponding to the event expression \texttt{a} (respectively \texttt{b}).
%
Note that it is acceptable for clock-time and star-order to `contradict'.
%
For example, $f \leq_{ct} e \wedge e <_{so} f$.

\begin{table}[h]
\caption{Star ordering constraints}
\label{table:star}
\begin{tabular}{lll}
\toprule
\textbf{Event expression pair} & \textbf{Interpretation}   & \textbf{Formula} \\
\midrule
\texttt{a}, \texttt{b*+}       & pure alpha                & $e <_{so} f$ \\
\texttt{a}, \texttt{b*-}       & pure beta                 & $f <_{so} e$ \\
\texttt{a}, \texttt{b*!+}      & not alpha (beta or gamma) & $e =_{so} f \vee f <_{so} e$ \\
\texttt{a}, \texttt{b*!-}      & not beta (alpha or gamma) & $e =_{so} f \vee e <_{so} f$ \\
\bottomrule
\end{tabular}
\end{table}

\subsection{Complete events}
%
If a path contains a \texttt{COMPLETE} event then we must encode the following condition.
%
Let $e$ be the match variable corresponding to the complete event and $\mathsf{starts}$ be the set of match variables corresponding to the event expressions along the path up to the complete event.
%
Then $\mathsf{isComplete}(e, \mathsf{starts})$ is defined as
%
\begin{align*}
\exists x \in \mathsf{starts} & : e\; \isEndFor\; x\; \wedge \\
\forall s \in \mathsf{starts} & : \exists u \in Ev : u\; \isEndFor\; s \wedge u \leq_{ct} e \\
\end{align*}
%
where $e \isEndFor s$ means that the event $e$ is the \texttt{@e} corresponding to the \texttt{@s} of the start event $s$ (i.e., all event actuals match except for the start/end annotation).

We now consider if the path is taken from the load-store or read-write domain.
%
\subsection{Load-Store Domain Path-Formulas}
%
A path from the load-store domain results in a formula with:
%
\begin{itemize}
\item Fresh match variables for each event expression
%
\item A clock-time path constraint over these match variables
%
\item A set of vacuous escape conjuncts for each vacuous escape path
\end{itemize}

\subsection{Read-Write Domain Path-Formulas}
%
A path from the read-write domain is similar, except we consider two further problems:
%
\begin{itemize}
%
\item Star constraints as given above.
%
\item Preload paths: if the path contains a preload then we do not generate a constraint (i.e., the resulting formula is $\mathsf{true}$); this is because the preload means that we cannot make a one-to-one correspondence between the loads and reads of the DOG.
%
In this case, we rely on the data independence property.
%
\item If the last state of the path is an accepting state and there are transitions out of this state to non-accepting states then we recursively generate the formula for the path for each non-accepting state and conjunct its \emph{negation}.
%
Hence, we encode the property of reaching the accepting state \emph{and proceeding no further}.
%
\end{itemize}

\subsection{Syncs}
%
Finally, we consider the constraint generated from two paths where one contains a synchronization assignment (e.g., \texttt{start = 1} in the transition from \state{L0} to \state{L1}) and the other contains a synchronization test (e.g., \texttt{start == 1} from \state{M0} to \state{M1} from Figure~\ref{fig:load-load-dog}).
%
In this case we generate a clock-time constraint between the synchronization assignment and test since the latter cannot transition until after the former.
%
For example, if $e$ (respectively, $e'$) is the match variable corresponding to the synchronization assignment (respectively, the synchronization test) then the generated constraint is $e\leq_{ct} e'$.\NCComment{Should be strict clock-time?}
%
One subtlety is the scope of the match variable corresponding to the synchronization assignment.
%
It is necessary to hoist this variable to the top-level of the formula so that its scope is correct.

\subsection{Implementation}

As part of this work we have developed an Ocaml implementation that generates a formula from a DOG input.
%
See \url{https://github.com/nchong/dogir}.
%
For example, the (annotated) output of the tool given the DOG in Figure~\ref{fig:load-load-dog} is:
%
\begin{figure}[h]
\begin{verbatim}
exists f0 sync :
  exists e0 e1 e2 e3 :
    (* match structure *)
    e0 matches { DSB()@s, ST(A0, !D#)@s } /\
    e1 matches { LD(A0, D0)@s } /\
    e2 matches { LD(A0, D1)@s} /\
    e3 matches { LD(A0, D1)@e} /\
    (* path constraint *)
    e0 <=ct e1 <=ct e2 <=ct e3 /\
    (* vacuous escapes *)
    not (exists x0 :
      x0 matches { ST(A0, D#)@s, DSB()@s, ST(A0, !D#)@s } /\
      e0 <=ct x0 <=ct e1 /\
    not (exists x1 :
      x1 matches { ST(A0, D#)@s, DSB()@s, ST(A0, !D#)@s, LD(A0, !D#)@s } /\
      e1 <=ct x1 <=ct e2 /\
    (* synchronization assignment *)
    e0 = sync
  =>
  exists f1 :
    (* match structure *)
    f0 matches { RD(A0, D0)@s } /\
    f1 matches { RD(A0, D1)@s ) /\
    (* path constraint *)
    f0 <=ct f1 /\
    (* star constraint *)
    f0 <so f1 /\
    (* synchronization test *)
    sync <=ct f0
\end{verbatim}
\caption{Formula of Load-Load DOG}
\label{fig:load-load-formula}
\end{figure}

\subsection{Daryl's alternative transformation}

An alternative transformation is given here.
%
\NCComment{Copy and pasted. Should be reviewed.}
%
\begin{verbatim}
An Execution = (Ev, po, rf, co)

/* The guarding observable events of a transition */
guardsOf(s0, s1) = { e | s0->s1: ...∨ e ∨... }

/* Selecting an appropriate event from an execution which can satisfy a transition's guarding observable event */

(complete accepts x from Ev for observed stars)
iff /* x is an @e event for some  observed @s event */
    ∃w∈observed. x isEndFor w
    /* and all other observed *s events have completed */
    ∧ ∀v∈observed. isStartEvent(v) → ∃u∈Ev. u isEndFor v ∧ ct≤(u, x)

(e accepts x from Ev for observed stars)
iff /* e is an observable event (i.e. event pattern) which would be satisfied by an occurrence of event x */
    x isAcceptableInstanceOfObservableEvent e
    ∧ ( isStartEvent(x) ∨ ∃y∈observed. x isEndFor y)

/* accepting a lone star event - the empty condition could be removed for something different */
(e*   accepts x from Ev for observed stars) iff (e accepts x from Ev for observed stars) ∧ empty(stars)

/* cases for star ordering observable events */
(e*+  accepts x from Ev for observed stars) iff (e accepts x from Ev for observed stars) ∧ ∀s∈stars. s so< x
(e*!- accepts x from Ev for observed stars) iff (e accepts x from Ev for observed stars) ∧ ∀s∈stars. s so≤ x
(e*!+ accepts x from Ev for observed stars) iff (e accepts x from Ev for observed stars) ∧ ∀s∈stars. s so≥ x
(e*-  accepts x from Ev for observed stars) iff (e accepts x from Ev for observed stars) ∧ ∀s∈stars. s so> x
(E accept x from Ev for observed stars) iff ∃e∈E. (e accepts x from Ev for observed stars )

/* how to notice a loneStar transition
 * assumes that for a well-formed DOG, no two observable events on a single transition can both accept a particular event x
 */
(E sawStar x from Ev for observed stars) iff ∃e∈E. (e accepts x from Ev for observed stars ) ∧ isLoneStar(e)

/* An event can cause a transition... */
((s,s') transitionsOn x from Ev for observed stars ) iff guardsOf (s,s') accept x from Ev for observed stars)

/* how to notice a loneStar transition */
((s,s') transitionsOnStar x from Ev for observed stars ) iff guardsOf (s,s') sawStar x from Ev for observed stars)

/* Creating a FOX from a path */

/* Helper for creating a FOX from a path */
  FOXpaths(s0::s1::path, observed as xin::_, stars) = /* recursive case for creation of FOX from path */
    /* There is an event which can be used to transition from s0 to s1.
     * Herein we associate this with identifier x01, but note that name is actually a fresh name not found in observed
     */
  ∃x01∈Ev\observed.
  ( ((s0, s1) transitionsOn x01 from Ev for observed stars) ∧ ct≤(xin, x01)
    ∧   /* There is no other event which could have caused the transition to s1 earlier than x01 */
        ( ∄x00∈Ev\observed. ((s0, s1) transitionsOn x00 from Ev for observed stars) ∧ ct≤(xin, x00) ∧ ct<(x00, x01) )
    ∧
        ( ∀s∈successors(s0)\s1.
            /* No higher priority state could have been reached from s0 instead */
            ( priority(s)>priority(s1)
                → ∄x0s∈Ev\observed. (s0, s) transitionsOn x0s from Ev for observed stars
                                  ∧ ct≤(xin, x0s) ∧ ct≤(x0s, x01)
            )
        ∧   /* no same priority state could have been reached from s0 instead */
            ( priority(s)=priority(s1)
                → ∄x0s∈Ev\observed. (s0, s) transitionsOn x0s from Ev for observed stars
                                  ∧ ct≤(xin, x0s) ∧ ct<(x0s, x01)
            )
        )
    ∧ /* and the rest of the path can be traversed
       * Note that newStars is actually a fresh name
       */
        ∃newStars⊆Ev. stars⊆newStars /* REVISIT - choice operator usable in coq?? */
                    ∧ x01∈newStars⟺((s0, s1) sawStar x01 from Ev for observed stars)
                    ∧ FOXpaths(s1::path, x01::observed, newStars)
  )

  FOXpaths(s0::[], xin, observed, stars) = /* base case for creation of FOX from path */
    /* There is no event leading from the last state of the path */
  ∀s∈successors(s0). ∄xout∈Ev\observed. (s0, s) transitionsOn xout from Ev for observed stars ∧ ct≤(xin, xout)

/* Main entrypoint to create a FOX from a path */
FOXpath(path) = FOXpaths(path, [time0], []); /* The event causing entry to the initial state is t0 */
\end{verbatim}

\subsection{Alternative: Deep embedding in Coq/HOL}

Lastly, we have considered, but not implemented a deep embedding of a theory of DOGs in Coq/HOL.
%
The advantage of this approach is that higher-order logic can be brought to bear on the problem.
%
\NCComment{The main disadvantage is that we have to use Coq ;-)}

\section{Hand-written \litmus{coRR} proof}
%
Using formula generated from load-load DOG we can reason about the \litmus{coRR} test.

\section{Hand-written \litmus{coRW1} proof}

\begin{verbatim}
  // [x] = 0 initially
  // P0
  LDR r0, [x]
  STR [x], #1
\end{verbatim}

The outcome $\texttt{r0} = 1$ is forbidden due to the \axiom{SC-PER-LOC} axiom.

\subsection{Load-Store DOG formula}

Given in Figure~\ref{fig:loadstore-formula}.

\begin{figure}[p]
\begin{verbatim}
exists sync :
  exists e0 e1 e2 e3 :
    (* match structure *)
    e0 match BE /\
    e1 match { LD(A0,D0)@s } /\
    e2 match { ST(A0,D1)@s } /\
    isComplete(e3, { e0, e1, e2 }) /\
    e0 = sync /\
    (* ct path constraint *)
    e0 <=ct e1 <=ct e2 <=ct e3 /\
    (* vacuous escapes *)
    not (exists x0 : x0 match { ST(A0,D1)@s } /\
                     x0 <=ct e0) /\
    not (exists x1 : x1 match { ST(A0,DX)@s } U BE /\
                     e0 <=ct x1 <=ct e1) /\
    not (exists x2 : x2 match { LD(A0,DX)@s, ST(A0,!D1)@s } U BE /\
                     e1 <=ct x2 <=ct e2)

  =>

  (* path ABD *)
  exists f0 :
    f0 match { RD(A0,D0)@s } /\
    sync <=ct f0 /\
    (* no step to bad state E *)
    not (exists y0 : y0 match { WR(A0,D1)@s } /\
                     f0 <=ct y0 /\
                     (f0 =so y0 \/ y0 <so f0))

  \/

  (* path ABCF *)
  exists f1 f2 :
    f1 match { WR(A0,D1)@s } /\
    f2 match { RD(A0,D0)@s } /\
    sync <=ct f1 <=ct f2 /\
    f2 <so f1
\end{verbatim}
\caption{Simplified formula for Uniproc Load-Store DOG}
\label{fig:loadstore-formula}
\end{figure}

\section{Related Work}

PipeCheck and ARMOR work by Princeton.

\bibliographystyle{alpha}
\bibliography{references}

\end{document}


\section{Data Structures}

\subsection{Compact Sparse Merkle Tries}

In order to allow for the desired set of diverse verifiable data
structures that are necessary in a blockchain system, we elected to
select a general purpose tool that could fill most roles in our system
without unacceptably sacrificing performance.
Although more optimal structures may exist for some applications,
forcing developers to build many complex forms of verifiable data
structures complicates initial engineering and future ports to other languages.
Thus, a single structure was selected that could be used in all current cases.
This solution is based on a modified form of the open source Aergo State Trie.
This is a Compact Sparse Merkle Trie that allows for compact proofs of
inclusion/exclusion, stores a leaf at the optimal sub height, and
operates as a base sixteen trie while actually being a true binary trie
implementation.
This polymorphic nature is derived from the manner in which the trie is built.
The length of a generalized encoded proof of inclusion/exclusion
requires $O(\log n)$ merkle proof keys, with an additional overhead of 64
bytes to communicate the key and value of the proof.
The proof also requires $O(\log n)$ bits to communicate the positioning of
the merkle proof keys in the trie.
Thus, the total size is effectively $O(\log n)$.

The trie itself is a binary trie where all nodes are stored in height
four sub tries.
Each sub-trie may be loaded as a batched operation that accommodates a
compact encoding through the use of bit fields that are similar to
those used in the proofs.
Thus, for a given height four sub trie, only nonempty nodes need be
stored, and the bit field may be parsed to determine the positions
these nodes should occupy in the actual trie.
The process of updating the trie may be performed in a concurrent
manner by applying all leaf transformations first and then concurrently
calculating the hashes with a blocking operation on each height four
sub-trie at the leaf levels.
Thus, each sub-sub-trie may be calculated before the root of the
sub-trie itself is calculated, and this concurrency may be arbitrarily
extended across the trie.

\subsection{State Trie}

The State Trie is a CSMT that includes two types of objects and has the
root value of the trie updated and stored in every block.
The first object is the hash of a UTXO stored in the location of its UTXOID.
The second type of object is the hash of a deposit nonce that is stored
in the location that is equal to the deposit nonce.
The first objects, the UTXO type objects, allow a compact proof of
inclusion or exclusion to be formed for the UTXO at a given block height.
This is useful for proofs of datastores, proofs to light clients, and
allows fast syncing of the chain from a known good block, such as a snapshot.
When a UTXO is created, it is added to the trie.
When the UTXO is consumed, it is removed from the trie.
Deposits work in an inverse manner.
When a deposit is spent, it is added to the trie and shall remain there
in perpetuity.
This addition tracks the spending of deposits.

\subsection{BlockHash Trie}

The BlockHash Trie is a CSMT that stores the block hash of each block
mined at a leaf value where the key is the zero padded block number.
This value is included in every block, where the value in each block
references the root of the trie after inclusion of the previous block.
The structure is maintained in order to allow light clients to query
blocks on a singular basis with proof of inclusion for each queried block.
The proofs themselves are intended to start from a snapshot block and
allow previous blocks from the snapshot to be proven as valid blocks
from the entire chain.
This system also has the benefit of allowing proofs of sidechain state
to be formed much more readily in the Ethereum blockchain due to the
ability to easily prove a block header as having a specific block hash
as a fixed complexity procedure.
Alternative systems must either track every block hash in the Ethereum
blockchain, or large batches of contiguous blocks must be injected in
order to prove state at an arbitrary block.
We overcome these limitations through the proofs of inclusion and
exclusion offered by the CSMT.


\chapter{Theoretical Foundations}\label{ch:theoretical-foundations}

This chapter explains the theoretical foundations that are necessary to understand the neural network that was developed in this work.
It first describes how a machine learns from data following with an introduction of a specific neural network architecture, the Transformer.
Finally, a language model based on Transformers is introduced.

% ===============
% LEARNING FROM DATA
% ===============

\section{Learning from Data}

\subsection{Labeled and Unlabeled Data}

When learning from data, it is usually either labeled or unlabeled.
An example of labeled data can be images with a caption, where the image itself is the data and the caption is the label.
For abstractive summarization tasks, data is usually labeled with the source text being the data and the summary of this text being the label.
Unlabeled data can be any data, such as plain images or texts.
Unlike labeled data, the amount of unlabeled data is usually much higher as labeling data is an expensive manual process, while unlabeled data is usually produced as a by-product.
A popular example of unlabeled text data is Wikipedia, which contains over a billion words in English alone \cite{wikipediaStats}.

%TODO Source?

\subsection{Neural Networks}

Artificial neural networks are currently a popular and common way to learn from data due to the availability of better hardware and an increasing amount of training data.
A typical neural network consists of neurons that are inspired by real brain neurons and connections between these neurons.
Each neuron can be seen as a function that takes the values of all its predecessor nodes as input and outputs a value that, by itself, is used as the input for its successor nodes.
To compute a node's output, the value $x \in X$ of every predecessor node is multiplied with a weight $w \in W$.
Then, a constant bias $b$ is added to the sum of these multiplications, and finally an activation function $\Phi$ is applied to this sum as shown in the following equation:
\[
	y = \Phi(\sum_{i=0}^{d}x_iw_i + b)
\]
with $d$ being the number of predecessor nodes and $y$ being the node's output.
For most nodes, the activation function $\Phi$ is usually a nonlinear function that maps its input to values between $-1$ and $1$ or $0$ and $1$.
A popular example of an activation function is sigmoid, which maps negative values to values close to $0$ and positive values to values close to $1$ \cite[p.~4--13]{Aggarwal2018}.

\paragraph{Feed-Forward Networks}

One of the simplest and most common neural network architectures is the feed-forward network.
Such a network consists of multiple layers, where neurons only have predecessors in the previous layer and successors in the next layer.
The first layer with no predecessors is called the input layer as here the actual input for the neural network is used as the node's values.
The last layer with no successors is called the output layer.
The values of the nodes in the output layer are used as the output of the whole network.
Every layer in between these two layers is called a hidden layer.
While the nodes in the hidden layer usually have a nonlinear activation function, nodes in the output layer sometimes have a linear activation function like ReLU if the output is an absolute value.
Such a network is shown in \cref{fig:neural-network} \cite[p.~17--20]{Aggarwal2018}.

\begin{figure}[h]
\centering
\includegraphics[width=0.45\paperwidth]{figures/neural-network}
\caption{Simple feed-forward neural network with two hidden layers}
\label{fig:neural-network}
\end{figure}

\paragraph{Backpropagation}

When training a network, the goal is to adjust all the weights and biases in a way that the error of the neural network is minimized.
This error is measured with a loss function that computes how different the output of the network for the training data is compared to the desired output (the labels).
Backpropagation is an algorithm that computes how the weights and biases should be adjusted.
The backpropagation algorithm consists of two phases: In the forward phase, a portion (batch) of the training data is fed into the network, and the loss is computed.
In the backward phase, the gradients of the loss function are computed, and then these gradients are used to update the weights of the network.
This processes is repeated multiple times \cite[p.~21--24]{Aggarwal2018}.

% TODO Maybe add an "Overfitting" paragraph

\subsection{Automatic Evaluation}\label{ssec:automatic-evaluation}

Evaluating summaries is essential for multiple reasons (\eg, to monitor progress during training or to compare own results to results from other researchers).
While human evaluation seems like an obvious approach, it is usually not feasible because it is slow and expensive, and results from different people are not always comparable.
Thus, the more common approach is automatic evaluation.
To automatically evaluate summaries, Recall-Oriented Understudy for Gisting Evaluation (ROUGE) \cite{lin-2004-rouge} is widely adapted as it tends to correlate well with human judgments.
In addition, ROUGE supports multiple evaluation metrics.
To report the results achieved in this thesis, the following three metrics are used:
\begin{itemize}
\item \textbf{ROUGE-1}, which calculates the overlap of single words (unigrams) between the candidate summary and the reference summary
\item \textbf{ROUGE-2}, which is similar to ROUGE-1, but instead of using the overlap of single words, it uses word pairs of two (bigrams)
\item \textbf{ROUGE-L}, which uses the longest common subsequence to calculate the score. A subsequence must not be confused with a substring. For the sentences "My favorite name is John Doe" and "My name is Maria Doe," the longest common subsequence would be "My name is Doe," whereas the longest substring would just be "name is."
\end{itemize}

When comparing word overlap, one must differentiate between recall and precision \cite{Ting2010}.
While recall indicates how much information of the reference summary is present in the candidate summary, precision indicates how much of the information in the candidate summary is actually present in the reference summary.
It is common to combine these two measures to a single one, called F1 measure.
It is calculated by the following formula:
\[
	F_1 = 2 \cdot \frac{Recall \cdot Precision}{Recall + Precision}
\]

% ==========
% TRANSFORMER
% ==========

\section{Transformer}\label{sec:transformer}

The Transformer is a neural network architecture for sequence learning that was published in 2017 \cite{1706.03762}.
It achieved new state-of-the-art results for many machine translation tasks while at the same time requiring significantly less training time compared to previous well-performing models.

\subsection{Motivation}\label{ssec:transformer-motivation}

In recent years, recurrent neural networks (RNNs) have been the most common way to solve sequence learning tasks.
However, one of the major downsides of RNNs is their sequential behavior.
To train an RNN, the elements of the input sequence are fed into the neural network one after another.
While the sequence elements are passed through the RNN, a hidden state gets computed at every step, which is part of the input for the next step.
This process is called encoding.
Afterward, the last hidden state of the encoding process is used as an input for the first step of the decoding process.
Because of this, it is not possible to parallelize the training within a training example as the hidden state of the previous step is needed to calculate the output of the current step \cite[p.~2]{1706.03762}.
A simplified RNN for sequence-to-sequence tasks like machine translation is shown in \cref{fig:rnn-visualization}.

\begin{figure}[h]
\centering
\includegraphics{figures/rnn-visualization}
\caption[Simplified visualization of a sequence-to-sequence RNN network for machine translation]{Simplified visualization of a sequence-to-sequence RNN network for machine translation. The input sequence is fed into the encoder with one word for each step. The final hidden state, which is computed in the last encoding step, is used as input for the first decoding step. The initial hidden state, $h_{start}$, can be initialized as a vector of zeros or learned during training.}
\label{fig:rnn-visualization}
\end{figure}

Another problem with RNNs is the long distance, that information must "travel" before it is used. % TODO Illustrate this long "travel" distance
Information that was encoded into the hidden state must pass through multiple encoding and decoding steps, before it is finally used in one of the decoding steps.
Besides having the hidden state as an information bottleneck, this also makes it very hard for the network to learn long range dependencies because of the problem of vanishing or exploding gradients \cite{Hochreiter01gradientflow}. % TODO Maybe briefly explain the problem
Modern variants of RNNs, like the long short-term memory (LSTM) architecture \cite{Hochreiter1997}, try to mitigate these effects but are not able to fully eliminate them.
A fairly recent solution to this problem is the attention mechanism that was first proposed in 2014 \cite{1409.0473}.
It allows the network in the decoding stage to retrieve information from any hidden state that was computed during the encoding stage.
This eliminates the issue of learning long range dependencies.

The Transformer network takes advantage of the attention mechanism while avoiding the recurrence of RNNs to achieve faster training and allow for more parallelization.

\subsection{Model Architecture}\label{ssec:transformer-model-architecture}

Just like sequence-to-sequence RNNs, the Transformer consists of an encoder and a decoder that both have multiple stacks of layers.

An encoder layer consists of two sublayers.
The first sublayer is a multi-head self-attention mechanism, which is described later in this section.
The second sublayer is a position-wise\footnote{The identical network is applied to each position separately \cite[p.~5]{1706.03762}.} fully connected feed-forward network.
Both sublayers have residual connections \cite{1512.03385} around them that are followed by layer normalization \cite{1607.06450}.
The outputs of the layers of the Transformer base model have $d_{model}=512$ dimensions and $d_{model}=1024$ for the big model \cite[p.~9]{1706.03762}.

An encoder layer has a similar structure but with one additional sublayer and a slight modification of the multi-head self-attention sublayer.
The new sublayer attends to the output of the encoder.
The multi-head self-attention sublayer is masked to prevent it from attending to future words \cite[p.~3]{1706.03762}.

In the original Transformer, $N = 6$ stacked encoder and decoder layers are used for both the base and the big model.
The complete architecture of the Transformer is visualized in \cref{fig:transformer-model}.

\begin{figure}[h]
\centering
\includegraphics{figures/transformer-model}
\caption[Model architecture of a Transformer]{Model architecture of a Transformer \cite[p.~3]{1706.03762}}
\label{fig:transformer-model}
\end{figure}

The inputs are simple learned embedding vectors of dimension $d_{model}$ that are concatenated with a fixed positional embedding \cite[p.~5--6]{1706.03762}.
These fixed input embeddings encode the position of the sequence elements as this information is otherwise no longer accessible to the network because of the missing recurrence.
The positional encoding is calculated with sine and cosine functions of different frequencies:
\begin{align*}
	PE_{(pos,2i)} & = sin(pos/10000^{2i/d_{model}}) \\
	PE_{(pos,2i+1)} & = cos(pos/10000^{2i/d_{model}})
\end{align*}

This encoding allows the network to obtain information about the absolute and relative position of each element.
\Cref{fig:positional-encoding-sine-cosine} illustrates the embedding for some dimensions.
It is clearly visible how each position has a different positional embedding.

For the outputs, a learned linear transformation and a softmax function are used to compute next-token probabilities \cite[p.~5]{1706.03762}.

\begin{figure}[h]
\centering
\includegraphics[width=0.7\paperwidth]{figures/positional-encoding-sine-cosine}
\caption[Visualization of positional encoding with sine and cosine]{Visualization of positional encoding with sine and cosine \cite{annotated.transformer}}
\label{fig:positional-encoding-sine-cosine}
\end{figure}

\paragraph{Attention}

The Transformer attention consists of query, key, and value vectors.
The query vectors are used to select values by their corresponding keys.
The Transformer implements this as a scaled dot-product attention, wherein queries and keys have the dimension $d_k$ and the values the dimension $d_v$.
To calculate the attention, the following formula is used:
\[
	\textrm{Attention}(Q,K,V) = \textrm{softmax}(\dfrac{QK^T}{\sqrt{d_k}})V
\]
where $Q$, $K$, and $V$ are matrices that contain multiple sets of queries, keys, and values \cite[p.~3--4]{1706.03762}.
Masking for the decoder layer is easily performed by setting the input values of the softmax function to $-\infty$ if they are not allowed.

Unlike normal attention, the Transformer uses multi-head attention.
This means that instead of having a single attention function, the queries, keys, and values are linearly projected with learned linear projections \cite[p.~4--5]{1706.03762}.
After performing the attention function on the projected matrices, they are concatenated and projected again.
The dimension of each learned linear projection is the model's dimension $d_{model}$ divided by the number of attention layers $h$.
\Cref{fig:scaled-dot-product-multihead-attention} shows both the scaled dot-product attention (left) and the multi-head attention (right).

\begin{figure}[h]
\centering
\includegraphics[width=0.7\paperwidth]{figures/scaled-dot-product-multihead-attention}
\caption[Visualization of scaled dot-product attention and multi-mead attention]{Visualization of scaled dot-product attention and multi-head attention \cite[p.~4]{1706.03762}}
\label{fig:scaled-dot-product-multihead-attention}
\end{figure}

\subsection{Advantages}

As described in \cref{ssec:transformer-motivation}, the Transformer allows for significantly more parallelization than RNNs because of the absence of sequential behavior within one training example.
This yields much faster and less expensive training compared to previous state-of-the-art models for machine translation.
The big Transformer model is trained for 3.5 days on 8 P100 GPUs and the base Transformer model for just 12 hours \cite[p.~7]{1706.03762}.
In comparison, the first RNN network to utilize attention took between 109 and 252 hours for training \cite[p.~14]{1409.0473}.

\subsection{Disadvantages}

Transformer networks tend to get very large and require a large amount of memory, especially for longer sequence lengths as the self-attention connects every position with every other position in the sequence.
This results in quadratic complexity for the sequence length.
As a solution, the authors of the Transformer suggest limiting the self-attention to the neighbors of each position \cite[p.~6--7]{1706.03762}.
The more recent Reformer \cite{kitaev2020reformer} solves this issue by using locality-sensitive hashing to reduce the complexity from $O(L^2)$ to $O(L)$.

% ====
% BERT
% ====

\section{BERT}\label{sec:bert}

Bidirectional Encoder Representations from Transformers (BERT) is a general-purpose language model that was released by Google AI researchers in mid-2018.
It broke several records for natural language processing benchmarks \cite[p.~5--7]{devlin2018bert}, such as  SQuAD v1.1 \cite{rajpurkar-etal-2016-squad} and GLUE \cite{1804.07461}.

\subsection{Motivation}

It has been shown that results for many natural language processing tasks can be improved with the help of language models.
The most famous among them are OpenAI's GPT-2 \cite{radford2019language} and ELMo \cite{1802.05365}.
The benefit of language models is that most of them can be trained on unlabeled data like Wikipedia articles or large book corpora.
This allows them to learn from a huge amount of data as there is no need for time-consuming labeling of the training data and any texts can be used.
By pretraining them on normal texts, the networks can get a general sense of natural language, which can then be used for more specific tasks.

What is new about BERT compared to other models like GPT-2 is that it is the first deeply bidirectional language model that takes advantage of the whole context around a word.
OpenAI's GPT-2 uses a left-to-right Transformer decoder \cite[p.~4]{radford2019language}, which only allows it to "look" at the words to the left when generating a representation for an input token.
For example, with a sentence that starts with "The lighter," the network can only look at these two words to generate a representation for the word "lighter."
However, the word can have completely different meanings depending on the whole sentence (\eg, "The lighter is an easy tool for making fire" compared to "The lighter shade of red looks better than the darker one").
ELMo compensates for this issue by training two LSTMs, one being a normal forward LSTM (left-to-right) and the other one being a backward LSTM (right-to-left), which is given the input sequence in reverse \cite[p.~2--3]{1802.05365}.
Afterward, it concatenates the representations of these two LSTMs to get a semi-bidirectional representation.
\Cref{fig:bert-gpt2-elmo-model-comparison} presents the different architectures.

% TODO Not sure if I keep using this or create an own one / just remove it
\begin{figure}[h]
\centering
\includegraphics[width=0.7\paperwidth]{figures/bert-gpt2-elmo-model-comparison}
\caption[Visualization of the different model architectures]{Visualization of the different model architectures \cite[p.~13]{devlin2018bert}}
\label{fig:bert-gpt2-elmo-model-comparison}
\end{figure}

\subsection{Model Architecture}

At its core, BERT is just a Transformer encoder, as described in \cref{ssec:transformer-model-architecture}.
What is innovative about BERT's architecture is mainly how it is trained and how the inputs and outputs are represented.

\paragraph{Input and Output Representations}

BERT's input is applicable to many different natural language processing tasks.
It allows the inputs of either one sentence or a pair of two sentences.\footnote{In the context of BERT, a sentence does not refer to an actual linguistic sentence but an arbitrary span of contiguous text \cite[p.~4]{devlin2018bert}.}
Texts are tokenized by using WordPiece embeddings with a vocabulary of 30,000 tokens.
WordPiece embeddings have the advantage of not needing out-of-vocabulary tokens like the more traditional word embeddings.
Every input sequence starts with a special \texttt{[CLS]} token, and every sentence ends with a special \texttt{[SEP]} token.
In addition to the token embeddings, the input consists of two more types of embeddings.
The first ones are learned segment embeddings that indicate if a token belongs to the first or the second token.
The second ones are position embeddings that help the network to understand the positions of each token in the sequence.
Unlike the original Transformer, BERT does not use fixed positional embeddings but learned ones.
Finally, these three embeddings are concatenated to form the actual input for the network as shown in \cref{fig:bert-input-representation}.

\begin{figure}[h]
\centering
\includegraphics{figures/bert-input-representation}
\caption[BERT input representation]{BERT input representation \cite[p.~5]{devlin2018bert}}
\label{fig:bert-input-representation}
\end{figure}

Depending on the task to perform, there are different ways to use the final hidden states of the BERT Transformer network.
For sentence classification tasks, the \texttt{[CLS]} token representation can be used.
This work uses the final hidden states for every single input token, as the decoder should have access to the presentation of every input token.
For the simpler extractive summarization task, which is basically a kind of sentence classification in which each sentence is either classified as \texttt{"include in summary"} or \texttt{"do not include in summary,"} it is sufficient to only use the final hidden state of the \texttt{[CLS]} token.
\cite{1903.10318} proposes a small input variation for BERT that inserts multiple \texttt{[CLS]} tokens into the input sequence to later use them for extractive summarization.

\paragraph{Pretraining Objectives}

BERT is pretrained with two different tasks \cite[p.~4--5]{devlin2018bert}.
For the first task, the network is given a sentence with 15\% of its WordPiece tokens randomly masked.
An example of masking is shown in \cref{fig:bert_masking_example}.

\begin{figure}[h]
\begin{lstlisting}[numbers=none]
Input: the man went to the [MASK1] . he bought a [MASK2] of milk.
Labels: [MASK1] = store; [MASK2] = gallon
\end{lstlisting}
\caption[Masked input example]{Masked input example \cite{bertGithub}}
\label{fig:bert_masking_example}
\end{figure}

The task of the network is to predict the masked tokens.
This task requires the network to learn to take the context of a word into consideration.

For the second task, the network is given two sentences.
The network must then predict if the second of these two sentences comes after the first one in the original text or if it is just a randomly selected sentence from any other text.
An example of next-sentence prediction is shown in \cref{fig:bert_next_sentence_example}.

\begin{figure}[h]
\begin{lstlisting}[numbers=none]
Sentence A: the man went to the store .
Sentence B: he bought a gallon of milk .
Label: IsNextSentence

Sentence A: the man went to the store .
Sentence B: penguins are flightless .
Label: NotNextSentence
\end{lstlisting}
\caption[Next-sentence prediction example]{Next-sentence prediction example \cite{bertGithub}}
\label{fig:bert_next_sentence_example}
\end{figure}

Ideally, this helps the network to understand the relationship between two sentences.
However, this task has been shown to be too simple for the network as it is easier for it to just learn if the topics of two sentences match instead of understanding the real relationship of the sentences.
To make the task harder, newer successors of BERT such as ALBERT use slightly changed training objectives, like sentence-order prediction \cite[p.~3]{1909.11942}, in which the goal is to predict which of two given sentences comes first.
This allows the network to better learn the relationship between sentences instead of just detecting matching topics.

Pretraining is usually expensive.
The training of the pretrained BERT models was performed for four days on four Cloud TPUs for BERT\textsubscript{BASE} and 16 Cloud TPUs for the larger model, BERT\textsubscript{LARGE}, with each Cloud TPU having four TPU chips \cite[p.~13]{devlin2018bert}.
However, it is only necessary to do pretraining once.

\paragraph{Fine-Tuning}

After the unsupervised pretraining, the network can be fine-tuned for a specific task.
This is done by just training a full network on the desired task for a short time.
Fine-tuning is far less expensive than pretraining and can usually be done in less than one hour on a single Cloud TPU or a few hours on a normal GPU \cite[p.~5]{devlin2018bert}.

% TODO Either summarize this chapter or lead to the next chapter


\section{Problem Statement}

One of the core challenges in applying Programming by Example to DSP (DSP-PBE) is to quantify how close our synthesized solution is to the correct solution.
However, the closeness of two audio files is a subjective measure - we do not necessarily know what features of the input/output audio pair the user is trying to emulate.
We use an audio distance measure from~\cite{SantolucitoFARM} to calculate the distance between two audio files.
This metric approximates a measure of how different two audio clips sound to the human ear in a similar style to audio fingerprinting~\cite{wang2003industrial,fingerprinting}.
To calculate the distance beteen two files, the function takes a sliding window FFT of fixed size, denoted $FFT(x,t)$ for the FFT of audio $x$ at time point $t$, over all time points $ts$.
It then plots the constellation of amplitude and frequency peaks $p$, and takes the euclidean distance between the points of the constellation.

%
\begin{align*}
\distFxn(x,y) =  \sum_{t=0}^{ts}\ \sum_{i=0}^{p} euclid\Big(\ &peak(i,FFT(x,t)), \\ &peak(i,FFT(y,t))\ \Big)
\end{align*}
%

While measuring frequency peaks requires temporally aligned audio, alignment can be achieved by using recent advances in dynamic time warping techniques for audio~\cite{carabias2015audio} and other audio alignment techniques~\cite{ArztL18}.
To contain the scope of the DSP-PBE problem, we assume that the user has provided audio that is temporally aligned.
As a technical note, this also requires that the sample rates of the audio match and the amplitudes of the examples have been normalized.
Again we assume the user handles amplitude normalization and resampling to align sample rates as necessary for simplicity.

In DSP-PBE the goal is to find a DSP filter $F$ to minimize the distance between $F$ applied to an input audio file, $i$, and an output audio file $o$.
The intuition is that if $\distFxn(F(i),o) = 0$, we have successfully mapped the input to the output, i.e. $F(i) = o$.
This property is called the \textit{identity of indiscernibles}~\cite{frechet1906quelques} - given a distance metric $d$, it must hold that $d(x,y) = 0 \Leftrightarrow x = y$.
Unfortunately however, this property does not hold for the \distFxn function that we use.
An easy counterexample to show that $\distFxn$ violates the identity of indiscernibles property is to take two audio files that have identical peaks, but differ in amplitudes at frequencies outside the peaks.
These inputs then violate the property as two audio files have distance zero as calculated by $\distFxn$, but are not identical audio files.
This is something of a pathological example, as in most cases $\distFxn$ is close enough for the purposes of DSP-PBE.
We simply remark that finding an aural distance that obeys the identity of indiscernibles property is a valuable direction for future work

\subsection{Defining the search space}

In order to formalize the search space of DSP-PBE, we define a grammar of filters, as shown in Fig.~\ref{fig:grammar}.
This grammar defines the tree-like structure of all allowable DSP programs that our synthesis technique is able to find.
The grammar states that we may construct a DSP filter program, $DSP$, by composing programs in one of two ways.
We allow sequential composition with the $\arrComp$ operator, which indicates the two component programs share input, and their output is summed in the time domain.
We also allow parallel composition with the $\parallelCompose$ operator, which indicates that the first component program pipes its output audio to the next component program.
The content at base level of a filter is a \dspnode, which is selected from the set of primitive filters available to synthesis.
This set can be expanded as needed by the user depending on the use case.
We limit the set of \dspnode here to a core set illustrating both commutative ($LFP, HPF, WN$) and noncommutative ($Reverb, PitchShift$) operators.
The app level for each node indicates amplitude scaling of the output.

\begin{figure}
\begin{flalign*}
DSP & := \\
& |\ DSP \arrComp DSP\ \qquad & \text{\color{gray} sequential composition}  \\
& |\ DSP \parallelCompose DSP\ & \text{\color{gray} parallel composition} \\
& |\ \dspnode& \\
DSP&Node :=  & \\
& |\ LPF \ [0,20k] \ [0,1] &\text{\color{gray} threshold, app}\\
& |\ HPF \ [0,20k] \ [0,1] &\text{\color{gray} threshold, app}\\
& |\ Reverb \ [0,1] \ [0,1] \ [0,1] & \text{\color{gray} room size, mix, app}\\
& |\ PitchShift\ [-1,1] \ [0,1] & \text{\color{gray} shift amount, app}\\
& |\ WhiteNoise \ [0,1] &\text{\color{gray} app}
\end{flalign*}
\caption{The grammar of DSP filters we consider includes lowpass filters (LPF), highpass filters (HPF), and can also be extended with other user supplied primitives.}
\label{fig:grammar}
\end{figure}


%!TEX root = ../thesis.tex
% ******************************* Thesis Appendix C ********************************
\ifpdf
\graphicspath{{Appendix4/figs/}}
\else
\graphicspath{{Appendix4/figs/}}
\fi

\chapter{Measuring Entropy Efficiently}
	In percolation we measure the Shannon entropy \cite{Shannon1948} to measure entropy of a system. But measuring Shannon entropy is very costly. Since we have to measure entropy for each value of $p$ (total $L^2$ iteration in site percolation) and there can be at most $2L^2$ cluster in site percolation (initial state). Therefore it would cost as $\mathcal{O} (L^2m)$ where $1 \geq m \geq 2$. Which is not good for taking $1000$s of ensemble.

	To improve the entropy measuring system we can do the following: we measure entropy initially in the traditional way (taking sum over all the clusters) and call it $H_{old}$. We then select a site and without occupying it measure the entropy of the cluster that is related to the site first and call it $H_s$. Then we occupy the site and measure the entropy of the cluster that emerges after occupying the site and call it $H_a$. Therefore at this point the entropy of the system is
	\begin{equation}
		H = H_{old} - H_s + H_a
	\end{equation}
	Then we set $H_{old} = H$ and repeat the process. This way we don't need to measure entropy for each cluster at each iteration, we only need to measure the change in entropy. Recall that $mu_i=\frac{S_i}{\sum_i S_i}$ where $S_i$ is the size of the $i$-th cluster and $\sum_i S_i = M = 2L^2$ in site percolation.

	At this point an worked out example is helpful. Say we have a lattice of length $L=8$ then there are $N=64$ sites and $M=128$ bonds \ref{fig:efficient-entropy-1}. And initially there are $M$ clusters of size $1$ (only one bond per cluster). Then the initial entropy $H_{init}$ is
	\begin{align}
		H_{init} &= -\sum_{i} \mu_i \log_2 \mu_i \nonumber\\
		  &= -128 \times \frac{1}{128} \log_2 \frac{1}{128} \nonumber\\
		  &= -\log_2 \frac{1}{128} \nonumber\\
		H_{init}  &= \log_2 128 = 7.0
	\end{align}
	Since $\mu_i = 1/128$
	When we occupy one site it will connect with four bonds to create a cluster of size $4$.
	\begin{align}
		H_{s}
		&= - \sum_{i=1}^{4} \frac{1}{128} \log_2 \frac{1}{128} \nonumber \\
		&= \frac{4}{128} \log_2(128) = 0.21875
	\end{align}
	\begin{align}
		H_{a}
		&= -\frac{4}{128} \log_2 \frac{4}{128} \nonumber \\
		&= \frac{4}{128} \log_2 \frac{128}{4} = 0.15625
	\end{align}

	Therefore
	\begin{align}
		H_{new}
		&= H_{init} - H_s + H_a \nonumber
		\\
		&= \log_2(128) - \frac{4}{128} \log_2(128) + \frac{4}{128} \log_2 \frac{128}{4} \nonumber \\
		&= 6.9375
	\end{align}

	\begin{figure}
		\centering
		\captionsetup[subfigure]{width=0.9\textwidth}
		\begin{subfigure}[t]{0.45\textwidth}
		\centering
		\includegraphics[width=6.5cm]{{{square-lattice-8x8-empty}}}
		\caption{Entropy is maximum. $H=\log_2(128) = 7.0$}
		\end{subfigure}
		\begin{subfigure}[t]{0.45\textwidth}
		\centering
		\includegraphics[width=6.5cm]{{{square-lattice-8x8-1-site}}}
		\caption{$H_s=\frac{4}{128} \log_2(128)$ for $4$ bonds that are to be connected and $H_a=\frac{4}{128} \log_2 \frac{128}{4}$ for new cluster of $4$ bonds. This gives $H_{new} = H_{old} - H_s + H_a = 0.69375$. Evidently entropy is decreasing.}
		\end{subfigure}
		\caption{Entropy of a system in initial state and one step after it.}
		\label{fig:efficient-entropy-1}
	\end{figure}

	In the same system after some time there will be less small cluster and more large. Say we are in a state of the system consists of $2$ large cluster. We mark them as green and red cluster. Then entropy for green and led cluster is
	\begin{align}
		H_{green} &= -\frac{13}{128} \log \frac{13}{128} \\
		H_{red} &= -\frac{7}{128} \log \frac{7}{128}
	\end{align}
	Since $S_{green} = 13$ and $S_{red} = 7$.
	And all other clusters are of size $1$. Therefore the entropy of the clusters of size $1$ is
	\begin{equation}
		H_{one} = -\sum_{i=1}{128-7-13} \frac{1}{128} \log_2 \frac{1}{128} = \frac{108}{128} \log_{2} 128
	\end{equation}
	Thus the total entropy is $H = H_{one} + H_{green} + H_{red}$ (see figure \ref{fig:efficient-entropy-2a}).

	Now say we select the site connecting  green and red cluster to occupy. Then
	\begin{equation}
		H_s = H_{green} + H_{red}
	\end{equation}
	and
	\begin{equation}
		H_a = - \frac{22}{128} \log_{2} \frac{22}{128}
	\end{equation}
	since size of the new cluster after occupying the selected site is $22$.

	Then the entropy of the system is
	\begin{equation}
		H = H_{one}
	\end{equation}
	\ref{fig:efficient-entropy-2b}.
	\begin{figure}
	\centering
	\captionsetup[subfigure]{width=0.9\textwidth}
	\begin{subfigure}[t]{0.45\textwidth}
		\centering
		\includegraphics[width=6.5cm]{{{entropy-square-lattice-8x8-2-cluster-1}}}
		\caption{$H_{old} = H_{one} + H_{green} + H_{red}$}
		\label{fig:efficient-entropy-2a}
	\end{subfigure}
	\begin{subfigure}[t]{0.45\textwidth}
		\centering
		\includegraphics[width=6.5cm]{{{entropy-square-lattice-8x8-2-cluster-2}}}
		\caption{$H_{new} = H_{old} - H_{s} + H_{a}$ where $H_s = H_{green} + H_{red}$ is to be found from figure \ref{fig:efficient-entropy-2a}}
		\label{fig:efficient-entropy-2b}
	\end{subfigure}
	\caption{Entropy of a system in a middle state and one step after.}
	\label{fig:efficient-entropy-2}
	\end{figure}