\begin{figure}[ht]
\centering
\includegraphics[width=0.7\textwidth]{figures/eca_images_and_complexity}
\caption{Example ECA rules from the four Wolfram classes. Each ECA has a state
    size of $N=200$ states and is run for $400$ steps. The latter $200$ steps
    are visualized above. We compute the HIC and statistical complexity (SC)
    for each ECA.}
\label{fig:eca_images_and_complexity}
\end{figure}

\section{Experiments}

\subsection{Experimental Setting}

We examine HIC empirically using elementary cellular automata (ECA). We also
compare HIC to statistical complexity for various ECA. Cells in an ECA
can take on one of two states. The update rule for a given state depends on the
state at the previous time step along with its two neighbors; hence there are
$2^{(2^3)} = 256$ possible ECAs. \citet{wolfram1983} proposed a four class
system to categorize ECA rules by their typical behavior. Class 1 ECAs converge
to a constant state, class 2 ECAs tend to exhibit periodic oscillations, class
3 ECAs display random-like behavior, and class 4 ECAs show complex behavior.
Complex behavior of an ECA is not well defined and is determined mostly through
inspection. The complex ECAs usually have various high-level structures
propagating through time with some degree of apparent randomness interspersed
between. Despite the simplicity of the rules, ECAs can yield astonishingly
complex behavior. Rule 110, for example, has been shown to be computationally
universal~\citep{cook2004universality}. We follow the Wolfram classification
given by~\citet[table 2]{martinez2013note} to classify all 256 ECA rules.

For each ECA we use a state size of $N$. The information in the initial state
requires at least $N/2$ updates to propagate across the complete state space.
We update the ECA for $2N$ time steps and discard the first $N$ which provides
a buffer for information in the initial state to fully propagate. We use a
circular update so the cells at the edges are influenced by neighbors at the
opposite edge of the state.  Because of this, rules which propagate information
in a single direction can influence the full state space after $N$ updates.

We use the remaining $N \times N$ cells to compute the HIC. At each level we
set $X^\ell$ and $Y^\ell$, the variables used to compute the mutual information
in equation~\ref{eq:mutual_information}, to be neighboring sequences each
consisting of $\ell$ cells. So for the first level ($\ell = 1$) $X$ and $Y$ are
single and consecutive cells, for $\ell = 2$ the variables are consecutive but
disjoint pairs of cells, and so on. We estimate the distributions $P(X^\ell)$
and $P(X^\ell \mid Y^\ell)$ used to compute the mutual information at each
level from counts over the states.

To construct the causal states for statistical complexity, we use the
Kullback-Leibler divergence with a threshold of $1.0$. We use a past light cone
with $T_{\textrm{P}}=2$ and a future light cone with $T_{\textrm{F}} = 3$.  For
each time step the number of states in the light cone grows by two. For
example, the third time step of the future light cone contains five cells. As
with HIC we estimate the distributions using counts over the states.

\begin{figure}[ht!]
\centering
\begin{subfigure}{\textwidth}
    \centering
    \includegraphics[width=0.8\textwidth]{figures/hic_by_class}
    \caption{Hierarchical information content (HIC)}
    \label{fig:hic_by_class}
\end{subfigure}
\begin{subfigure}{\textwidth}
    \centering
    \includegraphics[width=0.8\textwidth]{figures/sc_by_class}
    \caption{Statistical complexity (SC)}
    \label{fig:sc_by_class}
\end{subfigure}
\caption{The median and 95\% confidence intervals for the HIC for each Wolfram
    class. We compute these by randomly sampling 100 rules with random initial
    states and computing the HIC of each. For each class, the first twenty
    trials within one standard deviation of the median are plotted.}
\label{fig:complexity_by_class}
\end{figure}

\subsection{Comparing HIC to Statistical Complexity}

Figure~\ref{fig:eca_images_and_complexity} shows a canonical ECA rule from each
class: rule 128 is class 1, rule 2 is class 2, rule 30 is class 3, and rule 110
is class 4. For these simulations we use $N\!=\!200$ and an initial state of all
inactive cells (zeros) with a single active cell (one) at the center. We
compute the HIC over $L\!=\!3$ levels. The HIC is given for each rule above the
corresponding image. Note that HIC correctly distinguishes rule 110 as being
the most complex ECA from the remainder, which have HICs very close to zero.
Note in particular that the HIC of the class 3 ECA (rule 30) is even smaller
than the corresponding class 2 ECA (rule 2). In this case HIC peaks in between
order and disorder, as desired. We also see that the statistical complexity
(SC) correctly orders the rules from each class. However, unlike HIC the class
3 ECA (rule 30) has a fairly large statistical complexity. This is the case
because the ECA exhibits randomness over the spatial dimension but has some
structure over the temporal dimensions.

Figure~\ref{fig:complexity_by_class} demonstrates that both HIC and statistical
complexity can distinguish the class 4 ECA from the other three complexity
classes. These results are obtained by sampling $100$ rules for each of the
four ECA classes. Each rule is run from a random initial state for $1000$
steps. We use a state size of $N\!=\!500$ and hence a $500 \times 500$ grid to
compute the corresponding complexity measure. For these experiments we use
$L=6$ levels for the HIC. The class medians are well separated by the median
HIC (fig.~\ref{fig:hic_by_class}) with the 95\% confidence interval of class 4
distinct from that of any other class.

We also plot in figure~\ref{fig:hic_by_class} the first twenty trials for each
class with an HIC within one standard deviation of the median. Rules from ECA
class 2 have the widest range of HIC. Qualitative inspection of the high HIC
results shows many cases where the resulting state is simply shifted to the
left or right by the rule. The state exhibits a simple periodic pattern over
time but because of the random initialization, the pattern over space does not
appear to be simple. This is one failure mode of HIC -- it does not
capture the simplicity over the time dimension given that it is only applied to
the space dimension.

The statistical complexity medians are less well separated than those computed
from the HIC. On the other hand, the statistical complexity
(fig.~\ref{fig:sc_by_class}) has narrower 95\% confidence intervals for all
classes. Overall, both measures are quite good at distinguishing complexity
class 4 ECA from the remainder. Both measures could be useful depending on
whether one intends to measure process or structural complexity. The two may
also be combined to produce a single more comprehensive measure.

\subsection{Number of Levels}

\begin{figure}[t]
\centering
\begin{subfigure}{0.45\textwidth}
  \includegraphics[width=1.0\textwidth]{figures/hic_vs_num_levels_size_200}
  \caption{State size $N = 200$}
  \label{fig:hic_levels_200}
\end{subfigure}
\begin{subfigure}{0.45\textwidth}
  \includegraphics[width=1.0\textwidth]{figures/hic_vs_num_levels_size_500}
  \caption{State size $N = 500$}
  \label{fig:hic_levels_500}
\end{subfigure}
\caption{We show the HIC versus the number of levels $L$ used in the
    computation. We compute the HIC for one rule from each of the four Wolfram
    classes. Each ECA is run for $2N$ time steps and the HIC is computed from
    the latter half.}
\label{fig:hic_vs_levels}
\end{figure}

Using the same four rules (class 1 rule 128, class 2 rule 2, class 3 rule 30,
and class 4 rule 110), we observe the effect of the number of levels used to
compute the HIC in figure~\ref{fig:hic_vs_levels}.
Figure~\ref{fig:hic_levels_200} uses a state size of $N\!=\!200$, and
figure~\ref{fig:hic_levels_500} uses a state size of $N\!=\!500$. In both
figures, we see that the class 1 and 2 rules (the curves overlap at zero) do
not grow with an increase in the number of levels. The class 4 rule grows but
plateaus with the number of levels. We expect to see a plateau once the size of
the variables $X^\ell$ and $Y^\ell$ fully capture the largest structures, and
so the mutual information should become constant.  This shows that complexity
does not grow via compositions of order from order.  Interposing disorder
somewhere near the level of the largest structures would result in further
complexity growth.

The class 3 ECA is the only one which exhibits a qualitative difference between
the two figures. This is simply a finite sample statistical artifact which is
instructive to elucidate. In the $N\!=\!200$ case, we do not have enough states to
estimate the mutual information at the highest level ($\ell\!=\!7$) accurately.
The mutual information consists of two terms, the entropy of the single
variable $H(X^\ell)$ and the conditional entropy between the two variables
$H(X^\ell \mid Y^\ell)$. The entropy requires fewer samples to estimate
accurately than the conditional entropy. For example $X^7$ can take on $2^7$
possible values whereas the pair $(X^7, Y^7)$ admits $2^{14}$ distinct values.
With insufficient sample sizes, the conditional entropy shrinks prior to the
entropy and the mutual information is artificially inflated. This can be
remedied by increasing $N$. However, a parametric or otherwise more sample
efficient model for $P(X)$ and $P(X \mid Y)$ is an alternative that will likely
scale more robustly with the number of values of $X^\ell$.


\documentclass[a4paper,onecolumn]{article}
\usepackage[width=180truemm,height=260truemm]{geometry}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsfonts}
\usepackage{lmodern}
\usepackage{fourier}
\usepackage[T1]{fontenc}
\usepackage{graphicx}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{fancyhdr}
\usepackage{lastpage}
\usepackage{indentfirst}

\pagestyle{fancy}
\lhead{}
\rhead{}
\cfoot{-- \thepage{}/{}\pageref{LastPage} --}
\renewcommand{\headrulewidth}{0pt}
\renewcommand{\footrulewidth}{0pt}

\newcommand{\bs}[1]{\boldsymbol{#1}}
\newcommand{\tran}{^{\mkern-1.5mu\mathsf{T}}}

\title{Gaussian Process Bootstrapping Layer}
\author{Tetsuya Ishikawa}

\begin{document}

\maketitle
\thispagestyle{fancy}

\section{Theoretical details}
% {{{

The Gaussian process with random Fourier features can be regarded as a variant of a fully connected layer.
We can design a new fully connected layer that can compute variance of it's intermediate features.

\subsection{Gaussian process with random Fourier features}

Let $\mathcal{D} = \{ (\boldsymbol{x}_n, y_n) \}_{n=1}^{N}$ be a training dataset where
$\boldsymbol{x}_n$ is a $n$-th data and $y_n$ is a label of $n$-th data.
The Gaussian process with random Fourier features can be formulated as follows:
\begin{align}
    \boldsymbol{\phi}_{\boldsymbol{x}_n}
    &= f(\boldsymbol{Wx}_n), \\
    \boldsymbol{P}
    &= \sum_{n = 1}^{N} \boldsymbol{\phi}_{\boldsymbol{x}_n}
    \boldsymbol{\phi}_{\boldsymbol{x}_n}^{\mkern-1.5mu\mathsf{T}}, \\
    m(\boldsymbol{x}_i)
    &= \frac{1}{\sigma^2} \boldsymbol{y}^{\mkern-1.5mu\mathsf{T}} \boldsymbol{\Phi}^{\mkern-1.5mu\mathsf{T}}
    \left( \boldsymbol{I} - (\boldsymbol{P} + \sigma^2 \boldsymbol{I})^{-1} \right) \boldsymbol{\phi}_{\boldsymbol{x}_i}, \\
    v(\boldsymbol{x}_i, \boldsymbol{x}_j)
    &= \boldsymbol{\phi}_{\boldsymbol{x}_i}^{\mkern-1.5mu\mathsf{T}} \left\{
    \boldsymbol{I} - \frac{1}{\sigma^2} \boldsymbol{P} \left( \boldsymbol{I} - (\boldsymbol{P} + \sigma^2 \boldsymbol{I})^{-1} \right)
    \right\} \boldsymbol{\phi}_{\boldsymbol{x}_j},
    \label{eqn:gp_variance}
\end{align}
where $f$ is a non-linear function and $\boldsymbol{W}$ is a random matrix derived from a kernel function of the Gaussian process.

\subsection{Analogy to a fully connected layer}

The formula $f(\boldsymbol{Wx})$ looks like a fully connected layer of neural network with activation function.
For example, if the kernel function is RBF kernel $k(\bs{x}_1, \bs{x}_2) = \exp \bigl( \| \bs{x}_1 - \bs{x}_2 \|^2 \bigr)$,
then $\boldsymbol{\phi}_{\boldsymbol{x}_n}$ can be written as
\begin{equation}
    \boldsymbol{\phi}_{\boldsymbol{x}_n} =
    \begin{pmatrix}
        \cos \bs{Wx}_n \\
        \sin \bs{Wx}_n
    \end{pmatrix},
    \hspace{10pt}
    \bs{W} \sim \mathcal{N} (0, \sigma^2).
\end{equation}
In this case, $\bs{Wx}_n$ corresponds to a fully connected layer and the
circuler functions correspond to an activation function.

On the other hand, $v(\boldsymbol{x}_n, \boldsymbol{x}_n)$ correspond to the variance of input data $\boldsymbol{x}_n$.
Notable point is that the $v(\boldsymbol{x}_n, \boldsymbol{x}_n)$ is not depend on the label $y_n$,
in other words, variance $v(\boldsymbol{x}_n, \boldsymbol{x}_n)$ is the same for any label.
See the equation (\ref{eqn:gp_variance}).

\subsection{Gaussian process bootstrapping layer}

Bacause of the independence of the variance and $v(\boldsymbol{x}_n, \boldsymbol{x}_n)$ and the label $y_n$
mentioned in the previous subsection, we can replace the expectation prediction part as a identity function
(theoretically, this situation correspond that $y_n = \bs{\phi}_{\bs{x}_n}$). See the figure \ref{fig:sketch_gpbl}.
Therefore, we've got a new layer that can predict variance of intermediate features by replacing
the random Fourier features as a fully connected layer, and expectation prediction as a identity function.

Gaussian process bootstrapping layer is a layer to add noises to the intermediate features
where the variance of the noises is the variance of the intermediate features.

\begin{figure}[t]
    \center
    \includegraphics[width=300pt]{../figures/sketch_gpbl.pdf}
    \caption{Illustration of the analogy between GP w/ RFF and GPB layer}
    \label{fig:sketch_gpbl}
\end{figure}

\subsection{Psuedo code of GPB layer}

The algorithm \ref{alg:gpbl} is the pseudo code of the GPB layer.

\begin{algorithm}[t]
    \caption{Gaussian process bootstrapping layer}
    \label{alg:gpbl}
    \textbf{Input} \\
    \hspace*{\algorithmicindent}
    \begin{tabular}{rl}
        $\bs{X}$: & \hspace*{-10pt} tensor with shape $(N, C)$ \\
    \end{tabular} \\
    \textbf{Output} \\
    \hspace*{\algorithmicindent}
    \begin{tabular}{rl}
        $\bs{Y}$: & \hspace*{-10pt} tensor with shape $(N, C)$
    \end{tabular} \\
    \textbf{Hyperpatameters} \\
    \hspace*{\algorithmicindent}
    \begin{tabular}{rl}
        $\sigma$: & \hspace*{-10pt} standard deviation of measurement error \\
        $\alpha$: & \hspace*{-10pt} coefficient of exponential moving average \\
        $s$:      & \hspace*{-10pt} number of steps to skip bootstrapping \\
    \end{tabular} \\[10pt]
    \textbf{function}
    \textsc{Gaussian process bootstrapping layer}$(X, \alpha, \sigma)$ \\[5pt]
        \noindent\hspace*{\algorithmicindent}
        \texttt{\# Update matrix P with exponential moving average.} \\[1pt]
        \noindent\hspace*{\algorithmicindent}
        $\bs{P} = \alpha \bs{X}\tran \bs{X} + (1 - \alpha) \bs{P}$ \\

        \vspace*{-6pt}
        \noindent\hspace*{\algorithmicindent}
        \texttt{\# Compute matrix M.} \\[1pt]
        \noindent\hspace*{\algorithmicindent}
        $\bs{M} = \bs{I} - \frac{1}{\sigma^2} \bs{P} \bigl( \bs{I} - (\bs{P} + \sigma^2 * \bs{I})^{-1} \bs{P} \bigr)$ \\

        \vspace*{-6pt}
        \noindent\hspace*{\algorithmicindent}
        \texttt{\# Compute variance v[n].} \\[1pt]
        \noindent\hspace*{\algorithmicindent}
        $\textbf{for} \,\, n \,\, \textbf{in} \,\, [0, N):$ \\
        \noindent\hspace*{\algorithmicindent}\hspace*{\algorithmicindent}
        $v[n] = X[n, :]\tran M X[n, :]$ \\

        \vspace*{-6pt}
        \noindent\hspace*{\algorithmicindent}
        \texttt{\# Add perturbation to the input tensor X.} \\[1pt]
        \noindent\hspace*{\algorithmicindent}
        $\textbf{for} \,\, n \,\, \textbf{in} \,\, [0, N):$ \\
        \noindent\hspace*{\algorithmicindent}\hspace*{\algorithmicindent}
        $Y[n, :] = X[n, :] + \sqrt{v[n]} \, \bigl( \textrm{sampling from normal distribution with shape} \,\, (1, C) \bigr)$ \\[5pt]
    \textbf{end function}

\end{algorithm}

% }}}

\begin{thebibliography}{9}
% {{{

    \bibitem{Rasmussen2006}
        C. Rasmussen and C. Williams, “Gaussian Processes for Machine Learning”, MIT Press, 2006.

    \bibitem{Rahimi2007}
        A. Rahimi and B. Recht, "Random Features for Large-Scale Kernel Machines", NIPS, 2007.

% }}}
\end{thebibliography}

\end{document}

% vim: expandtab tabstop=4 shiftwidth=4 fdm=marker


\subsection{Model Description}

The plane-strain slope stability problem has a height of $50m$ and a depth of $80m$ with a $30m$ high slope with a grade of $300\%$ (Figure \ref{fig:slopeGeom}). This geometry provides enough space for the failure mechanisms to occur with little influence from the boundaries. The lateral boundary conditions are zero displacement in the x-direction, and the bottom boundary conditions are zero displacement in the y-direction. The slope and top boundaries are free. A uniformly distributed load was applied over a 5m section on the backslope in a linear incremental fashion until failure.


\documentclass[a4paper,12pt]{report}

\usepackage{amsmath,amsfonts,mathtools}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{hyperref}

\begin{document}
\title{MAT292 Abridged}
\author{Aman Bhargava}
\date{September 2019}
\maketitle

\tableofcontents

\section{Introduction}
The textbook and lectures for this course offer a great comprehensive guide for the
methods of solving ODE's. The goal here is to give a very concise overview of the things you
need to know (NTK) to answer exam questions. Unlike
some of our other courses, you don't need to be very intimately familiar with the derivations
of everything in order to solve the problems (though it certainly doesn't hurt). Think of this
as a really good cheat sheet.

\chapter{Qualitative Things and Definitions}
\section{Definitions}
\begin{enumerate}
\item \textbf{Differential Equation: } Any equation that contains a differential of dependent variable(s) with respect to any independent variable(s)
\item \textbf{Order: } The order of the highest derivative present.
\item \textbf{Autonomous: } When the definition of the $\frac{dy}{dt}$ doesn't contain $t$
\item \textbf{ODE and PDE: } Ordinary derivatives or partial derivatives.
\item \textbf{Linear Differential Equations: } $n$th order Linear ODE is of the form: $$\sum a_i(t)y^{(i)} = 0$$
\item \textbf{Homogenous: } if the $0$th element of the above sum has $a_0(t) = 0$ for all $t$.
\end{enumerate}

\section{Qualitative Analytic Methods to Know}
\begin{enumerate}
\item Phase lines
\item Slope fields
\end{enumerate}

\section{Types of Equilibrium}
\begin{enumerate}
\item Asymptotic stable equilibrium
\item Unstable equilibrium
\item Semistable equilibirium
\end{enumerate}

\chapter{1st Order ODE's}
\section{Separable 1st Order ODE's}
If you can write the ODE as: $$\frac{dy}{dx} = p(x)q(y)$$

Then you can put $p(x)$ with $dx$ on one side and $q(y)$ with  $dy$ on the other and
integrate them both so solve the ODE.

\section{Method of Integrating Factors}
This is used to solve ODE's that can be put into the form
$$\frac{dy}{dt} + p(t)*y = g(t)$$

The chain rule can be written as: $\int (f'(x)g(x) + f(x)g'(x)) dx =  f(x)g(x)$

We can use an \textbf{integrating factor} equivalent to $e^{\int p(t) dt}$ to multiply
both sides and arrive at a form that can be integrated with ease using the reverse chain
rule.

\section{Exact Equations}
If the equation is of the form $$M(x, y) + N(x, y) \frac{dy}{dx} = 0$$
and $$M_y(x, y) = N_x(x, y)$$
then $\exists$ a function $f$ satistfying $$f_x(x, y) = M(x, y); f_y(x, y) = N(x, y)$$

\paragraph{The solution: } $f(x, y) = C$ where $C$ is an arbitrary constant.

\section{Modeling with First Order Equations}
These are some vague tips on how to solve these types of problems from textbook section
2.3
\begin{itemize}
\item To \textbf{create} the equation, state physical principles
\item To \textbf{solve}, solve the equation and/or find out as much as you can about the nature of the solution.
\item Try \textbf{comparing} the solution/equation to the physical phenomenon to `check' your work.
\end{itemize}

\section{Non-Linear vs. Linear DE's}
\paragraph{Theorem on Uniqueness of 1st Order Solutions}
$$y' + p(t)y = g(t)$$
There exists a unique solution $y=\Phi(t)$ for each starting point $(y_0, t_0)$ if
$p$, $g$ are continuous on the given interval.

\section{Population Dynamcis with Autonomous Equations}
\paragraph{Autonomous: } $\frac{dy}{dt} = f(y)$

\subsection{Simple Exponential}
$\frac{dy}{dt} = ry$
Problem: doesn't take into account the upper bound for population/sustainability.

\subsection{Logistic equation}
$$\frac{dy}{dt} = (r-ay)y$$
Equivalent form:
$$\frac{dy}{dt} = r(1 - \frac{y}{k})y$$
$f$ is the \textit{intrinsic growth rate}.

\chapter{Systems of Two 1st Order DE's}
\section{Set Up}
Your first goal is to get the system in the form $$\frac{d \pmb{u}}{dt} = \pmb{Ku + b}$$
Where $\pmb{K}$ is a 2 by 2 matrix, $\pmb{u}$ is your vector of values you want to
predict, and $\pmb{b}$ is a $2$-long vector of constants.

\paragraph{More generally, } the equation is of the type
$$\frac{d\pmb{x}}{dt} = \pmb{P}(t)x + \pmb{g}(t)$$

Called a \textbf{first order linear system of two dimensions}. If $\pmb{g}(t) = \pmb{0} \forall t$
then it is called \textbf{homogenous}, else \textbf{non-homogenous}. We let $x$ be composed
of values $$\pmb{x} = \begin{bmatrix}
                           x \\
                           y \\
                      \end{bmatrix}$$

\section{Existence and Uniqueness of Solutions}
\paragraph{Theorem: } $\exists$ unique solution to $$\frac{d\pmb{x}}{dt} = \pmb{P}(t)x + \pmb{g}(t)$$
so long as the functions $\pmb{P}(t)$ exist and are continuous on the interval $I$ in equestion.

\subsection{Linear Autonomous Systems}
If the right side doesn't depend on $t$, it's autonomous. In this case, the autonomous
version looks (familiarly) like: $$\frac{d\pmb{x}}{dt} = \pmb{A}x + \pmb{b}$$

\paragraph{Equilibrium points} arise when $\pmb{A}x = -\pmb{b}$

\section{Solving}
\subsection{General Solution}
We start with $y' = \pmb{A}y+b$
\begin{itemize}
\item Find eigen values $\lambda$ s.t. $det(A-I\lambda) = 0$
\item Find eigen vectors $v$ s.t. $(A-I\lambda)v = 0$
\item Enter and simplify $y(t) = C_1 v_1 e^{\lambda_1 t} + C_2 v_2 e^{\lambda_2 t}$
\end{itemize}
\paragraph{Converting to homogenous equation: } Let $y_{eq}$ be the equilibrium value of $y$ that
can be found when $y' = 0 = Ay + b$.
$$y_{eq} + \bar{y} = y$$ and $y_{eq}$ is the solution to $\bar{y}' = A\bar{y}$.

\subsection{Special Case 1: Repeated Eigen Value}
Start in the same fashion as above. You will easily be able to find the eigen value and
at least one eigen vector. Then, the path diverges:

\paragraph{Case 1: Another can easily be found - } Now you find your $v_2$ and proceed.

\paragraph{Case 2: Another cannot easily be found - } You must use the following formula
to find your second vector if this is the case:
$$(A-I\lambda)v_2 = v_1$$
This is known as the "general" eigen vector.

\subsubsection{Final Form}
Your final form for this case is going to be rather different than the others:
$$x = C_1e^{\lambda_1 t}tv_1 + C_2e^{\lambda_2 t}v_2$$

\subsection{Special Case 2: Two Complex Eigen Values}

\chapter{Numerical Methods}
$$\frac{dy}{dt} = f(t, y)$$
\section{Euler's Method}
We start with a first order ODE. Let us define a fixed step $\Delta t$.
$$y_{n+1} = y_n + \Delta t(f(t_n, y_n))$$
Error = $|y(t_n)-y_n| \approx \Delta t$

\subsection{Basic Idea: Integrate The ODE}
$$\int_{t_n}^{t_n+1} \frac{dy}{dt} dt = \int_{t_n}^{t_n+1} f(t, y(t)) dt$$
Euler's method makes the following approximation.
$$\int_{t_n}^{t_n+1} f(t, y(t)) dt \approx \Delta t f(t_n, y_n)$$
But we can do better.

\subsubsection{Mean Value Theorem for Integrals}
If y is continuous on $[a, b]$ then $\exists c \in (a, b)$ so that
$$\frac{1}{b-a} \int_a^b g(t) dt = g(c)$$

Euler's method would just assume that $g(c)$ is at the far left hand side of the
Riemann sum, so we can improve upon this! If we can guess $c$ more accurately,
our final answer will be a lot better.

\paragraph{Since $c$} is more likely to be inside the interval $[t_n, t_{n+1}$, we could
try the following estimations to improve upon Euler's method. We will now try \textbf{sampling}.

\section{Improved Euler Method}
Let $g(t) = f(t, y(t))$. We literally use that riemann sum trapezoidal rule for this approximation.

$$y_{n+1}-y_n \approx \frac{\Delta t}{2}(f(t_n, y_n)+f(t_{n+1}, y_{n+1}))$$
Where we make the approximation for $y_{n+1}$ as
$$y_{n+1} \approx y_n + \Delta t f(t_{n+1}, y_n)$$

\paragraph{Steps: }
\begin{itemize}
\item Evaluate $K_1 = f(t_n, y_n)$
\item Predict $u_{n+1} = y_n + \Delta t K_1$
\item Evaluate $K_2 = f(t_{n+1}, u_{n+1})$
\item Update $u_{n+1} = u_n + \Delta t \frac{k_1+k_2}{2}$
\end{itemize}

This method is consider \textbf{second order}, so $$|y(t_n)-y_n| \approx C(\Delta t)^2$$ (global error).

\paragraph{The expense } of a numerical method is roughly the \textbf{number of function calls to $f()$}.
Therefore, improved Euler's method comes at the cost of one more function evaluation of $f()$.

\section{Runge Kutta Method}
Modern workhorse of solving ODE's. It's 4th order, so requires 4 function $f$ calls.

\paragraph{Steps}
\begin{itemize}
\item $k_1 = f(t_n, y_n)$
\item $u_n = y_n + \frac{\Delta t}{2} k_1$ (half step)
\item $k_2 = f(t_n + \frac{\Delta t}{2}, u_n$
\item $v_n = y_n + \frac{\Delta t}{2} k_2$
\item $k_3 = f(t_n+\frac{\Delta t}{2}, v_n$
\item $w_n = y_n + \Delta t k_3$
\item $k_{-1} = f(t_{n+1}, w_n$
\item $y_{n+1} = y_n + \Delta t(\frac{k_1}{6}+\frac{k_2}{3}+\frac{k_3}{3}+\frac{k_4}{6}$
\end{itemize}

\section{Above and Beyond 4th Order}
If we can just increase our accuracy by adding more functional evaluations, then why can't we just keep on
adding function evaluations and increasing the order?

\begin{tabular}{l|llllllllll}
\textbf{Order} & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 \\
\hline
\textbf{Min Function Evaluations} & 1 & 2 & 3 & 4 & \textbf{6} & 7 & \textbf{9} & 11 & 14 & ? \\
\end{tabular}

Answer: Past a 4 evaluations, it's not really worth while.

\chapter{Systems of First-Order Equations}
\section{Theory of First-Order Linear Systems}
\paragraph{$n\times n$ Linear System: } $$\vec{x}' = \pmb{P}(t)\vec{x} + \vec{g}(t)$$
\paragraph{Existence and Uniqueness Theorem: } If $\pmb{P},\vec{g}$ are continuous on $[a, b]$, there exists unique $\vec{x}(t)$ for the IVP.

For constant $n\times n $ matrix, we use $\pmb{A} = \pmb{P}(t)$.

\paragraph{SUPERPOSITION: } For linear systems, any linear combination of solutions is another solution.

\section{Wronskians}
$$W = W[x_1,x_2,..., x_n](t) = det(\pmb{X}(t))$$
Where $\pmb{X}(t)$ is an $n\times n$ matrix with column vectors being solutions to the problem.

\paragraph{THEOREM: } If each column vector $\vec{x}(t)$ solution is linearly independent, then $W[x_1, ..., x_n] \neq 0$ for all time in the
given interval. If $W(t) = 0$, that tells us that the solutions are not linearly independent.

\paragraph{THEOREM: } There exists at least one funadmental set of solutions for all linear systems.

\section{Fundamental Matrices}
If $\{\vec{x}_1(t), \vec{x}_2(t), ..., \vec{x}_n(t)\}$ are solutions to $\vec{x}' = \pmb{P}(t)\vec{x}$ then the \textbf{FUNDAMENTAL MATRIX} is

$$\pmb{X}(t) = [\vec{x}_1(t), ..., \vec{x}_n(t)]$$
\begin{enumerate}
\item It is invertible
\item Any solution to the IVP can be written as $\vec{x}(t) = \pmb{X}(t)\vec{c}$ where $\vec{c} \in \mathbb{R}^n$.
\item $\pmb{X}' = \pmb{P}(t)\pmb{X}$
\end{enumerate}

\section{Matrix Exponential}
Motivation: For $x' = ax$, the solution is $x = x_0 e^{at}$. What's the equivalent to $e^{at}$ for matrices?
\paragraph{Definition of Matrix Exponential: } $$e^{\pmb{A}t} = \sum_{k=0}^{\infty}\pmb{A}^k\frac{t^k}{k!}$$
That's not terribly useful, so we dive deeper. \textbf{THEOREM: }

$$e^{\pmb{A}t} = \pmb{\Phi(t)}$$

Where $\pmb{\Phi(t)} = \pmb{X}(t)\pmb{X}^{-1}(t_0)$ such that $\pmb{\Phi(t)} = \pmb{I}$ at $t = t_0$. In essence, it's a
special fundamental matrix.

\paragraph{Properties: } for $\pmb{A}, \pmb{B} \in \mathbb{R}^{n\times n}$:
\begin{enumerate}
\item $e^{A(t + \tau)} = e^{At}e^{A\tau}$
\item $Ae^{At} = e^{At}A$
\item $(e^{At})^{-1} = e^{-At}$
\item $e^{(A+B)t} = e^{At + Bt}$ it $AB = BA$
\end{enumerate}

\subsection{Constructing Matrix Exponential}
Let $[\vec{x}_1(t), ..., \vec{x}_n(t)] = \pmb{X}(t)$ be a fundamental set of solutions. Then $$e^{At} = \pmb{X}(t)\pmb{X}^-1(t_0)$$

\paragraph{When A is non-defective } (i.e. has complete set of eigen values and is diagonalizable):

$$X(t) = [e^{\lambda_1 t}V_1 + ... + e^{\lambda_n t}V_n]$$

$$\Phi(t) = \pmb{I}[e^{\lambda_1 t}, e^{\lambda_2 t}, ..., e^{\lambda_n t}]$$

\chapter{Second-Order Linear Equations}
These equations combine $t, y, y', y''$: $$y'' = f(t, y, y')$$ Initial conditions are specified as $y(0), y'(0)$.

\paragraph{Linearity: } If and only if it is of the form $y'' + p(t)y' + q(t)y = g(t)$.
\paragraph{Homogenous: } If $g(t) = 0$ for all $t$ in the interval.
\paragraph{Constant Coefficients: } $ay'' + by' + cy = g(t)$

\section{Dynamical System Formulation}
We can convert this into a first-order system by stating:
\begin{enumerate}
\item $x'_1 = x2$
\item $x'_2 = f(t, x1, x2)$
\end{enumerate}
Where $x_1 = y$, and $x_2 = y'$. In vector notation:

$$\vec{x}' = \vec{f}(t, x_1, x_2) = [x_2, f(t, x_1, x_2)$$

\section{Theory of Second Order Linear Homogenous Systems}
\paragraph{Existence: } There exists a solution for $y'' + p(t)y' + q(t)y = g(t)$ as long as $p, q, g$ are cts.

\subsection{Abel's Theorem}
Let $\vec{x}' = \pmb{P}(t)\vec{x}$. Then the Wronskian $$W(t) = c[\exp(\int tr(\pmb(P)(t)))dt]$$
Where the trace is the sum of the diagonal.

\section{Linear Homogenouse Equations with Constant Coefficients}
$$ay'' + by' + cy = 0$$
$$\vec{x}' = \pmb{A}\vec{x} = \begin{bmatrix}
0 & 1 \\
\frac{-c}{a} & \frac{-b}{a} \notag
\end{bmatrix}
\vec{x}
$$

\subsection{Solution}
$$\vec{x} = \sum e^{\lambda_n t}\vec{V}_n$$
\begin{enumerate}
\item Find eigen values of $\pmb{A}$.
\item Vector $V_n$ is $$\begin{bmatrix}
1 \\
\lambda \\
\end{bmatrix}$$
\end{enumerate}

\subsection{Phase Portraits}
\begin{itemize}
\item Real + Negative $\to $ Asymmetrically stable.
\item Real + Positive $\to $ Asymmetrically unstable.
\item Complex + Negative $\to $ Stable spiral in.
\item Complex + Positive $\to $ Unstable spiral out.
\end{itemize}
\textbf{ALWAYS CLOCKWISE}.

\section{Mechanical and Electrical Vibration}
Pretty much just what we did in physics.

\section{Method of Undetermined Coefficients}
$$y'' + y' + y = f(t)$$
This can be applied when $f(t)$ is one of the following:
\begin{enumerate}
\item $f(t) = e^{st}$: Try $y_p = ae^{st}$
\item $f(t) = \text{polynomial}^n$: Try $y_p = a_n t^n + a_{n-1}t^{n-1} + ...$
\item $f(t) = \sin t$: Try $y_p = c_1\cos(t) + c_2\sin(t)$
\item $f(t) = t\sin t$: Try $y_p = (a+bt)\cos(t) + (c+dt)\sin(t)$
\end{enumerate}
\subsection{Resonance}
The one situation that breaks the system is $y'' - y = e^t$. We might try $y_p = ae^t$, but that will cancel out no matter what!
This is called \textbf{resonance}.

To get around this, we add in a $t$ term: $y_p = tae^t$

\section{Variation of Parameters}
$$y'' + B(t)y' + C(t)y = f(t)$$
First we must find two null solutions $y_1(t), y_2(t)$ that solve $y'' + by' + cy = 0$. Our final solution will be of the form:

$$y(t) = c_1(t)y_1(t) = c_2(t)y_2(t)$$

Where $c_1, c_2$ are ``varying parameters''. When we plug in the hypothesized $y(t)$, we get

$$c_1' y_1 + c_2' y_2 = 0$$

$$c_1' y_1' + c_2' y_2' = f(t)$$

From these two, we can solve for $c_1'$ and $c_2'$. Then:

$$y(t) = y_1(t) \int_0^t \frac{(-y_2)f(t)}{W(t)} dt + y_2(t) \int_0^t \frac{y_1(t)f(t)}{W(t)} dt$$

Where $W(t) = \det \begin{vmatrix}
y_1 & y_2 \\
y_1' & y_2' \notag
\end{vmatrix}$

\chapter{The Laplace Transform}
The value-add of the Laplace trasnform is that it lets you convert an ODE into a different form, solve that different form,
then convert back.
\section{Definition and Properties}
$$\mathcal{L}\{f(t)\} = F(S) = \int_0^{\infty} e^{-st} f(t)dt$$

\subsection{Linearity}
$$\mathcal{L}\{ c_1f_1 + c_2f_2\} = c_1\mathcal{L}\{ f_1 \} + c_2 \mathcal{L}\{ f_2 \}$$

\subsection{Exponential Order}
\paragraph{IF } $|f(t)| \leq Ke^{at}$ as $t\to\infty$ for some $K, a \in \mathbb{R}$ it is of exponential order.

\subsection{Existence of $\mathcal{L}\{ f \}$}
\paragraph{CONDITION I: } The function must be piecewise continuous.
\paragraph{CONDITION II: } The function must be of exponential order.

If these two hold, the Laplace transform is defined and approaches zero as $s\to\infty$.

\section{Properties of Laplace Transform}
\begin{enumerate}
\item If $c$ is a constant then $$\mathcal{L}\{ e^{ct}f(t) \} \to F(s-c)$$
\item Derivatives of $f(t)$:
% $$\mathcal{L}\{ f'(t) \} \to sF(s) - f(0)$$
% $$\mathcal{L}\{ f''(t) \}  \to s^2 F(s) - sf(0)-f'(0)$$
$$\mathcal{L}\{ f^{(n)}(t) \} \to s^n \mathcal{L}\{ f(t) \} + s^{n-1} f(0) + ... + sf^{(n-2)}(0) + f^{(n-1)}(0)$$
\item Multiplying by powers of $t$:
$$\mathcal{L}\{ t^n f(t) \} \to (-1^n)F^{(n)}(s)$$
$$\mathcal{L}\{ t^n \} \to \frac{n!}{s^{n+1}}(-1)^n$$
\end{enumerate}

\section{Inverse Laplace Transform}
$$f = \mathcal{L}^{-1}\{ F(s) \}$$
This is usually a 1-1 correspondence, so we use a lookup table.

\subsection{Partial Fractions}
\begin{enumerate}
\item Factor the denominator
\item Write in terms of sums of $\frac{X}{factored term}$
\item Solve for coefficients of $X$ terms.
\item Use linearity of $\mathcal{L}^-1\{  \}$ to solve for $f(t)$.
\end{enumerate}

\paragraph{Rules for different types of factored denominators: }
\begin{itemize}
\item Single linear roots: $\frac{A}{s - c_1}$
\item Repeated linear roots: $\frac{A}{s - c_1} + \frac{B}{(s-c_1)^2} + ... + \frac{Z}{(s-c_1)^k}$
\item Quadratic roots: $\frac{A + Bs}{s^2 + c_1s + c_2}$
\end{itemize}

\section{Solving ODE's with $\mathcal{L} \{\}$}
\paragraph{Steps: }
\begin{enumerate}
\item Convert to $F(s), Y(s)$ space with $$\mathcal{L} \{\}$$
\item Solve for $Y(s)$ in terms of $s$.
\item Solve for $\mathcal{L}^{-1} \{Y(s)\}$
\end{enumerate}

\subsection{Characteristic Polynomial}
Let $$ay'' + by' + cy = f(t),\,\,\,\,\, y(0) = 0,\,y'(0) = 0$$
If we let $Z(s) = as^2 + bs + c$:
$$Y(s) = \frac{(as + b)y(0) + ay'(0)}{Z(s)} + \frac{F(s)}{Z(s)}$$

In the general case, if $Z(s) = \sum_{n=0}^{k} a_ns^n$:
$$Y(s) = \frac{ [\sum_{n=1}^m a_n s^{n-1}] y(0) + ... + [a_ns + a_{n-1}]y^{n-2}(0) + a_n y^{(n-1)}(0) }{Z(s)} + \frac{F(s)}{Z(s)}$$

\subsection{Systems of Differential Equations}
$$\vec{y}' = \pmb{A}\vec{y} + \vec{f}(t)$$ with $\vec{y}(0) = \vec{y}_0$.

We then (1) Take $\mathcal{L} \{\}$ of both sides and (2) re-arrange to find:
$$(s\pmb{I} - \pmb{A})\vec{Y}(s) = \vec{y}_0 + \vec{F}(s)$$
\begin{itemize}
\item $\vec{Y}(s)$ is just each function in $\vec{y}(t)$ run through laplace transform.
\item $\vec{F}(s)$ is just each function in $\vec{f}(t)$ run through laplace transform.
\end{itemize}

\paragraph{Therefore: }
$$\vec{Y}(s) = (s\pmb{I} - \pmb{A})^{-1} \vec{y}_0 + (s\pmb{I} - \pmb{A})^{-1}\vec{F}(s) $$

\section{Discontinuous and Periodic Functions}
\subsection{Heaviside function}
(a.k.a. unit step function) is $u(t)$:
\[ u(t) =
\begin{cases}
0 & \text{if } t < 0 \\
1 & \text{if } t \geq 0
\end{cases}
\]

\[ u_c(t) =
\begin{cases}
0 & \text{if } t < c \\
1 & \text{if } t \geq c
\end{cases}
\]

\[ u_{cd}(t) =
\begin{cases}
0 & \text{if } t < c \text{ or } t > d \\
1 & \text{if } c \leq t < d
\end{cases}
\]

\[
u_{cd} = u_c(t) - u_d(t)
\]

\paragraph{Laplace Transforms of Heaviside functions: }
$$\mathcal{L}\{ u_c(t) \} \to \frac{e^{-cs}}{s}$$
$$\mathcal{L}\{ u_{cd}(t) \} = \mathcal{L}\{ u_c(t) - u_d(t) \} \to \frac{e^{-cs} - e^{-ds}}{s}$$

\subsection{Time-Shifted Functions}
Consider $y = g(t) =
\begin{cases}
0 & \text{if } t < c \\
f(t-c) & \text{if } t \geq c
\end{cases}$. Therefore $g(t) = u_c(t)f(t-c)$

$$\therefore \mathcal{L}\{ u_c(t)f(t-c) \} \to e^-cs\mathcal{L}\{ f(t) \}$$
$$\therefore \mathcal{L}^{(-1)}\{ e^{-cs} F(s) \} \to u_c(t)f(t-c)$$

\subsection{Periodic Functions}
\paragraph{DEFINITION: } $f(t+T) = f(t) \forall t \in \mathbb{R}, T \in \mathbb{R} $

\paragraph{WINDOW FUNCTION: } $f_T(t) = f(t)[1-u_T(t)$. It corresponds the first period then zeros everywhere else.
The laplace transform is understandably simple $$\mathcal{L}\{f_T(t)\} \to \int_0^T e^{-st}f(t)dt$$

\paragraph{Laplace Transform of Periodic Functions: } $$F(s) = \frac{F_T(s)}{1-e^{-sT}}$$

\section{Discontinuous Forcing Functions}
$ay'' + by' + cy = g(t)$ still has the same process for solving with Laplace transforms as before, even with a discontinuous forcing function.

\subsection{Impulse Functions}
These functions show a forcing function that is zero everywhere other than in $[t, t+\epsilon]$, and is very large in the non-zero range.
$$I(\epsilon) = \int_{t_0}^{t_0+\epsilon} g(t) dt$$
After the forcing, the momentum of the system is $I$. We define $\delta_{\epsilon}(t) = \frac{u_0(t)-u_{\epsilon}(t)}{\epsilon}$ so that
$$g(t) = I_0\delta_{\epsilon}(t)$$

As $\epsilon \to 0$, we get instantaneous \textbf{unit impulse function} $\delta(t)$. The properties are as follows:
\begin{itemize}
\item $\delta(t-t_0) = \lim_{\epsilon \to 0} \delta_{\epsilon}(t-t_0)$
\item $int_a^b f(t)\delta(t-t_0) dt = f(t_0)$
\item $\mathcal{L}\{\delta(t-t_0)\} \to e^{-st_0}$
\item $\mathcal{L}\{\delta(t)\} \to 1$
\item $\delta(t-t_0) = \frac{d}{dt} u(t-t_0)$
\end{itemize}

\section{Convolution Integrals}
\paragraph{Definition: } $$f * g = h(t) = \int_0^t f(t-\tau)g(\tau)d\tau$$

\subsection{Convolution Properties}
\begin{enumerate}
\item $f*g = g*f$
\item $f*(g_1 + g_2) = f*g_1 + f*g_2$
\item $(f*g)*h = f*(g*h)$
\item $f*0 = 0$
\end{enumerate}

\subsection{Convolution Theorem}
Let $F(s) = \mathcal{L}\{f(t)\}$, $G(s) = \mathcal{L}\{g(t)\}$. If $$H(s) = F(s)G(s) = \mathcal{L}\{h(t)\}$$
then $$h(t) = f*g = \int_0^t f(t-\tau)g(\tau) d\tau$$

\subsection{Free and Forced Response}
If we try solving $ay'' + by' + cy = g(t)$ with $y_0, y_1=y'(0)$ we find that the laplace transformed solution is:
$$
Y(s) = H(s)[(as+b)y_0 + ay_1]+H(s)G(s)
$$
where $H(s) = \frac{1}{Z(s)} = \frac{1}{as^2 + bs + c}$. Then, in the time domain, we get:
$$
y(t) = \mathcal{L}^{-1}\{H(s)[(as+b)y_0 + ay_1]\} + \int_0^1 h(t-\tau)g(\tau)d\tau
$$

On the right, the \textbf{first term} is the solution to $ay'' + by' + cy = 0$. It is the \textbf{FREE RESPONSE}.

Meanwhile, the second term is the \textbf{forced response}.

\paragraph{In summary: } Total response = free + forced
$$Y(s) = H(s)[(as+b)y_0 + ay_1] + H(s)G(s)$$
$$y(t) = \alpha_1y_1(t) + \alpha_2y_2(t) + \int_0^t h(t-\tau)g(\tau)d\tau$$

\paragraph{Transfer function: } Function describing ratio of forced response to free response. In this case, it is $H(t)$. It has
all the characteristics of the system.

$h(t)$ is also called the \textbf{impulse response} because it is the response when an impulse is used so that $G(s) = 1$.

$y_g(t) = h(t)*g(t)$ is, therefore, the forced response in the time domain while $Y_g(s) = H(s)*G(s)$ is the forced response in the s-domain.

\paragraph{How to get system response in the time domain: }
\begin{enumerate}
\item Determine $H(s)$
\item Find $G(s)$
\item Construct $Y_g(s) = H(s)G(s)$
\item Convert to the time-domain with $\mathcal{L}^{-1}(Y_g(s))$
\end{enumerate}

\end{document}


\documentclass[a4paper]{article}

\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}

\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{xcolor}
\usepackage{amsthm}
\usepackage[mathcal]{euscript}

\usepackage{url}

\newcommand{\Hcal}{\mathcal{H}}
\newcommand{\real}{\mathbb{R}}
\newcommand{\interior}[1]{%
  {\kern0pt#1}^{\mathrm{o}}%
}
\newcommand{\Tcal}{\mathcal{T}}
\newcommand{\Lcal}{\mathcal{L}}

\title{Notes on Hahn-Banach}
\author{Nazarov Ivan}

\date{\today}

\begin{document}
\maketitle

Consider a Banach space $(\Hcal, \|\cdot\|)$ with the norm $\Tcal^{\|\cdot\|}_\Hcal$
topology on it.

\paragraph{Linear spans} % (fold)
\label{par:linear_spans}

Let $M \subset \Hcal$ be a linear subspace of $(\Hcal, \|\cdot\|)$ and $x \notin M$.
Observe that since $M$ is a linear space $0\in M$, whence $x \neq 0$. Define $M_1$
as the linear span of $M_0 \cup \{x\}$: $M_1 = \{\alpha x + y \colon \alpha \in \real
\,,\, y \in M_0\}$.

For any $z\in M_1$ we have $z = \alpha x + y$ for some $\alpha\in \real$ and $y\in M$.
Suppose there are $(\alpha_1, y_1)$ and $(\alpha_2, y_2)$ such that
\begin{equation*}
  \alpha_1 x + y_1 = \alpha_2 x + y_2
  \Leftrightarrow
    y_1 - y_2 = (\alpha_2 - \alpha_1) x
    \,,
\end{equation*}
If $\alpha_1 = \alpha_2$, then $y_1 - y_2 = 0$ and thus $y_1 = y_2$. If $\alpha_1
\neq \alpha_2$, then we proceed by contradiction: dividing both sides by the difference
yields $x = -\tfrac{y_1 - y_2}{\alpha_1 -\alpha_2}$, which implies $x\in M$, since
$M$ is a linear subspace. Hence for every $z\in M_1$ there is a unique decomposition
$z = \alpha x + y$ with $\alpha\in\real$ and $y\in M$.

We can define a pair of maps $\alpha\colon M_1\to \real$ and $y\colon M_1 \to M$,
that are given, respectively, by the $\alpha\in\real$ and $y\in M$ components of
the decomposition $z = \alpha x + y$ for each $z\in M_1$. These maps are injections,
since $\alpha(z_1) = \alpha(z_2)$ and $y(z_1) = y(z_2)$ imply
\begin{equation*}
  z_1 = \alpha(z_1) x + y(z_1)
    = \alpha(z_2) x + y(z_2)
    = z_2
    \,.
\end{equation*}
By definition of $M_1$ the maps are also surjective. It is noteworthy that they are
also linear: for $u, v\in M_1$ and $\beta \in \real$
\begin{align*}
  \alpha(u + \beta v) x + y(u + \beta v)
    &= u + \beta v
    = u + \beta (v)
    \\
    &= \alpha(u) x + y(u)
      + \beta \bigl(\alpha(v) x + y(v) \bigr)
    \\
    &= \bigl( \alpha(u) + \beta\alpha(v) \bigr) x
    + \bigl( y(u) + \beta y(v) \bigr)
    \,,
\end{align*}
from which uniqueness of the decomposition implies linearity. Finally $M\subset M_1$,
since $y = 0 x + y \in M_1$ for any $y\in M$.

Therefore, the linear subspace $M_1$ is homomorphic to $\real \times M$ -- a linear
product space with addition and scaling over the field $\real$ defined naturally.
One can also easily make it into a normed space, by additively composing the norms
on $\real$ and $M$. And if $M$ was complete, then $M_1$ would also be complete with
respect to the newly defined norm.

% paragraph linear_spans (end)

\paragraph{Simple one dim extension} % (fold)
\label{par:simple_one_dim_extension}

Let $M_0\subseteq \Hcal$ be a linear subspace of $(\Hcal, \|\cdot\|)$, and $l$ be
a linear functional $M_0\to \real$.

For a given arbitrary $x_1\notin M_0$ let $M_1 = \{\alpha x_1 + y \colon \alpha \in
\real\,,\, y \in M_0\}$. Hence there exist linear bijections $\alpha\colon M_1\to
\real$ and $y_0\colon M_1\to M_0$ such that $z = \alpha(z) x_1 + y_0(z)$ for all
$z\in M_1$.

Define $\tilde{l}\colon M_1 \to \real$ as follows: for some $c \in \real$ for any
$z \in M_1$ put
\begin{equation*}
  \tilde{l}(z; c)
    = \tilde{l}\bigl(\alpha(z) x_1 + y_0(z) \bigr)
    = \alpha(z) \tilde{l}(x_1) + l(y_0(z))
    = \alpha(z) c + l(y_0(z))
    \,.
\end{equation*}
Note that $\tilde{l}(\cdot; c)$ inherits linearity from the maps $\alpha$ and $y_0$.
Also $\alpha(z) = 0$ and $y_0(z) = z$ on $z\in M_0$ imply that $\tilde{l}(\cdot; c)%
\big\vert_{M_0} = l$.

Thus $\tilde{l}(\cdot; c)$ for any $c\in \real$ is an extension of $l$ from $M_0
\subset \Hcal$ to the linear span of $M_0 \cup \{x_1\}$ for an $x_1\notin M_0$.

Note that the map $c\mapsto \tilde{l}(\cdot; c)$ from $\real$ to $L_{M_0}(M_1, \real)$,
i.e. the set of all extensions of $l$ to $M_1$, is a bijection. Indeed, consider some
other linear functional $f\colon M_1 \to \real$ with $f\big\vert_{M_0} = l$. Then
$\tilde{l}\bigl(\cdot; f(x_1)\bigr) = f$, since
\begin{equation*}
  f\bigl(\alpha x_1 + y \bigr)
    = \alpha f(x_1) + f(y)
    = \alpha f(x_1) + l(y)
    = \tilde{l}\bigl(\alpha x_1 + y; f(x_1)\bigr)
    \,.
\end{equation*}
For injectivity suppose $\tilde{l}(\cdot; c_1) = \tilde{l}(\cdot; c_2)$. This implies
that $\alpha (c_1 - c_2) + l(y - y) = 0$ for any $\alpha \in \real$ and $y\in M_0$.
In particular for $\alpha = c_1 - c_2$ we get $(c_1 - c_2)^2 = 0$, whence $c_1 = c_2$.

% paragraph simple_one_dim_extension (end)

\paragraph{Extension with continuity} % (fold)
\label{par:extension_with_continuity}

Let $M_0 \subseteq \Hcal$ be a linear subspace and $l\in M_0^*$ be a continuous
linear functional on $M_0$ with $\|l\|_{M_0^*} = 1$, where
\begin{equation*}
  \|l\|_{M_0^*}
    = \sup\bigl\{\lvert l(z) \rvert\colon
        z\in M_0\,,
        \,\|z\|\leq 1
      \bigr\}
    \,.
\end{equation*}
Note that if $\|l\|_{M_0^*} \neq 1$ is nonzero and finite then we define $\hat{l}$
as $\tfrac1{\|l\|_{M_0^*}} l$, and work with that.

Pick and $x_0\notin M_0$ and let $\tilde{l}(\cdot; c)$ be an extension of $l$ from
$M_0\subset \Hcal$ to $M_1 = \{\alpha x + y \colon \alpha \in \real\,,\, y \in M_0\}$.
Since the any such extension is uniquely identified by $c$, i.e. its value on $x_1$,
by carefully choosing $c$ we shall try to make $\tilde{l} = \tilde{l}(\cdot; c)$ a
bounded linear functional.

First, note that the definition of the norm of a linear functional (operator) $\phi
\colon M\to \real$ implies that $\tfrac{\lvert \phi(z) \rvert}{\|z\|} \leq \|\phi\|_{M^*}$
for any $z\in M$ with $z \neq 0$, and $\|\phi\|_{M^*}$ is the smallest $K\in [0,
+\infty]$ such that $\lvert \phi(z) \rvert \leq K \|z\|$ for all $z\in M$.

Take $\tilde{l}$: even if its norm is $+\infty$ we still have $\lvert \tilde{l}(z)\rvert
\leq \|\tilde{l}\|_{M_1^*} \|z\|$ for all $z\in M_1$. In particular, for $z\in M_0$
we get $\lvert l(z)\rvert = \lvert \tilde{l}(z)\rvert \leq \|\tilde{l}\| \|z\|$,
whence $\|l\|_{M_0^*} \leq \|\tilde{l}\|_{M_1^*}$.

To get an upper bound on $\|\tilde{l}\|_{M_1^*}$, note the equivalence
\begin{align*}
  &\lvert \tilde{l}(z) \rvert \leq \|z\|
      \quad \forall z \in M_1
  \\ &\Leftrightarrow
    - \|\alpha x_1 + y\|
      \leq \alpha c + l(y)
      \leq \|\alpha x_1 + y\|
      \quad \forall \alpha \in \real\,, y \in M_0
  \\ &\Leftrightarrow
    - \bigl\|x_1 + \tfrac{y}\alpha\bigr\|
      \leq c + l\bigl(\tfrac{y}\alpha\bigr) \leq
        \bigl\|x_1 + \tfrac{y}\alpha\bigr\|
      \quad \forall \alpha \neq 0\,, y \in M_0
  \\ &\Leftrightarrow
    - \|x_1 + y\| \leq c + l(y) \leq \|x_1 + y\|
      \quad \forall y \in M_0
    \,.
\end{align*}
The transition from $\alpha \in \real$ to $\alpha = 0$, is valid, since $\|l\|_{M_0^*}
= 1$, $z = \alpha x_1 + y \in M_0$ whenever $\alpha = 0$, and $\tilde{l}\big\vert_{M_0}
= l$. Dividing by $\alpha \neq 0$ effectively keeps the inequality signs intact, which
can be seen by considering cases $\alpha > 0$ and $\alpha < 0$ separately. Final
equivalence, stems $M_0$'s being a linear space.

We shall show that $\|\tilde{l}\|_{M_1^*} \leq 1$ if we could find such $c$ that for
every $y \in M_0$
\begin{equation*}
  - \|x_1 + y\| - l(y) \leq c \leq \|x_1 + y\| - l(y)
    \,.
\end{equation*}
For any $y_1, y_2 \in M_0$ we have
\begin{align*}
  l(y_1) - l(y_2)
    % = l(y_1 - y_2)
    &\leq \|l\|_{M_0^*} \|y_1 - y_2\|
    % \leq \|l\|_{M_0^*} \|y_1 + x_1 - (y_2 + x_1)\|
    \leq \|y_1 + x_1 \| + \|x_1 + y_2\|
    \,, \\
  &\Leftrightarrow
  - \|x_1 + y_2\| - l(y_2)
    \leq \|y_1 + x_1 \| - l(y_1)
    \,.
\end{align*}
This implies that the left-hand side for any $y_2\in M_0$ is bounded by the right-%
hand side at any given $y_1\in M_0$, which means that for
\begin{equation*}
  c_-
    = \sup_{y_2\in M_0} - \|x_1 + y_2\| - l(y_2)
    \,\text{ and }\,
    c_+ = \inf_{y_1\in M_0} \|y_1 + x_1 \| - l(y_1)
    \,,
\end{equation*}
we have $c_- \leq c_+$ and that $[c_-, c_+]$ is nonempty. If $c\in [c_-, c_+]$, then
\begin{equation*}
  - \|x_1 + y\| - l(y) \leq c_- \leq c \leq c_+ \leq \|x_1 + y\| - l(y)
    \,,
\end{equation*}
for every $y \in M_0$. Therefore $1 = \|l\|_{M_0^*} \leq \|\tilde{l}\|_{M_1^*} \leq 1$,
and $\tilde{l} = \tilde{l}(\cdot; c)$ is a bounded linear functional on $M_0$ with
$\tilde{l}\big\vert_{M_0} = l$ and $\|l\|_{M_0^*}=\|\tilde{l}\|_{M_1^*}$.

% paragraph extension_with_continuity (end)

\paragraph{Hahn-Banach extension discussion} % (fold)
\label{par:hahn_banach_extension_discussion}

Note that $M_0$ must be a linear space, but is not required to be a closed subset
of $\Hcal$. If it is note closed, then it is possible to extend $l$ to $[M_0]$ by
setting $\tilde{l}$ to the limits of Cauchy sequences in $M_0$, provided the linear
functional (operator) $l$ is bounded.

If the space $\Hcal$ is finite dimensional, then applying the one-dim extension
until we run out of the dimensions will yield a continuous linear functional $\tilde{l}$
that extends $l$ from $M_0$ to $\Hcal$.

If the space is separable, then there is a countable subset $X$, which is everywhere
dense in $\Hcal$. Then, assuming that $M_0$ is a closed subspace, we can construct
a sequence of linear bounded extensions $\tilde{l}_n$ of $l$ from $M_0$ to the span
$M_n$ of $M_0 \cup \{x_1,\,\ldots, \,x_n\}$ for each $n\geq 1$, where $(x_n)_{n\geq1}$
enumerates the countable $X$. We get $M_n\subset M_{n+1}$ and $\tilde{l}_{n+1}\big\vert_{M_n}
= \tilde{l}_{n}$. Then $M_\infty = \bigcup_{n\geq1} M_n$ is a linear subspace, that
is also dense in $\Hcal$, since it contains $X$. At the same time $\tilde{l}_\infty$,
the {\bf union} of linear functionals $(\tilde{l}_n)_{n\geq1}$, is a linear functional
with $\tilde{l}_\infty\big\vert_{M_n} = \tilde{l}_{n}$. Thus $\tilde{l}_\infty$ can
be extended from $M_\infty$ to $[M_\infty]$ via the same Cauchy sequence limit machinery.

The core idea in the sketch of a proof resembles the main argument in the proof
of the Hahn-Banach extension in the case of a general $\Hcal$ via Zorn's lemma. The
Zornified version first claims that the family of all pairs $(M, f)$, where $M$ is
a linear subspace of $\Hcal$ and $f\in M^*$ is an extension of $l$ to $M$, has a
partially ordered set structure. In this family every totally ordered subset has
an upper bound, which by Zorn's lemma implies that the whole family has at least
one maximal element.

% paragraph hahn_banach_extension_discussion (end)

\paragraph{Hahn-Banach point-set separation} % (fold)
\label{par:hahn_banach_point_set_separation}

Let $C\subset \Hcal$ be a closed, convex subset with $0\in \interior{C}$. Then for
any $x_0\notin C$ there is $l_{x_0} \in \Hcal^*$, a continuous linear functional on
$\Hcal$, such that $l_{x_0}(x) \leq 1 < l_{x_0}(x_0)$ for $x\in C$.

Consider a linear subspace $\Lcal_{x_0} = \{t x_0\colon t\in \real\}$. $\Lcal_{x_0}$
is nontrivial, since $0 \in C$ and $x_0\notin C$. Observe that for a given $x\in \Lcal_{x_0}$
there is a unique $t_x \in \real$ such that $x = t_x x_0$ (if $(t_1 - t_2) x_0 = 0$,
then $t_1 = t_2$ because $x_0 \neq 0$).

Uniqueness implies the existence of a linear bijection $\tau\colon \Lcal_{x_0} \to
\real$ such that $z = \tau(z) x_0$ for any $z\in \Lcal_{x_0}$. Indeed, if $x, y\in
\Lcal_{x_0}$, then $\tau(x) = \tau(y)$ implies that $x = \tau(x) x_0 = \tau(y) x_0
= y$, and for any $\alpha\in \real$
\begin{equation*}
  \bigl( \tau(x) x_0 \bigr) + \bigl( \tau(y) x_0 \bigr) \alpha
    % structure of $x$ and $y$
    = x + y \alpha
    % linearity of $\Lcal_{x_0}$
    = \tau(x + y \alpha) x_0
    % uniquenes of x\mapsto t_x
  \,.
\end{equation*}

Let $p$ be the Minkowski functional of the set $C$. For this $C$ the functional $p$
is finte-valued, subadditive, positively homogenous, and continuous. The functional
$p$ is also sublinear: $0 = p(0) \leq p(t x) + p(-t x) = p(t x) - t p(x)$ for any
$t < 0$, and $t p(x) = p(t x)$ for $t \geq 0$.

Since $x_0\notin C$ and $C$ is closed, we must have $x_0\notin \{p\leq 1\}$, whence
$p(x_0) > 1$. Let $f(x) = \tau(x) p(x_0)$ for any $x \in \Lcal_{x_0}$. On $\Lcal_{x_0}$
we have $f\leq p$, since $\tau(x) p(x_0) \leq p\bigl(\tau(x) x_0\bigr) = p(x)$ by
sublinearity of $p$. At the same time $f$ inherits linearity from $\tau$.

Now, for such $C$ the Minkowski functional is bounded, i.e. there exists $M \in (0,
+\infty)$, such that $p(x) \leq M \|x\|$ for any $x\in \Hcal$. Therefore if $f\leq
p$, then by linearity of $f$ and we have
\begin{equation*}
  - M \|x\|
    \leq - p(-x)
    \leq -l(- x)
    = l(x)
    \leq p(x)
    \leq M \|x\|
    \,.
\end{equation*}
Hence the {\it Hahn-Banach Extension Theorem} (applied to $\tfrac1M f$, and then
scale back) there must be linear extension $l_{x_0}$ of $f$ from $\Lcal_{x_0}$ onto
$\Hcal$ with $l_{x_0}\big\vert_{\Lcal_{x_0}} = f$ and $l_{x_0} \leq p$ everywhere
on $\Hcal$. This extension is a bounded linear functional by an earlier result, and
satisfies $l_{x_0}(x) \leq p(x) \leq 1$ for any $x\in C$ and $1 < p(x_0) = \tau(x_0) p(x_0)
= f(x_0) = l_{x_0}(x_0)$.

% paragraph hahn_banach_point_set_separation (end)

\end{document}


\section{Data Structures}
\begin{center}
\begin{longtable}{|l|p{3cm}|l|l|p{3cm}|}
\hline
\textbf{Name} & \textbf{Insertion} & \textbf{Deletion} & \textbf{Index} & \textbf{search} \\
\hline

Array List & O(n) & O(n) & O(1) & O(n) \\

\hline

Singly Linked List & O(1) & O(n) & O(n) & O(n) \\

\hline

Hash Set/Map & O(1) & O(1) & n/a & O(1) \\

\hline

Random BST & O(log(n)) & O(log(n)) & O(log(n)) & O(log(n)) \\

\hline

Heap & O(log(n)) & O(log(n)) to remove max/min elem & n/a & n/a \\

\hline

Trie & O(k) where k is the length of the word & O(k) & n/a &  O(k) \\

\hline

Bloom filter & O(k) where k is the number of hash functions & n/a & n/a & O(k) to tell us that it is `definitely not in the set'\\

\hline

\end{longtable}
\end{center}


\problemname{Guessing Camels}

\illustration{.35}{jjt-cropped-scaled.jpg}{Jaap, Jan and Thijs on camels (in some order). Photo by Tobias
Werth, cc by-sa.}%
\noindent
Jaap, Jan, and Thijs are on a trip to the desert after having attended the ACM ICPC World Finals 2015 in Morocco.  The trip included a camel ride, and after returning from the ride, their guide invited them to a big
camel race in the evening. The camels they rode will also participate and it is
customary to bet on the results of the race.

One of the most interesting bets involves guessing the complete order in which the camels will
finish the race. This bet offers the biggest return on your money, since it is also the one that is
the hardest to get right.

Jaap, Jan, and Thijs have already placed their bets, but the race will not start until an hour from now, so they are
getting bored. They started wondering how many pairs of camels they have put in the same order. If
camel~$c$ is before camel~$d$ on Jaap's, Jan's and Thijs' bet, it means that all three of them put
$c$ and $d$ in the same order. Can you help them to calculate the number of pairs of camels for which
this happened?

\section*{Input}

The input consists of:
\begin{itemize}
\item one line with an integer $n$ ($2\leq n \leq 200\,000$), the number of
camels;
\item one line with $n$ integers $a_1, \ldots, a_n$ ($1 \le a_i \le n$ for all $i$), Jaap's bet.
  Here $a_1$ is the camel in the first position of Jaap's bet, $a_2$ is the camel in the second position, and so on;
\item one line with Jan's bet, in the same format as Jaap's bet;
\item one line with Thijs' bet, in the same format as Jaap's bet.
\end{itemize}

The camels are numbered $1, \dots, n$.  Each camel appears exactly
once in each bet.

\section*{Output}

Output the number of pairs of camels that appear in the same order in all $3$~bets.


\section{World Final 2004}
\subsection{A. Carl the Ant}
\raggedbottom\lstinputlisting[style=cpp]{code/2004/carl_the_ant.cpp}
\hrulefill
\subsection{B. Heliport}
\raggedbottom\lstinputlisting[style=cpp]{code/2004/heliport.cpp}
\hrulefill
\subsection{C. Image is Everything}
\raggedbottom\lstinputlisting[style=cpp]{code/2004/image_is_everything.cpp}
\hrulefill
\subsection{D. Insecure in Prague}
\raggedbottom\lstinputlisting[style=cpp]{code/2004/insecure_in_prague.cpp}
\hrulefill
\subsection{E. Intersecting Dates}
\raggedbottom\lstinputlisting[style=cpp]{code/2004/intersecting_dates.cpp}
\hrulefill
\subsection{F. Merging Maps}
\raggedbottom\lstinputlisting[style=cpp]{code/2004/merging_maps.cpp}
\hrulefill
\subsection{G. Navigation}
\raggedbottom\lstinputlisting[style=cpp]{code/2004/navigation.cpp}
\hrulefill
\subsection{H. Tree-lined Street}
\raggedbottom\lstinputlisting[style=cpp]{code/2004/tree_lined_street.cpp}
\hrulefill
\subsection{I. Suspense!}
\raggedbottom\lstinputlisting[style=cpp]{code/2004/suspense.cpp}
\hrulefill
\subsection{J. Air Traffic Control}
\raggedbottom\lstinputlisting[style=cpp]{code/2004/air_traffic_control.cpp}
\hrulefill

\section{World Final 2005}
\subsection{A. Eyeball Benders}
\raggedbottom\lstinputlisting[style=cpp]{code/2005/eyeball_benders.cpp}
\hrulefill
\subsection{B. Simplified GSM Network}
\raggedbottom\lstinputlisting[style=cpp]{code/2005/simplified_gsm_network.cpp}
\hrulefill
\subsection{C. The Traveling Judges Problem}
\raggedbottom\lstinputlisting[style=cpp]{code/2005/the_traveling_judges_problem.cpp}
\hrulefill
\subsection{D. cNteSahruPfefrlefe}
\raggedbottom\lstinputlisting[style=cpp]{code/2005/c_nte_sahru_pfefrlefe.cpp}
\hrulefill
\subsection{E. Lots of Sunlight}
\raggedbottom\lstinputlisting[style=cpp]{code/2005/lots_of_sunlight.cpp}
\hrulefill
\subsection{F. Crossing Streets}
\raggedbottom\lstinputlisting[style=cpp]{code/2005/crossing_streets.cpp}
\hrulefill
\subsection{G. Tiling the Plane}
\raggedbottom\lstinputlisting[style=cpp]{code/2005/tiling_the_plane.cpp}
\hrulefill
\subsection{H. The Great Wall Game}
\raggedbottom\lstinputlisting[style=cpp]{code/2005/the_great_wall_game.cpp}
\hrulefill
\subsection{I. Workshops}
\raggedbottom\lstinputlisting[style=cpp]{code/2005/workshops.cpp}
\hrulefill
\subsection{J. Zones}
\raggedbottom\lstinputlisting[style=cpp]{code/2005/zones.cpp}
\hrulefill

\section{World Final 2005}
\subsection{A. Low Cost Air Travel}
\raggedbottom\lstinputlisting[style=cpp]{code/2006/low_cost_air_travel.cpp}
\hrulefill
\subsection{B. Remember the A La Mode!}
\raggedbottom\lstinputlisting[style=cpp]{code/2006/remember_the_a_la_made.cpp}
\hrulefill
\subsection{C. Ars Longa}
\raggedbottom\lstinputlisting[style=cpp]{code/2006/ars_longa.cpp}
\hrulefill
\subsection{D. Bipartite Numbers}
\raggedbottom\lstinputlisting[style=cpp]{code/2006/bipartite_numbers.cpp}
\hrulefill
\subsection{E. Bit Compressor}
\raggedbottom\lstinputlisting[style=cpp]{code/2006/bit_compressor.cpp}
\hrulefill
\subsection{F. Building a Clock}
\raggedbottom\lstinputlisting[style=cpp]{code/2006/building_a_clock.cpp}
\hrulefill
\subsection{G. Pilgrimage}
\raggedbottom\lstinputlisting[style=cpp]{code/2006/pilgrimage.cpp}
\hrulefill
\subsection{H. Pockets}
\raggedbottom\lstinputlisting[style=cpp]{code/2006/pockets.cpp}
\hrulefill
\subsection{I. Degrees of Separation}
\raggedbottom\lstinputlisting[style=cpp]{code/2006/degree_of_separation.cpp}
\hrulefill
\subsection{J. Routing}
\raggedbottom\lstinputlisting[style=cpp]{code/2006/routing.cpp}
\hrulefill


\documentclass[12pt]{article}
\usepackage[dvips]{graphicx}
\topmargin -40pt
\headsep 30pt
\evensidemargin 0pt
\oddsidemargin 0pt
\textwidth 490pt
\textheight 650pt

\def\beq{\begin{equation}}
\def\eeq{\end{equation}}

\begin{document}

%\newpage
\section*{Verdict Hexahedral Metrics:}

\subsection*{Variable Definitions:}

\begin{figure}[htb]
  \begin{center}
    \includegraphics[height=2.0in]{hex.eps}
    \caption{Vertices of Hexahedron}
    \label{fig:blank}
  \end{center}
\end{figure}

\subsubsection*{Hex Vertices:}
\begin{center}
$\vec P_0 = (x_0, y_0, z_0)$
\end{center}

\begin{center}
$\vec P_1 = (x_1, y_1, z_1)$
\end{center}

\begin{center}
$\vec P_2 = (x_2, y_2, z_2)$
\end{center}

\begin{center}
$\vec P_3 = (x_3, y_3, z_3)$
\end{center}

\begin{center}
$\vec P_4 = (x_4, y_4, z_4)$
\end{center}

\begin{center}
$\vec P_5 = (x_5, y_5, z_5)$
\end{center}

\begin{center}
$\vec P_6 = (x_6, y_6, z_6)$
\end{center}

\begin{center}
$\vec P_7 = (x_7, y_7, z_7)$
\end{center}

\subsubsection*{Hex Edges:}

\begin{center}
$\vec L_0 = \vec P_1 - \vec P_0 $
\end{center}

\begin{center}
$\vec L_1 = \vec P_2 - \vec P_1 $
\end{center}

\begin{center}
$\vec L_2 = \vec P_3 - \vec P_2 $
\end{center}

\begin{center}
$\vec L_3 = \vec P_3 - \vec P_0 $
\end{center}

\begin{center}
$\vec L_4 = \vec P_4 - \vec P_0 $
\end{center}

\begin{center}
$\vec L_5 = \vec P_5 - \vec P_1 $
\end{center}

\begin{center}
$\vec L_6 = \vec P_6 - \vec P_2 $
\end{center}

\begin{center}
$\vec L_7 = \vec P_7 - \vec P_3 $
\end{center}

\begin{center}
$\vec L_8 = \vec P_5 - \vec P_4 $
\end{center}

\begin{center}
$\vec L_9 = \vec P_6 - \vec P_5 $
\end{center}

\begin{center}
$\vec L_{10} = \vec P_7 - \vec P_6 $
\end{center}

\begin{center}
$\vec L_{11} = \vec P_7 - \vec P_4 $
\end{center}

\begin{center}
$L_{min} = MIN\left(| \vec L_0 |,| \vec L_1|, .... |\vec L_{11}| \right) $
\end{center}

\subsubsection*{Hex Diagonals:}

\begin{center}
$\vec D_0 = \vec P_6 - \vec P_0 $
\end{center}

\begin{center}
$\vec D_1 = \vec P_7 - \vec P_1 $
\end{center}

\begin{center}
$\vec D_2 = \vec P_4 - \vec P_2 $
\end{center}

\begin{center}
$\vec D_3 = \vec P_5 - \vec P_3 $
\end{center}

\begin{center}
$D_{min} = MIN\left( | \vec D_0|, |\vec D_1|, |\vec D_2|, |\vec D_3| \right) $
\end{center}
\begin{center}
$D_{max} = MAX\left( | \vec D_0|, |\vec D_1|, |\vec D_2|, |\vec D_3| \right) $
\end{center}

\subsubsection*{Principal Axes:}

\begin{center}
$\vec X_1 = -\vec P_0 - \vec P_3 - \vec P_4 - \vec P_7
            +\vec P_1 + \vec P_2 + \vec P_5 + \vec P_6$
\end{center}

\begin{center}
$\vec X_2 = -\vec P_0 - \vec P_1 - \vec P_4 - \vec P_5
            +\vec P_2 + \vec P_3 + \vec P_6 + \vec P_7$
\end{center}

\begin{center}
$\vec X_3 = -\vec P_0 - \vec P_1 - \vec P_2 - \vec P_3
            +\vec P_4 + \vec P_5 + \vec P_6 + \vec P_7$
\end{center}

\subsubsection*{Cross Derivatives:}

\begin{center}
$\vec X_{23} = \vec P_0 + \vec P_1 + \vec P_6 + \vec P_7
          - \vec P_2 - \vec P_3 - \vec P_4 - \vec P_5$
\end{center}

\begin{center}
$\vec X_{13} = \vec P_0 + \vec P_3 + \vec P_5 + \vec P_6
          - \vec P_1 - \vec P_2 - \vec P_4 - \vec P_7$
\end{center}

\begin{center}
$\vec X_{12}= \vec P_0 + \vec P_2 + \vec P_4 + \vec P_6
          - \vec P_1 - \vec P_3 - \vec P_5 - \vec P_7$
\end{center}

\subsubsection*{Hex Matrices:}

\begin{center}
$A_0 = [ \vec L_0, \vec L_3, \vec L_4 ]$
\end{center}

where $\vec L_0$, $\vec L_3$, and $\vec L_4$ form the columns of $A_0$, etc.

\begin{center}
$A_1 = [ \vec L_1, -\vec L_0, \vec L_5 ]$
\end{center}

\begin{center}
$A_2 = [ \vec L_2, -\vec L_1, \vec L_6 ]$
\end{center}

\begin{center}
$A_3 = [ -\vec L_3, -\vec L_2, \vec L_7 ]$
\end{center}

\begin{center}
$A_4 = [ \vec L_{11}, \vec L_8, -\vec L_4 ]$
\end{center}

\begin{center}
$A_5 = [ -\vec L_8, \vec L_9, -\vec L_5 ]$
\end{center}

\begin{center}
$A_6 = [ -\vec L_9, \vec L_{10}, -\vec L_6 ]$
\end{center}

\begin{center}
$A_7 = [ -\vec L_{10}, -\vec L_{11}, -\vec L_7 ]$
\end{center}

\begin{center}
$A_8 = [ \vec X_1, \vec X_2, \vec X_3 ]$
\end{center}

\subsection*{Matrix Operations:}

Let $A$ be a $3 \times 3$ matrix defined by the column vectors $\vec v_1, \vec v_2, \vec v_3,$ i.e., $A = [\vec v_1, \vec v_2, \vec v_3]$.

\begin{flushleft}
Then,
\end{flushleft}

\begin{displaymath}
|A|^2 = |\vec v_1|^2 + |\vec v_2|^2 + |\vec v_3|^2
\end{displaymath}

\begin{displaymath}
|adj(A)|^2 = |\vec v_1 \times \vec v_2|^2 +
             |\vec v_2 \times \vec v_3|^2 +
             |\vec v_3 \times \vec v_1|^2
\end{displaymath}

\begin{displaymath}
\alpha = determinant(A) = \vec v_1 \cdot (\vec v_2 \times \vec v_3 )
\end{displaymath}

\begin{displaymath}
\alpha_k = determinant(A_k) ~ where ~ k=0,1,...,8
\end{displaymath}

\begin{displaymath}
\hat{A} = \left[ \frac {\vec v_1} {|v_1|},
                 \frac {\vec v_2} {|v_2|},
                 \frac {\vec v_3} {|v_3|} \right]
\end{displaymath}

\begin{displaymath}
\hat \alpha_k = determinant(\hat A_k) ~ where ~ k=0,1,...,8
\end{displaymath}

\subsection*{Overflow:}

\begin{flushleft}
The metrics below are all checked for overflow like so:
\end{flushleft}

\begin{flushleft}
\quad ${IF \quad metric\_value > 0;  \quad metric\_value = MIN( metric\_value, DBL\_MAX )}$
\end{flushleft}

\begin{flushleft}
\quad ${ELSE \quad metric\_value = MAX( metric\_value, -DBL\_MAX )}$
\end{flushleft}

\subsection*{Metric Ranges:}

\begin{flushleft}
Acceptable Range: Well-behaved elements will have metrics in this range.
\end{flushleft}
\begin{flushleft}
Normal Range:     All elements except those with degeneracies will have  \newline
                  metrics in this range.
\end{flushleft}
\begin{flushleft}
Full Range:       All elements including degenerate ones will have metrics \newline
                  in this range.
\end{flushleft}

%---------------------------Aspect Ratio---------------------------
\subsection*{Hex Aspect Ratio:}

\begin{displaymath}
aspect_{12} = \frac{ MAX \left( \left| \vec X_1 \right|,  \left| \vec X_2 \right|  \right) }
                    { MIN \left( \left| \vec X_1 \right|,  \left| \vec X_2 \right|  \right) }
\end{displaymath}

\begin{displaymath}
aspect_{13} = \frac{ MAX \left( \left| \vec X_1 \right|,  \left| \vec X_3 \right|  \right) }
                    { MIN \left( \left| \vec X_1 \right|,  \left| \vec X_3 \right|  \right) }
\end{displaymath}

\begin{displaymath}
aspect_{23} = \frac{ MAX \left( \left| \vec X_2 \right|,  \left| \vec X_3 \right|  \right) }
                    { MIN \left( \left| \vec X_2 \right|,  \left| \vec X_3 \right|  \right) }
\end{displaymath}

\begin{center}
$aspect = MAX\left( aspect_{12}, aspect_{13}, aspect_{23} \right)$
\end{center}

\begin{tabular}{lll}
& Metric Description:  & Maximum pairwise length ratio between principle axes. \\
& Dimension:           & $L^0$       \\
& Acceptable Range:    & 1 to 1.3               \\
& Normal Range:        & 1 to $DBL\_MAX$                      \\
& Full Range:          & 1 to $DBL\_MAX$ \\
& Value of Metric on Unit Cube:    & 1.0 \\
& Note:                & If $|\vec X_1 |$ or $|\vec X_2 |$ or $|\vec X_3| < DBL\_MIN$, \\
&                      & $aspect = DBL\_MAX$ \\
& Reference:           & Adapted from \cite{one} \\
\end{tabular}

%---------------------------Condition---------------------------
\subsection*{Hex Condition:}

\begin{displaymath}
\kappa(A) = |A| |A^{-1}| = \frac {|A| |adj(A)|} {\alpha}
\end{displaymath}

\begin{center}
$condition~number = \frac {1}{3} MAX( \kappa(A_0), \kappa(A_1), .....\kappa(A_8))$
\end{center}

\begin{tabular}{lll}
& Metric Description:  & Maximum condition number of the Jacobian matrix at  \\
&                      & 8 corners and center.\\
& Dimension:           & $L^0$       \\
& Acceptable Range:    & 1 to 3               \\
& Normal Range:        & 1 to $DBL\_MAX$                      \\
& Full Range:          & 1 to $DBL\_MAX$ \\
& Value of Metric on Unit Cube:    & 1.0 \\
& Note:                & If $\alpha_k \leq DBL\_MIN$, for any $k$, $condition = DBL\_MAX$\\
& Reference:           & \cite{three} \\
\end{tabular}

%---------------------------Diagonal---------------------------
\subsection*{Hex Diagonal:}

\begin{displaymath}
diagonal = \frac {D_{min}} { D_{max}}
\end{displaymath}

\begin{tabular}{lll}
& Metric Description:  & Ratio of minimum to maximum diagonal. \\
& Dimension:           & $L^0$                              \\
& Acceptable Range:    & 0.65 to 1  \\
& Normal Range:        & 0 to 1 \\
& Full Range:          & 0 to $DBL\_MAX$ \\
& Value of Metric on Unit Cube:    & 1.0 \\
& Note:                & If $D_{max} < DBL\_MIN$, $diagonal = DBL\_MAX $ \\
& Reference:           & Unknown \\
\end{tabular}

%---------------------------Distortion---------------------------
\subsection*{Hex Distortion:}

\begin{displaymath}
distortion = \frac{ |J|*master\textrm{-}volume} { true\textrm{-}volume }
\end{displaymath}

\begin{flushleft} Where, \end{flushleft}
$|J| = $ minimum determinant of the Jacobian computed over all Gauss points  \newline
of the element.  The location of the Gauss points is not documented. \newline
$master\textrm{-}volume =$ volume of the master hex, where the master hex's corner \newline
nodes are as follows: \newline

 $\vec P_0 = (-1, -1, 0)$,
 $\vec P_1 = (1, -1, 0)$,
 $\vec P_2 = (1, 1, 0)$,
 $\vec P_3 = (-1, 1, 0)$  \newline

 $\vec P_4 = (-1, -1, 1)$,
 $\vec P_5 = (1, -1, 1)$,
 $\vec P_6 = (1, 1, 1)$,
 $\vec P_7 = (-1, 1, 1)$   \newline

\begin{tabular}{lll}
& Metric Description:  & Minimum Gauss Point Jacobian divided by hex \\
&                      & volume, times master hex's volume. \\
& Dimension:           & $L^3$       \\
& Acceptable Range:    & 0.5 to 1 \\
& Normal Range:        & 0 to 1 \\
& Full Range:          & $-DBL\_MAX$ to $DBL\_MAX$ \\
& Value of Metric on Unit Cube:    & 1.0 \\
& Note:                & This metric is currently unsupported. \\
& Reference:           & Adapted from \cite{five} \\
\end{tabular}

%---------------------------Jacobian---------------------------
\subsection*{Hex Jacobian:}

\beq
jacobian = MIN\left( \alpha_0, \alpha_1, ...\alpha_7, \frac{\alpha_8} {64} \right)
\eeq

\begin{tabular}{lll}
& Metric Description:  & Minimum pointwise volume of local map at 8 corners \\
&                      & and center of hex. \\
& Dimension:           & $L^3$       \\
& Acceptable Range:    & 0 to $DBL\_MAX$ \\
& Normal Range:        & 0 to $DBL\_MAX$ \\
& Full Range:          & $-DBL\_MAX$ to $DBL\_MAX$ \\
& Value of Metric on Unit Cube:    & 1.0 \\
& Reference:           & \cite{three} \\
\end{tabular}

%---------------------------Oddy---------------------------
\subsection*{Hex Oddy:}

\begin{displaymath}
O(A) = \frac {| A^t A |^2 - \frac {1}{3} | A |^4 } {(\alpha)^{ \frac {4}{3}}}
\end{displaymath}

\begin{center}
$oddy = MAX( O(A_0), O(A_1), ... O(A_8) )$
\end{center}

\begin{tabular}{lll}
& Metric Description:  & Maximum deviation of metric tensor ($A^t A$) at hex \\
&                      & corners and center from identity matrix.\\
& Dimension:           & $L^0$       \\
& Acceptable Range:    & 0 to 0.5 \\
& Normal Range:        & 0 to $DBL\_MAX$ \\
& Full Range:          & 0 to $DBL\_MAX$ \\
& Value of Metric on Unit Cube:    & 0.0 \\
& Note:                & if $\alpha_k \leq DBL\_MIN$ for any $k$, $oddy = DBL\_MAX$ \\
& Reference:           & Adapted from \cite{six} \\
\end{tabular}

%---------------------------Relative Size Squared---------------------------
\subsection*{Hex Relative Size-Squared:}

\begin{displaymath}
D = {\frac {\sum_{i=0}^7 \alpha_i} {8*Average~Hex~Volume}}
\end{displaymath}

\begin{displaymath}
relative~size \textrm{-}squared = {MIN( D, \frac {1}{D})}^2
\end{displaymath}

\begin{flushleft}
$Average ~ Hex ~ Volume$ is the average volume of the hexes in the \\
group of hexes being analyzed. \\
\end{flushleft}

\begin{tabular}{lll}
& Metric Description:  & Square of minimum of the ratio of hex volume to  \\
&                      & average hex volume and the inverse ratio.\\
& Dimension:           & $L^0$       \\
& Acceptable Range:    & 0.5 to 1 \\
& Normal Range:        & 0 to 1 \\
& Full Range:          & 0 to 1 \\
& Note:                & If $Average~Hex~Volume < DBL\_MIN$, $relative~size \textrm{-}squared = 0$ \\
&                      & If $D \leq DBL\_MIN$, $relative~size \textrm{-}squared = 0$ \\
& Reference:           & \cite{four} \\
\end{tabular}

%---------------------------Scaled Jacobian---------------------------
\subsection*{Hex Scaled Jacobian:}

\begin{displaymath}
scaled~jacobian = MIN \left( \hat \alpha_0,
                             \hat \alpha_1,...,
                             \hat \alpha_8 \right)
\end{displaymath}

\begin{tabular}{lll}
& Metric Description:  & Minimum local Jacobian determinant divided by the \\
&                      & corresponding edge-lengths. \\
& Dimension:           & $L^0$       \\
& Acceptable Range:    & 0.5 to 1 \\
& Normal Range:        & -1 to 1 \\
& Full Range:          & -1 to $DBL\_MAX$ \\
& Value of Metric on Unit Cube:    & 1.0 \\
& Note:                & If ${L_{min}}^2 \leq DBL\_MIN$, $scaled~jacobian = DBL\_MAX$ \\
& Reference:           & \cite{three} \\
\end{tabular}

%---------------------------Shape---------------------------
\subsection*{Hex Shape:}

\begin{displaymath}
shape = 3*MIN \left( \frac {{\alpha_0}^{\frac {2}{3}}} {|A_0|^2},
                     \frac {{\alpha_1}^{\frac {2}{3}}} {|A_1|^2},...,
                     \frac {{\alpha_8}^{\frac {2}{3}}} {|A_7|^2} \right)
\end{displaymath}

\begin{tabular}{lll}
& Metric Description:  & 3/Mean Ratio of Jacobian matrix. \\
& Dimension:           & $L^0$       \\
& Acceptable Range:    & 0.3 to 1 \\
& Normal Range:        & 0 to 1 \\
& Full Range:          & 0 to 1 \\
& Value of Metric on Unit Cube:    & 1.0 \\
& Note:                & If $\alpha_k \leq DBL\_MIN$ for any $k$, $shape = 0$ \\
&                      & If $|A_k|^2 \leq DBL\_MIN$ for any $k$, $shape = 0$ \\
& Reference:           & \cite{four} \\
\end{tabular}

%---------------------------Shape and Size---------------------------
\subsection*{Hex Shape and Size:}

\begin{displaymath}
shape~and~size = shape * relative~size \textrm{-}squared
\end{displaymath}

\begin{tabular}{lll}
& Metric Description:  & Product of Shape and Relative Size.\\
& Dimension:           & $L^0$       \\
& Acceptable Range:    & 0.2 to 1 \\
& Normal Range:        & 0 to 1 \\
& Full Range:          & 0 to 1 \\
& Value of Metric on Unit Cube:    & 1.0 \\
& Reference:           & \cite{four} \\
\end{tabular}

%---------------------------Shear---------------------------
\subsection*{Hex Shear:}

\begin{displaymath}
shear = MIN \left( {\hat \alpha_0},
                   {\hat \alpha_1},...,
                   {\hat \alpha_8} \right)
\end{displaymath}

\begin{tabular}{lll}
& Metric Description:  & Minimum Jacobian divided by product of lengths of 3 \\
&                      & edge vectors. \\
& Dimension:           & $L^0$       \\
& Acceptable Range:    & 0.3 to 1 \\
& Normal Range:        & 0 to 1 \\
& Full Range:          & 0 to 1 \\
& Value of Metric on Unit Cube:    & 1.0 \\
& Note:                & If $\hat \alpha_k \leq DBL\_MIN$ for any k, $shear = 0$ \\
&                      & If ${L_{min}}^2 \leq DBL\_MIN; shear = 0$ \\
& Reference:           & \cite{four} \\
\end{tabular}

%---------------------------Shear and Size---------------------------
\subsection*{Hex Shear and Size:}

\begin{displaymath}
shear~and~size = shear * relative~size \textrm{-}squared
\end{displaymath}

\begin{tabular}{lll}
& Metric Description:  & Product of Shear and Relative Size-Squared.\\
& Dimension:           & $L^0$       \\
& Acceptable Range:    & 0.2 to 1 \\
& Normal Range:        & 0 to 1 \\
& Full Range:          & 0 to 1 \\
& Value of Metric on Unit Cube:    & 1.0 \\
& Reference:           & \cite{four} \\
\end{tabular}

%---------------------------Skew---------------------------
\subsection*{Hex Skew:}

\begin{displaymath}
\hat{X_1} = \frac {\vec X_1} {\left| \vec X_1 \right| }
\end{displaymath}

\begin{displaymath}
\hat{X_2} = \frac {\vec X_2} {\left| \vec X_2 \right| }
\end{displaymath}

\begin{displaymath}
\hat{X_3} = \frac {\vec X_3} {\left| \vec X_3 \right| }
\end{displaymath}

\begin{center}
$skew_{12} = \left| \hat{X_1} \cdot \hat{X_2} \right|$
\end{center}

\begin{center}
$skew_{13} = \left| \hat{X_1} \cdot \hat{X_3} \right|$
\end{center}

\begin{center}
$skew_{23} = \left| \hat{X_2} \cdot \hat{X_3} \right|$
\end{center}

\begin{center}
$skew = MAX\left( skew_{12}, skew_{13}, skew_{23} \right)$
\end{center}

\begin{tabular}{lll}
& Metric Description:  & Maximum absolute value of cosine of the angle between  \\
&                      & principle axes pairs. \\
& Dimension:           & $L^0$                              \\
& Acceptable Range:    & 0 to 0.5   \\
& Normal Range:        & 0 to 1     \\
& Full Range:          & 0 to $DBL\_MAX$ \\
& Value of Metric on Unit Cube:    & 0.0 \\
& Note:                & If $|\vec X_1 |$ or $|\vec X_2 |$ or $|\vec X_3| \leq DBL\_MIN$, \\
&                      & $skew = DBL\_MAX$ \\
& Reference:           & Adapted from \cite{one} \\
\end{tabular}

%---------------------------Stretch---------------------------
\subsection*{Hex Stretch:}

\begin{displaymath}
stretch = \frac { \sqrt{3} * L_{min} } { D_{max} }
\end{displaymath}

\begin{tabular}{lll}
& Metric Description:  & Sqrt of three times the ratio of minimum edge length to \\
&                      & maximum diagonal. \\
& Dimension:           & $L^0$                              \\
& Acceptable Range:    & 0.25 to 1.0 \\
& Normal Range:        & 0 to 1     \\
& Full Range:          & 0 to $DBL\_MAX$ \\
& Value of Metric on Unit Cube:    & 1.0 \\
& Note:                & If $D_{max} < DBL\_MIN$, $stretch = DBL\_MAX $ \\
& Reference:           & Adapted from \cite{two} \\
\end{tabular}

%---------------------------Taper---------------------------
\subsection*{Hex Taper:}

\begin{displaymath}
taper_{12} = \frac{\left| \vec X_{12} \right| }
{MIN \left( \left| \vec X_1 \right|, \left| \vec X_2 \right| \right) }
\end{displaymath}

\begin{displaymath}
taper_{13} = \frac{\left| \vec X_{13} \right| }
{MIN \left( \left| \vec X_1 \right|, \left| \vec X_3 \right| \right) }
\end{displaymath}

\begin{displaymath}
taper_{23} = \frac{\left| \vec X_{23} \right| }
{MIN \left( \left| \vec X_2 \right|, \left| \vec X_3 \right| \right) }
\end{displaymath}

\begin{center}
$taper = MAX\left( taper_{12}, taper_{13}, taper_{23} \right)$
\end{center}

\begin{tabular}{lll}
& Metric Description:  & Maximum ratio of cross derivative to any principal axis. \\
& Dimension:           & $L^0$                              \\
& Acceptable Range:    & 0 to 0.5                      \\
& Normal Range:        & 0 to $DBL\_MAX$                       \\
& Full Range:          & 0 to $DBL\_MAX$ \\
& Value of Metric on Unit Cube:    & 0.0 \\
& Note:                & If $|\vec X_1 |$ or $|\vec X_2 |$ or $|\vec X_3| < DBL\_MIN$,\\
&                      & $taper = DBL\_MAX$ \\
& Reference:           & Adapted from \cite{one} \\
\end{tabular}

%---------------------------Volume---------------------------
\subsection*{Hex Volume:}

\begin{displaymath}
volume = \frac {\alpha_8 } {64}
\end{displaymath}

\begin{tabular}{lll}
& Metric Description:  & Product of the magnitudes of the 3 principle axes. \\
& Dimension:           & $L^3$                              \\
& Acceptable Range:    & 0 to $DBL\_MAX$                       \\
& Normal Range:        & 0 to $DBL\_MAX$                       \\
& Full Range:          & $-DBL\_MAX$ to $DBL\_MAX$ \\
& Value of Metric on Unit Cube:    & 1.0 \\
& Reference:           & Traditional \\
\end{tabular}

\begin{thebibliography}{99}

\bibitem{one} L.M. Taylor, and D.P. Flanagan, Pronto3D - A
Three Dimensional Transient Solid Dynamics Program,
SAND87-1912, Sandia National Laboratories, 1989.

\bibitem{two} FIMESH code

\bibitem{three} P. Knupp, "Achieving Finite Element Mesh Quality
via Optimization of the Jacobian Matrix Norm and Quantities",
Intl. J. Numer. Meth. Engng. 2000, 48:1165-1185.

\bibitem{four} P. Knupp, "Algebraic mesh quality metrics for
unstructured initial meshes" Finite Elements in Analysis and
Design, Vol 39, p217-241, 2003.

\bibitem{five} SDRC/IDEAS Simulation: Finite Element
Modeling--User's Guide.

\bibitem{six} A. Oddy, J. Goldak, M. McDill, and M. Bibby, "A
distortion metric for isoparametric finite elements," Trans. CSME,
No. 38-CSME-32, Accession No. 2161, 1988.

\end{thebibliography}

\end{document}


\chapter{Similarity Analysis}\label{simanal}

%\textit{\textbf{This needs an introduction, the smaller the distance the more similar the songs are etc. -> smilarity = 1 / distance}}

This chapter introduces and evaluates different similarity measurements for timbral, melodic, and rhythmic features of music data. It explains the feature extraction, pre-processing and similarity estimation between different songs based on the different feature types.

\section{Timbre Similarity} \label{musly}

This section focuses on different similarity measurements and metrics based on MFCCs. Mel Frequency Cepstral Coefficients have already been introduced in Section~\ref{mfccsim} as a feature to describe timbre.\\

\subsection{Euclidean Distance}\label{mfcceuc}

To further reduce the dimensionality of the original MFCC features, a statistical summarization can be calculated. For each of the mel bands (13 in this case to reduce dimensionality) the mean and standard deviation over all frames are calculated, resulting in a vector of 13 mean values, a 13 by 13 covariance matrix ($\frac{13\cdot(13-1)}{2}$ covariance values, because of the triangular shape - the upper triangle contains the covariances and the main diagonal contains the variances) and 13 variances. These vectors are not dependent on the length of the actual song.~\cite[pp. 51ff]{knees1}\\
Using such a model, the distance between two songs can be calculated using the $L_p$ distance as in equation
\begin{equation} \label{eq:distGmm}
%p (v \vert \lambda)=\sum_{i=1}^{M}w_iN(v \vert \mu_i,\sum_i) \label{prob}
d(x, y) = ||x - y||_p = \left(\sum_{i=1}^{n}{|x_i - y_i|^p}\right)^{\frac{1}{p}},
\end{equation}
where $x$ and $y$ are the $n$-dimensional feature vectors of two different musical pieces.
Usually, the Euclidean ($L_2$) or the Manhattan ($L_1$) distance would be used in real-world scenarios~\cite[p. 58]{knees1}. This very basic metric of timbre similarity has been refined and improved over the past years.

\subsection{Single Gaussian Model}

\subsubsection{Symmetric Kullback-Leibler Divergence}\label{klth}

The second approach was first proposed by Mandel and Ellis in 2005~\cite{mandelellis1} and is briefly summarized in~\cite[pp. 65f]{knees1}.\\
Assuming two musical pieces $P$ and $Q$ are given, after computing the mean value of each MFCC (resulting in the vectors $\mu_P$ and $\mu_Q$) and the covariance matrix of the different MFCC vectors ($\Sigma_P$ and $\Sigma_Q$), the Kullback-Leibler divergence (KL divergence) can be calculated as follows, with $\text{tr}(\cdot)$ being the trace (i.e., the sum of the diagonal of a matrix), $d$ being the dimensionality (number of MFCCs) and $|\Sigma_P|$ being the determinant of $\Sigma_P$~\cite[pp. 65f]{knees1}.\\
\begin{equation} \label{eq:KL1}
\text{KL}_{(P||Q)} = \frac{1}{2}[\text{log}\frac{|\Sigma_P|}{|\Sigma_Q|} + \text{tr}(\Sigma_P^{-1}\Sigma_Q) + (\mu_P - \mu_Q)^T \Sigma_P^{-1} (\mu_Q - \mu_P) - d]
\end{equation}
As a second step the result has to be symmetrized~\cite[p. 44]{schnitzer1}:
\begin{equation} \label{eq:KL2}
d_{SKL}(P, Q) = \frac{1}{2} (\text{KL}_{(P||Q)} + \text{KL}_{(Q||P)})
\end{equation}
This approach is one of the two available similarity metrics in the Musly~\cite{musly1} toolkit (see Section~\ref{mirtoolkit}). It can be simplified and written as a closed form according to Schnitzer~\cite[p. 44]{schnitzer1}:
\begin{equation} \label{eq:SKL}
d_{SKL}(P, Q) = \frac{1}{4} (\text{tr}(\Sigma_P\Sigma_Q^{-1}) + \text{tr}(\Sigma_Q\Sigma_P^{-1}) + \text{tr}((\Sigma_Q^{-1}\Sigma_P^{-1})(\mu_P - \mu_Q)^2) - 2d)
\end{equation}

\subsubsection{Jensen-Shannon-Like Divergence}

The second available similarity method in the Musly toolkit by Schnitzer is using the Jensen-Shannon divergence (in a slightly adapted way). "The Jensen-Shannon (JS) divergence is another symmetric divergence derived from the Kullback-Leibler divergence. To compute it, a mixture $X_m$ of the two distributions is defined"~\cite[p. 43]{schnitzer1}. "To use the Jensen-Shannon divergence [...] to estimate similarities between Gaussians, an approximation of $X_m$ as a single multivariate Gaussian can be used [...] This approximation of $X_m$ is exactly the same as the left-type Kullback-Leibler centroid of the two Gaussian distributions [...]"~\cite[p. 45]{schnitzer1}
\begin{equation} \label{eq:jsl1}
\mu_m = \frac{1}{2} \mu_P + \frac{1}{2} \mu_Q
\end{equation}
\begin{equation} \label{eq:jsl2}
\Sigma_m = \frac{1}{2} (\Sigma_P + \mu_P\mu_P^T) + \frac{1}{2} (\Sigma_Q + \mu_Q\mu_Q^T) - \mu_m\mu_m^T
\end{equation}
\begin{equation} \label{eq:jsl3}
\text{JS}(P, Q) = \frac{1}{2} \text{log}|\Sigma_m| - \frac{1}{4} \text{log} |\Sigma_P| - \frac{1}{4} \text{log} |\Sigma_Q|
\end{equation}

\subsubsection{Mutual Proximity}\label{mprox}
After calculating a similarity matrix for all songs, Musly normalizes the similarities with mutual proximity (MP)~\cite{musly2}. This method aims to reduce the effect of a phenomenon called "hubness", which appears as a general problem of machine learning in high-dimensional data spaces. "Hubs are data points which keep appearing unwontedly often as nearest neighbors of a large number of other data points."~\cite[p. 66]{schnitzer1}.\\
Schedl and Knees state: "To apply MP to a distance matrix, it is assumed that the distances $D_{x,i = 1..N}$ from an object $x$ to all other objects in the data set follow a certain probability distribution; thus, any distance $D_{x,y}$ can be reinterpreted as the probability of $y$ being the nearest neighbor of $x$, given the distance $D_{x,y}$ and the probability distribution $P(x)$ [...] MP is then defined as the probability that $y$ is the nearest neighbor of $x$ given $P(x)$ and $x$ is the nearest neighbor of $y$ given $P(y)$"~\cite[p. 80]{knees1}\\
Resulting in:
\begin{equation} \label{eq:mp1}
P(X > D_{x,y}) = 1 - P(X \leq D_{x,y})% = 1 - \mathscr{F}(D_{x,y})
\end{equation}
\begin{equation} \label{eq:mp2}
\text{MP}(D_{x,y}) = P(X > D_{x,y} \cap Y > D_{x,y}).
\end{equation}
%according to~\cite[p. 80]{knees1}.\\
%Because in this thesis the full similarity matrix is not calculated,

\subsection{Gaussian Mixture Models and Block-Level Features}\label{blocklevel}
Another, more compute-heavy distance measurement would make use of Gaussian Mixture Models (GMMs) of MFCCs. As Knees and Schedl state, "Other work on music audio feature modeling for similarity has shown that aggregating the MFCC vectors of each song via a single Gaussian may work almost as well as using a GMM [...] Doing so decreases computational complexity by several magnitudes, in comparison to GMM-based similarity computations"~\cite[p. 65]{knees1}. Therefore, the usage of GMMs is not further considered in this thesis.\\
The last method mentioned, but not implemented in this thesis for timbral similarity uses block-level features as proposed by Seyerlehner~\cite{seyerlehnerblock} and described in short by Knees and Schedl~\cite[p. 67]{knees1}. Instead of using single frames and summarizing them into statistical or probabilistic models, block-level features use larger, e.g., multiple-second long, audio frames. Features like fluctuation patterns (later introduced in Section~\ref{flucpat}) and spectral patterns (containing timbre information) are computed for these larger blocks of frames.

\subsection{Validation}

For this thesis, the symmetric Kullback-Leibler (SKL) divergence, the Jensen-Shannon-like (JS) divergence and the Euclidean distance are chosen and tested. There is always a trade-off between the complexity and functionality of distance computing algorithms. A re-implementation of block-level features remains left open for future research due to its rather compute heavy nature.\\
Using the Musly toolkit, a first evaluation using the symmetric KL divergence is presented in this section. The feature extraction and distance calculation can also be done in Python using the librosa library, and a re-implementation of the Mandel-Ellis approach was tested as well.\\

\subsubsection{Genre Recall Rate / Construction Noise}
In general, a good measurement for the efficiency of timbre similarity algorithms is the ability to recommend songs of the same genre.
Alternatively, the example proposed by Dr. Bosse from the introduction was tested (see Chapter~\ref{intro}). Comparing a construction noise sound sample with the private music collection containing mostly metal, rock, pop, classical and hip hop music, the following six best results based on the JS divergence were returned in descending order:

\begin{enumerate}
	\setlength\itemsep{-0.5em}
	\item Ziegenm\"uhlen Session - Down On The Corner (Folk Musik)
	\item While She Sleeps - The Divide (Metalcore)
	\item Delain - Mother Machine (Live) (Symphonic Metal)
	\item Within Temptation - Sanctuary (Intro Live) (Symphonic Metal)
	\item Without A Martyr - Medusa's Gaze (Death Metal)
	\item 100 Meisterwerke der Klassik - Orpheus In The Underworld (Orph\'ee aux enfers) - Can-Can (Live At Grosser Saal, Musikverein) (Klassik)
\end{enumerate}

\noindent Figure~\ref{fig:constn} show the distribution of the genres of 100 most similar songs compared to the construction noise sample.

\begin{figure}[htbp]
	\centering
	\framebox{\parbox{1\textwidth}{
	\centering
	\includegraphics[scale=0.65]{Images/cnoise2.png}
	}}
	\caption{Construction noise, first 100 song recommendations based on Musly toolkit (JS)}
	\label{fig:constn}
\end{figure}

\noindent Using an extended dataset consisting of the private music collection, private field recordings, the full FMA dataset, and the musicnet data, the following results could be achieved:

\begin{enumerate}
	\setlength\itemsep{-0.5em}
	\item Born Pilot - Birds Fell (FMA, Electronic, Noise)
	\item mrandmrsBrian - sun is boring (FMA, Avant-Garde, field recordings)
	\item steps in snow (private field recording)
	\item Sawako - Paris Children (FMA, field recordings)
	\item Jeremy Gluck and Michael Dent - Olivier (FMA, Ambient Electronic)
\end{enumerate}

\noindent Especially the second test shows, that the timbre based recommendations are able to recommend similar sounding audio files by returning mostly music containing ambient noises once these were included to the dataset.

\subsubsection{Different Recordings and Cover Versions}\label{covermfcc}
Another experiment was to get the most similar songs to the famous 'Rondo alla Turca' by Mozart.
The recording used as a starting point was taken from the CD "100 Meisterwerke der Klassik" and has a length of 3:33 minutes. This piece by Mozart appears overall four times in the dataset and is recorded by different pianists.
Every recording has a different length as listed in the following overview of the recordings by CD.

\begin{itemize}
	\setlength\itemsep{-0.5em}
	\item 100 Meisterwerke der Klassik (3:33)
	\item Piano Perlen (3:30)
	\item The Piano Collection - Disk 18 (3:28)
	\item Mozart Premium Edition - Disk 31 (4:29)
\end{itemize}

\noindent The top ten most similar songs to the 3 minutes and 33 seconds version are listed below, and the recognized cover versions are underlined:

\begin{enumerate}
	\setlength\itemsep{-0.5em}
	\item Mozart - Concert No. 10 for 2 Pianos and Orchestra in E Flat Major, KV 365 - 2. Andante
	\item Schubert - Sonata in B Flat, D. 960 - III. Scherzo (Allegro vivace con delicatezza)
	\item Albeniz - Iberia, Book I - Evocaci\'on
	\item \underline{Mozart Sonate Nr. 11 in A-Dur, K. 33 - Mozart - Alla Turca Allegretto (3:28)}
	\item Beethoven - Bagatellen Op 119 -Allemande in D major
	\item Mozart - Rondo No. 1 in D Major, K. 485
	\item Mozart - Sonata For Piano No. 8 KV 310 A Minor - Allegro Maestoso
	\item Sonata For Piano No. 16 KV 545 C Major - Rondo: Allegretto
	\item \underline{Mozart Sonate Nr. 11 in A-Dur, K. 33 - III. Tuerkischer Marsch (3:30)}
	\item Mozart - Piano Sonata No. 13 in B flat major, K. 333 (K. 315c): Allegretto grazioso

\end{enumerate}
\noindent The interesting conclusion is that only two out of the three other versions were considered as most similar songs. The other recording was not even in the top 30 list of the most similar songs. However, the recommendations are all from the same genre (classical music). The inability to detect cover versions was also observable for other songs in the dataset like Serj Tankians song "Lie Lie Lie" from the CD "Harakiri" (just to give an example). This is probably due to the usage of MFCCs valuing the timbre of the music predominantly instead of the pitches and melody movements.


\chapter{Assembly-Line Scheduling Algorithm}
The objective of this challenge is to move from entry to exit point via stations
along two assembly lines. Each station has a fixed time of processing, and from
a station one can move to the next station along the line OR the next station on
the other assembly line. Elements can be of equal value.

\section{Brute Force}
With $n$ stations on each assembly line, there are $2^n$ possible ways to choose
stations. Therefore the brute force algorithm takes $\Omega\left(2^n\right)$ of
time complexity to be completely processed, making it infeasible.

\section{The Fastest Way}
\imb{\alg/assemblyline.cpp}
With a simple loop algorithm and a minimum-evaluating process every iteration,
this process shows a time complexity of: \[ T(n)=\Theta(n) \]