% Bismillahi-r-Rahmani-r-Rahim
\documentclass{article}
\author{Daoud Clarke}
\date{\today}
\title{Investigations into New Lattice Orderings}

\usepackage{color}
\usepackage{graphicx}
\usepackage{amssymb}
\usepackage[round]{natbib}

\newtheorem{definition}{Definition}

\newcommand{\R}{\mathbb{R}}

\begin{document}

\maketitle

Context-theoretic semantics \citep{Clarke:07} defines a lattice
ordering on the vector space formed from contexts of words. The
lattice meet and join are defined by taking component-wise minimums
and maximums with respect to the basis defined by contexts: there is a
basis vector (dimension) for each context a word can occur in.

However, this ordering is only one that is possible on the vector
space. Here we consider two others and evaluate their plausibility for
defining entailment between vector representations of meaning.

\section{Motivation}

In \citep{Clarke:12}, a context theory is defined as follows:
\begin{definition}[Context Theory]
A context theory is a tuple $\langle A, \mathcal{A}, \xi, V, \psi
\rangle$, where $A$ is a set (the alphabet), $\mathcal{A}$ is a unital
algebra over the real numbers, $\xi$ is a function from $A$ to
$\mathcal{A}$, $V$ is an abstract Lebesgue space and $\psi$ is an
injective linear map from $\mathcal{A}$ to $V$.
\end{definition}

The definition is quite complex because the space $V$ in which the
ordering is defined is a bigger space than the space $\mathcal{A}$
used to define composition. If we were able to define an ordering on
$\mathcal{A}$ instead, then this would significantly simplify the
definition, to $\langle A, \mathcal{A}, \xi\rangle$.

The second motivation for investigating new partial orderings is that
we can potentially learn them from data, instead of using a fixed
ordering defined by a basis. A partial ordering divides the space in a
similar way to the planes learnt by support vector machines, so it is
feasible that techniques from this area could be generalised to learn
cones.

\section{Background}

\begin{figure}
\begin{center}
\input{basis.tex}
\caption{The basis associated with a vector space can defines an
  ordering. This shows the join of two elements in the plane with the
  ordering defined by the basis formed from the $x$ and $y$ axes. The
  shaded areas are $u + C$ and $v + C$ where $C$ is the positive
  cone defined by this ordering. }
\end{center}
\end{figure}

A \textbf{partially ordered vector space} is a vector space $V$
together with a partial ordering $\le$ that satisfies:
\begin{itemize}
\item If $u \le v$ then $u + w \le v + w$
\item If $u \le v$ then $\alpha u \le \alpha v$
\end{itemize}
for all $u,v,w \in V$ and $\alpha \ge 0$. If the partial ordering is a
lattice, then the vector space is called a vector lattice or
\textbf{Riesz space}.

\begin{figure}
\begin{center}
\input{cone.tex}
\end{center}
\caption{A cone $C$ defining a lattice ordering on the two dimensional
  vector space in the plane.}
\end{figure}

\begin{figure}
\begin{center}
\input{cones.tex}
\end{center}
\caption{The join of two elements $u$ and $v$ and the spaces $u + C$
  and $v + C$ where $C$ is the cone defining the ordering of the
  space.}
\end{figure}

A \textbf{cone} is a subset $C$ of a vector space satisfying
\begin{itemize}
\item $C + C \subseteq C$
\item $\alpha C \subseteq C$ for all $\alpha \ge 0$
\item $C \cap (-C) = \{0\}$
\end{itemize}
If $V$ is a partially ordered vector space, then the set $V^+ = \{v
\in V : v \ge 0\}$ is a cone, called the \textbf{positive
  cone}. Conversely, given any cone $C$ for a vector space $V$, we can
define a partial ordering on $V$ by $u \le v$ iff $v - u \in C$.

Given a countable set $U = \{u_1, u_2, \ldots\}$ of vectors, the cone
generated by $U$ is the set $C_U = \{v : v = \sum_i \alpha_i u_i\}$ for
some $\alpha_i \ge 0$.

\section{Context Vector Cones}

We can define a positive cone on $\mathcal{A}$ to be the cone
generated by all context vectors, $\hat{A} = \{\hat{x} : x \in A^*\}$. There are
two questions we can ask:
\begin{itemize}
\item Is the ordering interesting, or is likely to turn out to be
  trivial (for example, all strings have meet $0$)?
\item Does the ordering define a lattice? We need the meet operation
in order to define a degree of entailment between strings.
\end{itemize}

In fact it seems that we can't have our cake and eat it, as if the
first question is true, it seems likely the second is false, and vice
versa. If the set $\hat{A}$ is linearly independent, then it seems
that the first is true, and the second false, while the converse holds
in general otherwise. (At the moment, I only have an intuition for
this --- I've yet to prove it).

\section{Plane Orderings}

\begin{figure}
\begin{center}
\input{plane.tex}
\end{center}
\caption{A plane found using a support vector machine can be
  considered as dividing the vector space, and thus defines a partial
  ordering.}
\end{figure}

There is an interesting similarity between how a cone divides the
vector space and the division of a space introduced by a plane, as
used for example in support vector machines. In fact the plane also
defines an ordering, if we define one side to be ``positive''. Then,
as before, $x \le y$ if $y - x$ is on the positive side of the plane.

The ordering is determined entirely by the dimension perpendicular to
the plane. Let $p$ be the vector from the origin to the plane, and
perpendicular to it, and $\bar{p}$ be the one dimensional subspace
generated by $p$. Then $x \le y$ iff $\|\hat{p}(y) - \hat{p}(x)\| \ge
\|p\|$, where by $\hat{p}(x)$ we mean the projection of $x$ onto the
dimension $\hat{p}$, that is, if the projection of $y - x$ onto the
subspace generated by $p$ is greater than or equal to $p$.

In general, this ordering is not a vector space ordering as defined
above unless it passes through the origin. The zero vector is either
positive or negative depending on which side of the plane it
falls. It also does not define a lattice ordering. For example, when
the plane does pass through the origin, the plane defines a total
ordering between vectors purely depending on their position when
projected onto $\hat{p}$. Any vectors on the plane perpendicular to a
point on $\hat{p}$ are equivalent under this partial ordering (thus it
is not strictly a partial ordering).

\section{Relating Strings of Different Lengths with Orderings}

We can use vector space orderings to relate strings of different
lengths. This is similar to the quotient algebra construction of
\cite{Clarke:10}, but allows us to specify the asymmetric relation of
entailment, instead of equating vectors. In fact the two techniques
complement one another: if we need to impose equality between vectors
we can use the quotient algebra construction, and if we need to relate
two vectors by entailment then we can change the ordering.

For example, if we wish to impose the conjunctive nature of ``and'' on
the vector space, we can do this by requiring that $uav \le u$ and
$uav \le v$ for all $u$ and $v$ where $a$ is the vector for
``and''. We can do the same for ``or'' with $u \le uov$ and $v\le uov$
where $o$ is the vector for ``or''.

Similarly, we can make all adjectives restrict the properties of the
nouns they operate on by making $au \le u$ for all adjective vectors
$a$ and all noun space vectors $u$.

In a sense, this is like imposing a logic on top of the vector space
structure, where the rules of the logic are encapsulated in the
positive cone.

\subsection{Application to Sentence Space}

This also gives us a surprising answer for what the sentence space
should look like: it doesn't really matter, as long as the ordering
between sentences can be specified correctly. For example, we can just
use the tensor product between sentences but ensure that $s_1\otimes s_2 \le
s_1$ and $s_1\otimes s_2\le s_2$.

Let $V$ be a fixed dimensional space used to represent the meaning of
a single sentence, then we use
$$T(V) = \mathbb{R}\oplus V \oplus (V\otimes V) \oplus (V\otimes V\otimes V) \oplus \cdots$$
to represent multiple sentences. Given a partial ordering $\le$ on
$V$, there is a natural extension to an ordering on $T(V)$, for
example $u_1\otimes v_1 \le u_2\otimes v_2$ iff $u_1 \le u_2$ and
$v_1 \le v_2$, and similarly for higher tensor powers.

We also assume that $V$ has a preferred basis that defines a natural
lattice ordering, by $u_1 \le u_2$ if every component of $u_1$ is less
than or equal to the corresponding component of $u_2$.

Let $C$ be the positive cone on $T(V)$ as defined above, i.e.~the set
of all vectors $u\in T(V)$ such that $u \ge 0$. Under the standard
ordering, strings of different lengths are not related, so we add
additional vectors to $C$ to relate them. Let
$$C' = C \cup \{ u - u\otimes v : u,v \in C \} \cup \{ v - u\otimes v : u,v \in C \}$$
This defines a cone which behaves as the normal ordering for strings
of the same length, but allows strings of different lengths to be
related.

\subsection{Problems with this Ordering}

There is freedom in choosing elements which allows the whole
half-plane to be generated. For example in $u - u\otimes v$, $v$ can
be a very large positive vector, or $u$ very small, and in the limit,
this is the same as adding $-u\otimes v$ to the positive cone which
leads to an uninteresting ordering.

% To get around this we can try and limit the size of vectors we add:
% $$D = \{ u - u\otimes v : u,v \in U(C) \} \cup \{ v - u\otimes v : u,v \in U(C) \}$$
% where $U(C)$ denotes the set of all vectors $u\in C$ with $\|u\| = 1$,
% then generate the new positive cone $C'$ from $D\cup C$.

\subsection{Basis Orderings}

Write $V^{\otimes n}$ for the subspace of $T(V)$ generated by tensor
powers of $V$ to the $n$th degree, and $T^n(V)$ for the subspace
generated by all tensor powers up to and including $n$.

Let $B = \{e_1, e_2, \ldots\}$ be an orthonormal basis for
$T(V)$. Assume for each basis vector $e_i$ in $T(V)$, if $e_i\in
V^{\otimes n}$ we have a vector $u_i \in T^{n-1}(V) \cap C$.

We then define a new cone $D$ to be that generated by
$$\{e_i - u_i\ : i \ge 1\} \cup B$$

\subsection{Example from Quotient Algebra}

Plain tensor algebra:

\begin{tabular}{llllllllll}
term & red book &  city &  apple & big city & red apple & big book &  book & big apple & red city\\
red book & 1 & 0 & 0 & 0 & 0.33 & 1 & 0 & 0.33 & 0\\
 city & 0 & 1 & 0.33 & 0 & 0 & 0 & 0 & 0 & 0\\
 apple & 0 & 0.33 & 1 & 0 & 0 & 0 & 0.33 & 0 & 0\\
big city & 0 & 0 & 0 & 1 & 0.22 & 0 & 0 & 0.33 & 0.67\\
red apple & 0.33 & 0 & 0 & 0.33 & 1 & 0.33 & 0 & 1 & 0.33\\
big book & 0.67 & 0 & 0 & 0 & 0.22 & 1 & 0 & 0.33 & 0\\
 book & 0 & 0 & 0.33 & 0 & 0 & 0 & 1 & 0 & 0\\
big apple & 0.22 & 0 & 0 & 0.33 & 0.67 & 0.33 & 0 & 1 & 0.22\\
red city & 0 & 0 & 0 & 1 & 0.33 & 0 & 0 & 0.33 & 1\\
\end{tabular}

Cone:

\begin{tabular}{llllllllll}
term & red book &  city &  apple & big city & red apple & big book &  book & big apple & red city\\
red book & 1 & 0 & 0.11 & 0 & 0.33 & 1 & 0.33 & 0.33 & 0\\
 city & 0 & 1 & 0.33 & 1 & 0.33 & 0 & 0 & 0.33 & 1\\
 apple & 0.33 & 0.33 & 1 & 0.33 & 1 & 0.33 & 0.33 & 1 & 0.33\\
big city & 0 & 0.25 & 0.083 & 1 & 0.25 & 0 & 0 & 0.33 & 0.75\\
red apple & 0.33 & 0.11 & 0.33 & 0.33 & 1 & 0.33 & 0.11 & 1 & 0.33\\
big book & 0.75 & 0 & 0.083 & 0 & 0.25 & 1 & 0.25 & 0.33 & 0\\
 book & 1 & 0 & 0.33 & 0 & 0.33 & 1 & 1 & 0.33 & 0\\
big apple & 0.25 & 0.083 & 0.25 & 0.33 & 0.75 & 0.33 & 0.083 & 1 & 0.25\\
red city & 0 & 0.33 & 0.11 & 1 & 0.33 & 0 & 0 & 0.33 & 1\\
\end{tabular}

\bibliographystyle{plainnat}

\bibliography{contexts}

\end{document}


%% ----------------------------------------------------------------
%% Appendix: Machine-learning model hyper-parameters tuning
%% ----------------------------------------------------------------
%!TeX root = subfile
\label{chapter:appendixD}
\markboth{Appendix D Machine-learning model hyper-parameters tuning}{Appendix D Machine-learning model hyper-parameters tuning}
\documentclass[../main.tex]{subfiles}
% ----------------------------------------------------------------
\begin{document}
A hyper-parametric study of the proposed ML model for the SANS equations is performed next.
Different MIMO-CNN architectures are considered, as shown in \fref{fig:cnn}.
All of the different CNNs are based on encoding the input information into some lower-dimensional latent space followed by a decoding procedure that maps such latent space into the target outputs, similarly to SegNet \citep{Badrinarayanan2015} or U-net \citep{Ronneberger2015} architectures.

For the hyper-parametric study, a dataset of 10152 snapshots containing the resolved quantities $\left\lbrace U, V, P \right\rbrace$ and the residual closure terms $\lbrace \mathcal{S}^R_x,\mathcal{S}^R_y\rbrace$ is generated throughout approximately 600 wake cycles.
The size of the recorded 2-D flow fields is $1216\times540$.
The data is split 80-20 for training and testing, respectively.
The training data is further split 75-25 for optimisation and validation, respectively.
The test dataset is generated after the 600 wake cycles, hence this data is completely hidden from the CNN during the training stage.
Only the inner uniform domain (see \fref{fig:computational_domain}) is stored during the data generation process.
For the training stage, an early-stopping approach is considered as the convergence criteria when the validation loss does not improve for 8 consecutive epochs (dataset iterations), or a maximum of 200 epochs is reached.
Success is measured by the Pearson correlation coefficient between the target $({\mathcal{S}}^{R})$ and predicted $(\mathcal{S}^{R,\,\mathrm{ML}})$ output fields, respectively
\begin{gather}
\mathcal{CC}_x=\mathcal{CC}\pars{\mathcal{S}^R_x,\mathcal{S}_x^{R,\,\mathrm{ML}}}=\frac{\mathrm{cov}\pars{\mathcal{S}^R_x,\mathcal{S}_x^{R,\,\mathrm{ML}}}}{\sqrt{\mathrm{cov}\pars{\mathcal{S}^R_x,\mathcal{S}^R_x}\mathrm{cov}\pars{\mathcal{S}_x^{R,\,\mathrm{ML}},\mathcal{S}_x^{R,\,\mathrm{ML}}}}},\\
\mathcal{CC}_y=\mathcal{CC}\pars{\mathcal{S}^R_y,\mathcal{S}_y^{R,\,\mathrm{ML}}}=\frac{\mathrm{cov}\pars{\mathcal{S}^R_y,\mathcal{S}_y^{R,\,\mathrm{ML}}}}{\sqrt{\mathrm{cov}\pars{\mathcal{S}^R_y,\mathcal{S}^R_y}\mathrm{cov}\pars{\mathcal{S}_y^{R,\,\mathrm{ML}},\mathcal{S}_y^{R,\,\mathrm{ML}}}}}.
\end{gather}

A mini-batch gradient descend method (4 to 6 samples depending on the GPU memory) using the Adam optimiser \citep{Kingma2014} is employed.
The learning rate is set to $10^{-4}$ with a decay rate of $10^{-5}$.
The full model is developed under Keras \citep{Chollet2015}, a deep-learning Python library using the TensorFlow \citep{Tensorflow} backend.

Because of the network weights are initialised randomly and stochastic gradient descent backpropagation is used to optimise the network, there is an irreducible level of randomness in the final results.
Hence, the training process is repeated for cases converged into a very shallow local minima and only the best results have been reported.

A flow field snapshot of resolved quantities and a target output is displayed in \fref{fig:flow_field}.
The complex flow structures at the shear layer roll-up region behind the cylinder can be particularly challenging to model.
This can indicate that a large CNN in terms of number of trainable parameters (weights $\matr{W}$ and bias $\vect{b}$) is required.
Also, beyond exploring different CNN architectures, we test different loss functions, activation functions, number of filters and kernel size, effect of input data normalisation, and different input sets.
A summary of the parametric study is detailed in \tref{tab:parameters} and the base-case parameters are shown in \tref{tab:models_fixed_parameters}.

\begin{table}
\begin{center}
\begin{tabular}{ll}
\toprule
Architecture & M1, M2, M3\\
Loss function & sse, sae \\
Activation function & ReLU, tanh, sigmoid \\
Max. filters & 16, 32, 64 \\
Kernel size & $3\times3, \,\,\, 5\times5, \,\,\,7\times7$ \\
Normalised input data & False, True \\
Input set & $\left\lbrace U, V, P \right\rbrace$, $\left\lbrace \Omega_z, d \right\rbrace$,  $\left\lbrace \nabla\vect{U}, \nabla P \right\rbrace$ \\
\bottomrule
\end{tabular}
\end{center}
\caption{Parametric study.
sse: sum square error.
sae: sum absolute error.
ReLU: rectified linear unit.
$\Omega_z$: Spanwise-averaged vorticity.
$d$: Distance function to the cylinder.}
\label{tab:parameters}
\end{table}

\begin{table}
\begin{center}
\begin{tabular}{ll}
\toprule
Architecture & M3 \\
Loss function & sse \\
Activation function & ReLU \\
Max. filters & 64 \\
Kernel size & $3\times3$ \\
Normalised input data & True \\
Input set & $\left\lbrace U, V, P \right\rbrace$ \\
\bottomrule
\end{tabular}
\end{center}
\caption{Base-case parameters of the parametric investigation.}
\label{tab:models_fixed_parameters}
\end{table}

\clearpage
\begin{figure}[!h]
	\centering
	\begin{subfigure}[t]{\linewidth}
		\centering
		\includegraphics[width=0.7\linewidth]{appendixD/cnn/cnn_1.pdf}%
		\caption{Model 1 (M1)}\vspace{0.7cm}
	\end{subfigure}
	\begin{subfigure}[t]{\linewidth}
		\centering
		\vspace{0.5cm}\includegraphics[width=0.95\linewidth]{appendixD/cnn/cnn_2.pdf}%
		\caption{Model 2 (M2)}\vspace{0.9cm}
	\end{subfigure}
	\begin{subfigure}[t]{\linewidth}
		\centering
		\includegraphics[width=0.6\linewidth]{appendixD/cnn/cnn_3.pdf}%
		\caption{Model 3 (M3)}
	\end{subfigure}
	\captionsetup[subfigure]{width=\linewidth}
	\caption{CNN architectures investigated.
The input $(\mathrm{X}_n)$ and output $(\mathrm{Y}_n)$ fields size is $1216\times540$.
Arrows color indicate layers of:
Dark blue: Conv2D $+$ batch normalisation $+$ activation function.
Yellow: MaxPooling $2\times2$.
Magenta: UpSampling $2\times2$.
Green: Conv2D $1\times1$ $+$ batch normalisation $+$ linear function.
The thickness of each block (noted on the last dimension) represents the number of filters.
On the encoding blocks, the number of filters is doubled at every convolutional layer, reaching a maximum number of filters just before the decoding block, where the operation is reversed halving the number of filters at every convolutional layer.
The merging operator $\frown$ indicates concatenation in the last dimension, hence stacking filters from multiple blocks.}\label{fig:cnn}
\end{figure}

\begin{figure}
	\centering
	\captionsetup[subfigure]{width=1\linewidth}
	\begin{subfigure}[t]{0.9\linewidth}
		\centering
		\includegraphics[width=\linewidth]{appendixD/flow_field/P.pdf}%
	\caption{$P$}
	\end{subfigure}
	\begin{subfigure}[t]{\linewidth}
		\centering
		\includegraphics[width=0.9\linewidth]{appendixD/flow_field/vort_lowres.pdf}%
	\caption{$\Omega_z$}
	\end{subfigure}
	\begin{subfigure}[t]{\linewidth}
		\centering
		\includegraphics[width=0.9\linewidth]{appendixD/flow_field/tau_x_lowres.pdf}%
		\caption{$\mathcal{S}^R_x$}
	\end{subfigure}
	\caption{Flow snapshot of resolved (top, middle) and target closure (bottom) fields.} \label{fig:flow_field}
\end{figure}

\section{Results}

\subsection{Model architecture and data normalisation}

We start investigating the CNN architecture based on three different models, namely M1, M2 and M3 (see \fref{fig:cnn}).
The base-case parameters are detailed in \tref{tab:models_fixed_parameters}.
We also study the effect of normalising each input data sample with its standard score, i.e. $\hat{x} = \pars{x-\bar{x}}/\sigma_x$.

Results for the training stage of each model are displayed in \fref{fig:models_convergence}.
It can be observed that normalising the input data provides a faster convergence, noting that the non-normalised M1 case hits the maximum number of epochs limit.
The correlation coefficient results are given in \tref{tab:models_correlation}.
All models provide a high correlation coefficient for both validation and test datasets (hence no over-fitting is detected).
The general trend is that accuracy increases with the network size.
It is expected that output decoding branches having access to a larger latent space (number of filters encoding information on the encoder output), such as M3, are more flexible and perform better than networks with a smaller latent space.
The M3 model with normalised input data is selected for all the subsequent analysis, unless specified otherwise.
The choice is motivated by the M3 model training consistency and its slightly faster convergence when compared to other models.

{\renewcommand{\arraystretch}{1.2}
\begin{table}[t]
\begin{center}
\begin{tabular}{ccccc}
\toprule
Case & \specialcell{Train.\\params.} & val.: $\mathcal{CC}_x,\,\mathcal{CC}_y$ & test: $\mathcal{CC}_x,\,\mathcal{CC}_y$ & Best epoch\\
\midrule
M1 & 258242 & 0.75, 0.83 &  0.74, 0.83 & 200\\
\specialcell{M1\\norm.} & 258242 & 0.84, 0.85 & 0.83, 0.84 & 70\\
M2 & 300194 & 0.85, 0.86 & 0.84, 0.84 & 81\\
\specialcell{M2\\norm.} & 300194 & {0.86, 0.88} & {0.85, 0.88}& 62\\
\specialcell{M2\\norm.\\stacked} & 58538 & 0.76, 0.77 &  0.74, 0.76 & 63\\
M3 & 338498 & 0.87, 0.88 & 0.86, 0.87 & 105\\
\specialcell{M3\\norm.} & 338498 & 0.86, 0.85 & 0.85, 0.84 & 70\\
\bottomrule
\end{tabular}
\end{center}
\caption{Validation and test results for different model architectures.}
\label{tab:models_correlation}
\end{table}}

\clearpage
\begin{figure*}[!ht]
\centering
\vspace{1cm}
\begin{multicols}{2}
	\begin{subfigure}[t]{\linewidth}
		\includegraphics[width=\linewidth]{appendixD/history/1.pdf}%
		\caption{M1}
	\end{subfigure}\vspace{0.3cm}
	\begin{subfigure}[t]{\linewidth}
		\includegraphics[width=\linewidth]{appendixD/history/3.pdf}%
		\caption{M2}
	\end{subfigure}\vspace{0.3cm}
	\begin{subfigure}[t]{\linewidth}
		\includegraphics[width=\linewidth]{appendixD/history/5.pdf}
		\caption{M3}
	\end{subfigure}
		\begin{subfigure}[t]{\linewidth}
		\includegraphics[width=\linewidth]{appendixD/history/2.pdf}%
		\caption{M1, norm.}
	\end{subfigure}\vspace{0.3cm}
		\begin{subfigure}[t]{\linewidth}
		\includegraphics[width=\linewidth]{appendixD/history/4.pdf}%
		\caption{M2, norm}
	\end{subfigure}\vspace{0.3cm}
	\begin{subfigure}[t]{\linewidth}
		\includegraphics[width=\linewidth]{appendixD/history/6.pdf}%
		\caption{M3, norm.}
	\end{subfigure}
\end{multicols}
	\caption{Training history of the different model architectures.
Left: Input data not normalised.
Right: Input data normalised.
Legend: Grey: fit loss.
Black: val. loss.
Blue: $\mathcal{CC}_x$ (val. dataset).
Purple: $\mathcal{CC}_y$ (val. dataset).
Red: Early stop mark.}
	\label{fig:models_convergence}
\end{figure*}

\clearpage
Additionally, a simple auto-encoder concatenating the resolved quantities in a single input branch (stacking them in the filter dimension, noted in \tref{tab:models_correlation} as ``stacked'') has been tested.
Even though this model is much smaller than the other ones in terms of number of trainable parameters, it still manages to provide acceptable predictions.

\subsection{Loss function}

The loss function between predicted and target fields is investigated next.
The same previous base-case parameters are chosen and results are summarised in \tref{tab:loss_correlation}.

{\renewcommand{\arraystretch}{1.2}
\begin{table}
\begin{center}
\begin{tabular}{ccccc}
\toprule
Case & \specialcell{Train.\\params.} & val.: $\mathcal{CC}_x,\,\mathcal{CC}_y$ & test: $\mathcal{CC}_x,\,\mathcal{CC}_y$ & Best epoch\\
\midrule
sse & 338498 & 0.86, 0.85 & 0.85, 0.84 & 70 \\
sae & 338498 & 0.86, 0.57 & 0.83, 0.48 & 45\\
\bottomrule
\end{tabular}
\end{center}
\caption{Validation and test results for different loss functions.}
\label{tab:loss_correlation}
\end{table}}

It can be observed that using the absolute error yields to a local minima where one of the outputs has not converged successfully.
The squared error function being an easier metric to differentiate (convex parabola) than the absolute error might be causing such difference.

\subsection{Activation function}

Activation functions ($h$) are used to dictate the output of a neuron based on some input value $x$ (the model input or a previous layer output), a weight and a bias: $a = h(wx+b)$.
Activation functions widely used in ANNs are tested here.
We consider the ReLU, tanh, and sigmoid activation functions.
Results are displayed in \tref{tab:activation_correlation}.

It can be observed that the ReLU activation function provides a clear advantage in comparison with the other tested functions.
The ReLU function allows for faster trainings of deep ANNs because it preserves the neuron relative intensity across layers (maximum value is not capped) \citep{Nair2010}.
This is also due to the fact that the ReLU function helps mitigating vanishing-gradient caveats which arise more naturally when using the sigmoid function.

{\renewcommand{\arraystretch}{1.2}
\begin{table}
\begin{center}
\begin{tabular}{ccccc}
\toprule
Case & \specialcell{Train.\\params.} & val.: $\mathcal{CC}_x,\,\mathcal{CC}_y$ & test: $\mathcal{CC}_x,\,\mathcal{CC}_y$ & Best epoch\\
\midrule
ReLU & 338498 & 0.86, 0.85 & 0.85, 0.84 & 70 \\
tanh & 338498 & 0.78, 0.76 & 0.77, 0.76 & 62\\
sigmoid & 338498 & 0.81, 0.82 & 0.80, 0.82 & 36\\
\bottomrule
\end{tabular}
\end{center}
\caption{Validation and test results for different activation functions.}
\label{tab:activation_correlation}
\end{table}}

\subsection{Number of filters and kernel size}

The number of filters per convolutional layer and the convolution kernel size is explored.
The results for the maximum number of filters per convolutional layer are displayed in \tref{tab:filters_correlation}, where the number of trainable parameters jumps by a factor of 4.
The results observed in this particular study are strongly biased by the number of training epochs.
It should be noted that the case using a maximum of 16 filters only converged after almost 400 epochs (the maximum epochs limit has been lifted here because of the network size), whereas the other cases are both converged under 100 epochs.
The slow convergence of the 16 filters might be due to its lack of flexibility (model degrees of freedom) compared to the models with more filters available.
Still, the 64 filters case displays correlation coefficients relatively close to the 16 filters case with only 70 epochs of training.
Hence it can be argued that increasing the number of filters allows the model to be optimised based on a wider range of spatial features thus achieving a faster convergence.
Simultaneously, increasing the number of filters does not automatically yield better accuracy.

From a computational cost standpoint, the training time for the 64 maximum filters case is approximately 26 min./epoch, whereas the 32 and 16 filters cases are both around 23 min./epoch (the latter only measured during its first 100 epochs).
Hence, the computational cost does not increase linearly with the network size.
On the other hand, the computational training time of the 16 filters case gradually increases with every new epoch.
For the 378 epochs until convergence, a total of 180 hours of computational time is required, averaging to roughly 28 min./epoch.
Therefore, the computational time per epoch increases with the number of epochs, making the 16 maximum filters case less efficient compared to the larger networks.

{\renewcommand{\arraystretch}{1.2}
\begin{table}
\begin{center}
\begin{tabular}{ccccc}
\toprule
Case & \specialcell{Train.\\params.} & val.: $\mathcal{CC}_x,\,\mathcal{CC}_y$ & test: $\mathcal{CC}_x,\,\mathcal{CC}_y$ & Best epoch\\
\midrule
64 & 338498 & 0.86, 0.85 & 0.85, 0.84  & 70\\
32 & 85154 & 0.81, 0.72 & 0.81, 0.71 & 34\\
16 & 21554 & 0.87, 0.87 & 0.87, 0.86 & 386\\
\bottomrule
\end{tabular}
\end{center}
\caption{Validation and test results for different maximum number of filters.}
\label{tab:filters_correlation}
\end{table}}

{\renewcommand{\arraystretch}{1.2}
\begin{table}
\begin{center}
\begin{tabular}{ccccc}
\toprule
Case & \specialcell{Train.\\params.} & val.: $\mathcal{CC}_x,\,\mathcal{CC}_y$ & test: $\mathcal{CC}_x,\,\mathcal{CC}_y$ & Best epoch\\
\midrule
$3\times3$ & 338498 & 0.85, 0.86 & 0.86, 0.84 & 70 \\
$5\times5$ & 937282 & 0.89, 0.89 & 0.89, 0.89 & 101\\
$7\times7$ & 1835458 & 0.87, 0.86 & 0.86, 0.86 & 34\\
\bottomrule
\end{tabular}
\end{center}
\caption{Validation and test results for different kernel sizes.}
\label{tab:kernels_correlation}
\end{table}}

The results for the kernel size effect are displayed in \tref{tab:kernels_correlation}.
The kernel size is important for the detection of small- and large-scale structures.
The kernel size to capture the smallest scale in the wake should be, at least, a $3\times3$ filter since our iLES solver is resolving spatial scales down to grid level.
Structures larger than the grid-level scale might be better identified by slightly larger kernels.
However, the nonlinear combination of feature maps across the CNN layers allows small kernels to reconstruct such large-scale structures, hence the implied advantage of slightly larger kernels is not as relevant.
On the other hand, large kernels will filter out information of the subkernel-scale structures.
This is why the current accepted practice in CNNs is to use either $3\times3$ or $5\times5$ kernels and our results agree with that practice.
The $5\times5$ kernel configuration provides the best prediction among the kernels set, where a compromise between the detection of small- and large-scale structures is achieved.

\subsection{Input data}

Last, different input sets are evaluated.
Here, we consider the M2 model without normalising the inputs.
This is motivated by the fact that the M3 model becomes very large when 6 different inputs are used, as it is the case with the $\left\lbrace \nabla\vect{U}, \nabla P \right\rbrace$ input set.
The remaining base-case parameters are employed.
By testing different inputs we seek to include features more closely related to the expected outputs so that the regression problem solved by the CNN becomes easier \citep{Gamahara2017, Beck2019}.

As summarised in \tref{tab:inputs_correlation}, it is found that the primitive quantities set and the gradient set provide the highest correlation between target and predicted closure terms.
On the other hand, the vorticity plus the distance function input set does not perform as well.
The closure terms are, in their core, convective terms of the spanwise-fluctuating velocity field.
Hence, it is reasonable to argue that spatial derivatives of the resolved spanwise-averaged velocity field carry more relevant information to the closure terms than the other input sets.
Still, the primitive quantities set performs very similarly to the gradient set.
This can be an indicator of the network being able to find the gradients on its own.
Also, compared to the vorticity input set, it seems that information on the pressure field is important although this has not been tested in direct comparison.

{\renewcommand{\arraystretch}{1.2}
\begin{table}
\begin{center}
\begin{tabular}{ccccc}
\toprule
Case & \specialcell{Train.\\params.} & val.: $\mathcal{CC}_x,\,\mathcal{CC}_y$ & test: $\mathcal{CC}_x,\,\mathcal{CC}_y$ & Best epoch\\
\midrule
$\left\lbrace U, V, P \right\rbrace$& 338498 & 0.86, 0.85 & 0.85, 0.84 & 70\\
$\left\lbrace \Omega_z, d \right\rbrace$& 255746 & 0.57, 0.52  & 0.56, 0.52 & 110\\
$\left\lbrace \nabla\vect{U}, \nabla P \right\rbrace$ & 433538 & 0.86, 0.87 & 0.85, 0.86 & 112\\
\bottomrule
\end{tabular}
\end{center}
\caption{Validation and test results for different input sets.}
\label{tab:inputs_correlation}
\end{table}}

\section{Conclusion}

The M2 model has provided the highest correlations across the tested models for a given set of base-case parameters, and, on the other hand, the M3 model offered the best consistency while training.
Also, normalising the input data allows for faster training convergence without significantly compromising accuracy.
The sse loss function, a maximum of 64 filters, and a kernel size of $5\times5$ proved to be the best possible parameters.
Finally, the gradient input set and the primitive quantities set performed similarly providing the highest correlation coefficients (see \tref{tab:best_model}).

From a qualitative standpoint, predictions and target closure terms are displayed in \fref{fig:predictions}.
The M3 model using the normalised primitive quantities input set, a kernel size of $5\times5$, the sse loss function, and 64 maximum filters has been employed to generate the predicted closure terms (highest $\mathcal{CC}$ observed: $\mathcal{CC}(\mathcal{S}^R_x)=0.89,\, \mathcal{CC}(\mathcal{S}^R_y)=0.89$).
The predicted fields have been post-processed including a wake mask step constructed from the resolved vorticity field and a Gaussian filter to mitigate the noise in the raw CNN output (note that all the presented correlation values do not include this post-processing step).
This post-processing step is similar to process described in \sref{sec:ML_model} although in that case it is embedded in the CNN.

\fref{fig:predictions} shows that both large- and mid-scale structures are well captured by the model.
On the other hand, small-scale structures are not so well identified.
Similar issues are encountered in \cite{Lee2019}, where deep-learning models employed for recursively generating snapshots of flow past a circular cylinder at high $Re$ failed to fully reproduce small-scale structures.
Also, the noise observed in the $\mathcal{S}^{R,\,\mathrm{ML}}_y$ output motivates the inclusion of a wake mask, as described in \sref{sec:ML_model}.

{\renewcommand{\arraystretch}{1.2}
\begin{table}
\begin{center}
\begin{tabular}{ll}
\toprule
Architecture &\noindent \specialcell{M2 (highest correlation)\\M3 (best training consistency)} \\
Loss function & sse \\
Activation function & ReLU \\
Max. filters & 64 \\
Kernel size & $5\times5$ \\
Normalise input data & True \\
Input dataset & $\left\lbrace U, V, P \right\rbrace$, $\left\lbrace \nabla\vect{U}, \nabla P \right\rbrace$\\
\bottomrule
\end{tabular}
\end{center}
\caption{Summary of best results.}
\label{tab:best_model}
\end{table}}

\begin{figure*}[!ht]
\begin{multicols}{2}
\begin{subfigure}[t]{\linewidth}
    \includegraphics[width=\linewidth]{appendixD/predictions/SR_x_3693.0.pdf}\caption{$\mathcal{S}^R_x$}
\end{subfigure}\vspace{0.5cm}
\begin{subfigure}[t]{\linewidth}
    \includegraphics[width=\linewidth]{appendixD/predictions/SR_x_ML_raw_3693.0.pdf} \caption{$\mathcal{S}^{R,\,\mathrm{ML}}_x$ (raw)}
\end{subfigure}\vspace{0.5cm}
\begin{subfigure}[t]{\linewidth}
    \includegraphics[width=\linewidth]{appendixD/predictions/SR_x_ML_pp_3693.0.pdf} \caption{$\mathcal{S}^{R,\,\mathrm{ML}}_x$ (post-processed)}
\end{subfigure}
\begin{subfigure}[t]{\linewidth}
    \includegraphics[width=\linewidth]{appendixD/predictions/SR_y_3693.0.pdf}\caption{$\mathcal{S}^R_y$}
\end{subfigure}\vspace{0.5cm}
\begin{subfigure}[t]{\linewidth}
    \includegraphics[width=\linewidth]{appendixD/predictions/SR_y_ML_raw_3693.0.pdf}\caption{$\mathcal{S}^{R,\,\mathrm{ML}}_y$ (raw)}
\end{subfigure}\vspace{0.5cm}
\begin{subfigure}[t]{\linewidth}
    \includegraphics[width=\linewidth]{appendixD/predictions/SR_y_ML_pp_3693.0.pdf}\caption{$\mathcal{S}^{R,\,\mathrm{ML}}_y$ (post-processed)}
\end{subfigure}
\end{multicols}
\caption{Qualitative comparison between a snapshot of target fields (top), CNN raw predictions (middle), and CNN post-processed predictions (bottom).
The post-processing step removes high-frequency oscillations with a Gaussian filter and applies a wake detection filter constructed from the resolved vorticity field.}
\label{fig:predictions}
\end{figure*}
% ----------------------------------------------------------------
\end{document}


\title{Inference Networks}

\subsection{Inference Networks}

An inference network is a flexible construction for parameterizing
approximating distributions during inference.
They are used in Helmholtz machines \citep{dayan1995helmholtz}, deep
Boltzmann machines \citep{salakhutdinov2010efficient}, and
variational auto-encoders \citep{kingma2014auto,rezende2014stochastic}.

Recall that probabilistic models often have \emph{local} latent
variables, that is, latent variables associated with a data point; for
example, the latent cluster assignment of a data point or the hidden
representation of a text or image.
Variational
inference on models with local latent variables
requires optimizing over each variational factor,
\begin{equation*}
q(\mathbf{z}; \mathbf{\lambda}) = \prod_{n=1}^N q(z_n; \lambda_n),
\end{equation*}
where $z_n$ are the latent variables associated to a data point $x_n$.
The set of parameters $\mathbf{\lambda}=\{\lambda_n\}$ grows with the
size of data. This makes computation difficult when the data
does not fit in memory.
Further, a new set of parameters $\mathbf{\lambda}'=\{\lambda_n'\}$
must be optimized at test time, where there may be a new set of data
points $\{x_n'\}$.

An inference network replaces local variational parameters with global
parameters. It is a neural network
which takes $x_n$ as input and which outputs its local variational parameters
$\lambda_n$,
\begin{equation*}
q(\mathbf{z}\mid \mathbf{x}; \theta)
= \prod_{n=1}^N q(z_n \mid x_n; \lambda_n)
= \prod_{n=1}^N q(z_n; \lambda_n = \mathrm{NN}(x_n; \theta)).
\end{equation*}
This amortizes inference by only defining a set of \emph{global} parameters,
namely, the parameters of the neural network. These parameters have a
fixed size, making the memory complexity of inference a constant.
Further, the same set of parameters can be used at test time:
regardless of seeing new or old data points $x_n'$, we simply pass it
through the neural network (a forward pass) to obtain its
corresponding variational factor $q(z_n'; \lambda_n' =
\mathrm{NN}(x_n'; \theta))$.

Note that the inference network is an approximation: it is a common
misunderstanding that the inference network produces a more expressive
variational model. The inference network restricts the size of
parameters, meaning it can only do as well as an approximation as the
original variational distribution without the inference network.

There are often good reasons beyond computational reasons
for wanting to use an inference network, depending on the problem
setting.
Originating from
Helmholtz machines, they are motivated by the hypothesis
that "the human perceptual system is a statistical inference engine
whose function is to infer the probable causes of sensory input." The
inference network is this function.
In cognition, "humans and robots are in the setting of amortized
inference: they have to solve many similar inference problems, and can
thus offload part of the computational work to shared precomputation
and adaptation over time" \citep{stuhlmuller2013learning}.

\subsubsection{Implementation}

Inference networks are easy to build in Edward.
In the example below, a data point \texttt{x} is a 28 by 28 pixel
image (from MNIST).

We define a TensorFlow placeholder \texttt{x\_ph} for the neural
network's input, which is a batch of data points.
\begin{lstlisting}[language=Python]
x_ph = tf.placeholder(tf.float32, [M, 28 * 28])
\end{lstlisting}

We build a neural network using Keras.
It takes a 28 by 28 pixel image and outputs a $10$-dimensional
output, one for the mean (\texttt{mu}) and one for the standard
deviation (\texttt{sigma}).
\begin{lstlisting}[language=Python]
from edward.models import Normal
from keras.layers import Dense

hidden = Dense(256, activation='relu')(x_ph)
qz = Normal(loc=Dense(10)(hidden),
            scale=Dense(10, activation='softplus')(hidden))
\end{lstlisting}
The variational model is parameterized by the neural network's
output.

During each step of inference, we pass in a batch of data points to feed the
placeholder. Then we train the variational parameters (inference
network's parameters) according to
gradients of the variational objective.

An example script using this variational model can be found at
\href{https://github.com/blei-lab/edward/blob/master/examples/vae.py}
{\texttt{examples/vae.py}} in the Github repository.
An example with a convolutional architecture can be found at
\href{https://github.com/blei-lab/edward/blob/master/examples/vae_convolutional_prettytensor.py}
{\texttt{examples/vae\_convolutional\_prettytensor.py}} in the Github repository.
%We experiment with this model in the
%\href{/tutorials/variational-autoencoder}{variational auto-encoder} tutorial.

\subsubsection{Footnotes}

Inference networks are also known as recognition models, recognition
networks, or inverse mappings.

Variational models parameterized by
inference networks are also known as probabilistic encoders, in
analogy from coding theory to
\href{/tutorials/decoder}{probabilistic decoders}.
We recommend against this terminology,
in favor of making explicit the separation of model and inference.
That is,
variational inference with inference networks is a
general technique that can be applied to models beyond
probabilistic decoders.

This tutorial is taken from text originating in \citet{tran2016variational}.
We thank Kevin Murphy for motivating the tutorial as it is based
on our discussions, and also related discussion with Jaan Altosaar.

\subsubsection{References}\label{references}


\title{Linear Mixed Effects Models}

\subsection{Linear Mixed Effects Models}

With linear mixed effects models, we wish to model a linear
relationship for data points with inputs of varying type, categorized
into subgroups, and associated to a real-valued output.

We demonstrate with an example in Edward.
An interactive version with Jupyter notebook is available
\href{http://nbviewer.jupyter.org/github/blei-lab/edward/blob/master/notebooks/linear_mixed_effects_models.ipynb}{here}.

\subsubsection{Data}

We use the \texttt{InstEval} data set from the popular
\href{http://lme4.r-forge.r-project.org}{lme4 R package}
\citep{bates2015fitting}.
It is a data set of instructor evaluation ratings, where the inputs
(covariates) include categories such as \texttt{students} and
\texttt{departments}, and our response variable of interest is the
instructor evaluation rating.

\begin{lstlisting}[language=Python]
import pandas as pd
from observations import insteval

data, metadata = insteval("~/data")
data = pd.DataFrame(data, columns=metadata['columns'])

# s - students - 1:2972
# d - instructors - codes that need to be remapped
# dept also needs to be remapped
data['s'] = data['s'] - 1
data['dcodes'] = data['d'].astype('category').cat.codes
data['deptcodes'] = data['dept'].astype('category').cat.codes
data['y'] = data['y'].values.astype(float)

train = data.sample(frac=0.8)
test = data.drop(train.index)
\end{lstlisting}

The data set denotes:
\begin{itemize}
\item \texttt{students} as \texttt{s}
\item \texttt{instructors} as \texttt{d}
\item \texttt{departments} as \texttt{dept}
\item \texttt{service} as \texttt{service}
\end{itemize}

\begin{lstlisting}[language=Python]
s_train = train['s'].values
d_train = train['dcodes'].values
dept_train = train['deptcodes'].values
y_train = train['y'].values
service_train = train['service'].values
n_obs_train = train.shape[0]

s_test = test['s'].values
d_test = test['dcodes'].values
dept_test = test['deptcodes'].values
y_test = test['y'].values
service_test = test['service'].values
n_obs_test = test.shape[0]
\end{lstlisting}

\begin{lstlisting}[language=Python]
n_s = max(s_train) + 1  # number of students
n_d = max(d_train) + 1  # number of instructors
n_dept = max(dept_train) + 1  # number of departments
n_obs = train.shape[0]  # number of observations

print("Number of students: {}".format(n_s))
print("Number of instructors: {}".format(n_d))
print("Number of departments: {}".format(n_dept))
print("Number of observations: {}".format(n_obs))
\end{lstlisting}

\begin{lstlisting}
Number of students: 2972
Number of instructors: 1128
Number of departments: 14
Number of observations: 58737
\end{lstlisting}

\subsubsection{Model}

With linear regression, one makes an independence assumption where
each data point regresses with a constant slope among each other. In
our setting, the observations come from groups which may have
varying slopes and intercepts. Thus we'd like to build a model that
can capture this behavior \citep{gelman2006data}.

For examples of this phenomena:
\begin{itemize}
\item The observations from a single student are not independent of
each other. Rather, some students may systematically give low (or
high) lecture ratings.
\item The observations from a single teacher are not independent of
each other. We expect good teachers to get generally good ratings and
bad teachers to get generally bad ratings.
\item The observations from a single department are not independent of
each other. One department may generally have dry material and thus be
rated lower than others.
\end{itemize}

Typical linear regression takes the form
\begin{equation*}
\mathbf{y} = \mathbf{X}\beta + \epsilon,
\end{equation*}
where $\mathbf{X}$ corresponds to fixed effects with coefficients
$\beta$ and $\epsilon$ corresponds to random noise,
$\epsilon\sim\mathcal{N}(\mathbf{0}, \mathbf{I})$.

In a linear mixed effects model, we add an additional term
$\mathbf{Z}\eta$, where $\mathbf{Z}$ corresponds to random effects
with coefficients $\eta$. The model takes the form
\begin{align*}
\eta &\sim \mathcal{N}(\mathbf{0}, \sigma^2 \mathbf{I}), \\
\mathbf{y} &= \mathbf{X}\beta + \mathbf{Z}\eta + \epsilon.
\end{align*}
Given data, the goal is to infer $\beta$, $\eta$, and $\sigma^2$,
where $\beta$ are model parameters (``fixed effects''), $\eta$ are
latent variables (``random effects''), and $\sigma^2$ is a variance
component parameter.

Because the random effects have mean 0, the data's mean is captured by
$\mathbf{X}\beta$. The random effects component $\mathbf{Z}\eta$
captures variations in the data (e.g.  Instructor \#54 is rated 1.4
points higher than the mean).

A natural question is the difference between fixed and random effects.
A fixed effect is an effect that is constant for a given population. A
random effect is an effect that varies for a given population (i.e.,
it may be constant within subpopulations but varies within the overall
population). We illustrate below in our example:

\begin{itemize}
\item
Select \texttt{service} as the fixed effect. It is a binary covariate
corresponding to whether the lecture belongs to the lecturer's main
department. No matter how much additional data we collect, it
can only take on the values in $0$ and $1$.
\item
Select the categorical values of \texttt{students}, \texttt{teachers},
and \texttt{departments} as the random effects. Given more
observations from the population of instructor evaluation ratings, we
may be looking at new students, teachers, or departments.
\end{itemize}

In the syntax of R's lme4 package \citep{bates2015fitting}, the model
can be summarized as

\begin{lstlisting}[language=JSON]
y ~ 1 + (1|students) + (1|instructor) + (1|dept) + service
\end{lstlisting}
where \texttt{1} denotes an intercept term, \texttt{(1|x)} denotes a
random effect for  \texttt{x}, and \texttt{x} denotes a fixed effect.

\begin{lstlisting}[language=Python]
# Set up placeholders for the data inputs.
s_ph = tf.placeholder(tf.int32, [None])
d_ph = tf.placeholder(tf.int32, [None])
dept_ph = tf.placeholder(tf.int32, [None])
service_ph = tf.placeholder(tf.float32, [None])

# Set up fixed effects.
mu = tf.get_variable("mu", [])
service = tf.get_variable("service", [])

sigma_s = tf.sqrt(tf.exp(tf.get_variable("sigma_s", [])))
sigma_d = tf.sqrt(tf.exp(tf.get_variable("sigma_d", [])))
sigma_dept = tf.sqrt(tf.exp(tf.get_variable("sigma_dept", [])))

# Set up random effects.
eta_s = Normal(loc=tf.zeros(n_s), scale=sigma_s * tf.ones(n_s))
eta_d = Normal(loc=tf.zeros(n_d), scale=sigma_d * tf.ones(n_d))
eta_dept = Normal(loc=tf.zeros(n_dept), scale=sigma_dept * tf.ones(n_dept))

yhat = (tf.gather(eta_s, s_ph) +
        tf.gather(eta_d, d_ph) +
        tf.gather(eta_dept, dept_ph) +
        mu + service * service_ph)
y = Normal(loc=yhat, scale=tf.ones(n_obs))
\end{lstlisting}

\subsubsection{Inference}

Given data, we aim to infer the model's fixed and random effects.
In this analysis, we use variational inference with the
$\text{KL}(q\|p)$ divergence measure. We specify fully factorized
normal approximations for the random effects and pass in all training
data for inference. Under the algorithm, the fixed effects will be
estimated under a variational EM scheme.

\begin{lstlisting}[language=Python]
q_eta_s = Normal(
    loc=tf.get_variable("q_eta_s/loc", [n_s]),
    scale=tf.nn.softplus(tf.get_variable("q_eta_s/scale", [n_s])))
q_eta_d = Normal(
    loc=tf.get_variable("q_eta_d/loc", [n_d]),
    scale=tf.nn.softplus(tf.get_variable("q_eta_d/scale", [n_d])))
q_eta_dept = Normal(
    loc=tf.get_variable("q_eta_dept/loc", [n_dept]),
    scale=tf.nn.softplus(tf.get_variable("q_eta_dept/scale", [n_dept])))

latent_vars = {
    eta_s: q_eta_s,
    eta_d: q_eta_d,
    eta_dept: q_eta_dept}
data = {
    y: y_train,
    s_ph: s_train,
    d_ph: d_train,
    dept_ph: dept_train,
    service_ph: service_train}
inference = ed.KLqp(latent_vars, data)
\end{lstlisting}

One way to critique the fitted model is a residual plot, i.e., a
plot of the difference between the predicted value and the observed
value for each data point. Below we manually run inference,
initializing the algorithm and performing individual updates within a
loop. We form residual plots as the algorithm progresses. This helps
us examine how the algorithm proceeds to infer the random and fixed
effects from data.

To form residuals, we first make predictions on test data. We do this
by copying \texttt{yhat} defined in the model and replacing its
dependence on random effects with their inferred means. During the
algorithm, we evaluate the predictions, feeding in test inputs.

We have also fit the same model
\texttt{(y ~ service + (1|dept) + (1|s) + (1|d)},
fit on the entire InstEval dataset, specifically) in lme4. We
have saved the random effect estimates and will compare them to our
learned parameters.

\begin{lstlisting}[language=Python]
yhat_test = ed.copy(yhat, {
    eta_s: q_eta_s.mean(),
    eta_d: q_eta_d.mean(),
    eta_dept: q_eta_dept.mean()})
\end{lstlisting}

We now write the main loop.

\begin{lstlisting}[language=Python]
inference.initialize(n_print=20, n_iter=100)
tf.global_variables_initializer().run()

for _ in range(inference.n_iter):
  # Update and print progress of algorithm.
  info_dict = inference.update()
  inference.print_progress(info_dict)

  t = info_dict['t']
  if t == 1 or t % inference.n_print == 0:
    # Make predictions on test data.
    yhat_vals = yhat_test.eval(feed_dict={
        s_ph: s_test,
        d_ph: d_test,
        dept_ph: dept_test,
        service_ph: service_test})

    # Form residual plot.
    plt.title("Residuals for Predicted Ratings on Test Set")
    plt.xlim(-4, 4)
    plt.ylim(0, 800)
    plt.hist(yhat_vals - y_test, 75)
    plt.show()
\end{lstlisting}

\subsubsection{Criticism}

Above, we described a method for diagnosing the fit of the model via
residual plots.
Below we show the residuals for the model post-convergence. (For
intermediate residual plots, check the Jupyter notebook.)

\includegraphics[width=450px]{/images/linear-mixed-effects-models-fig0.png}

The residuals appear normally distributed with mean 0. The residuals
appear normally distributed with mean 0. This is a good
sanity check for the model.

We can also compare our learned parameters to those estimated by R's
`lme4`.

\begin{lstlisting}[language=Python]
student_effects_lme4 = pd.read_csv('data/insteval_student_ranefs_r.csv')
instructor_effects_lme4 = pd.read_csv('data/insteval_instructor_ranefs_r.csv')
dept_effects_lme4 = pd.read_csv('data/insteval_dept_ranefs_r.csv')

student_effects_edward = q_eta_s.mean().eval()
instructor_effects_edward = q_eta_d.mean().eval()
dept_effects_edward = q_eta_dept.mean().eval()
\end{lstlisting}

\begin{lstlisting}[language=Python]
plt.title("Student Effects Comparison")
plt.xlim(-1, 1)
plt.ylim(-1, 1)
plt.xlabel("Student Effects from lme4")
plt.ylabel("Student Effects from edward")
plt.scatter(student_effects_lme4["(Intercept)"],
            student_effects_edward,
            alpha=0.25)
plt.show()
\end{lstlisting}

\includegraphics[width=450px]{/images/linear-mixed-effects-models-fig1.png}

\begin{lstlisting}[language=Python]
plt.title("Instructor Effects Comparison")
plt.xlim(-1.5, 1.5)
plt.ylim(-1.5, 1.5)
plt.xlabel("Instructor Effects from lme4")
plt.ylabel("Instructor Effects from edward")
plt.scatter(instructor_effects_lme4["(Intercept)"],
            instructor_effects_edward,
            alpha=0.25)
plt.show()
\end{lstlisting}

\includegraphics[width=450px]{/images/linear-mixed-effects-models-fig2.png}

Great!  Our estimates for both student and instructor effects seem to
match those from `lme4` closely.  We have set up a slightly different
model here (for example, our overall mean is regularized, as are our
variances for student, department, and instructor effects, which is not
true of `lme4`s model), and we have a different inference method, so we
should not expect to find exactly the same parameters as `lme4`.  But
it is reassuring that they match up closely!

\begin{lstlisting}[language=Python]
#  Add in the intercept from R and edward
dept_effects_and_intercept_lme4 = 3.28259 + dept_effects_lme4["(Intercept)"]
dept_effects_and_intercept_edward = mu.eval() + dept_effects_edward

plt.title("Departmental Effects Comparison")
plt.xlim(3.0, 3.5)
plt.ylim(3.0, 3.5)
plt.xlabel("Department Effects from lme4")
plt.ylabel("Department Effects from edward")
plt.scatter(dept_effects_and_intercept_lme4,
            dept_effects_and_intercept_edward,
            s=0.01 * train.dept.value_counts())
plt.show()
\end{lstlisting}

\includegraphics[width=450px]{/images/linear-mixed-effects-models-fig3.png}

Our department effects do not match up nearly as well with those from `lme4`.
There are likely several reasons for this:

\begin{itemize}
\item
  We regularize the overall mean, while `lme4` doesn't. This causes the
  Edward model to put some of the intercept into the department effects,
  which are allowed to vary more widely since we learn a variance
\item
  We are using 80% of the data to train the edward model, while our `lme4`
  estimate uses the whole `InstEval` data set
\item
  The department effects are the weakest in the model and difficult to
  estimate.
\end{itemize}

\subsubsection{Acknowledgments}

We thank Mayank Agrawal for writing the initial version of this
tutorial.

\subsubsection{References}\label{references}


\documentclass[a4paper,11pt]{article}

\input{preamble/settings}
\input{preamble/commands}
\input{preamble/math}
\input{preamble/semantic}

% ## title
\title{Title}
\author{
  Author\thanks{thanks}\\
  \texttt{email1}
  \and
  Author2\thanks{thanks}\\
  \texttt{email2}
}
\date{}

% ## bibliography using `biblatex`
% \addbibresource{bibliography/bib.bib}

\begin{document}
  \maketitle

  % ## common
  \tableofcontents
  % \listoftables
  % \listoffigures
  % {\let\cleardoublepage\clearpage \chapter{Chapter}\label{ch:label}}
  % \listoftodos

  \section{Essential} \label{sec:label}
  % ## annotation
  \todo{\textbackslash todo}
  \marginpar{\textbackslash marginpar}
  \mcomment{\small \textbackslash mcomment}

  % ## equation
  \begin{IEEEeqnarray}{rCl}
    \binom{n}{k}
      &=& \frac{n!}{k!(n-k)!} \label{eq:label1}
      \\[0.5em]
      &=& \frac{1}{2\pi i}\oint\limits_{\Gamma} \frac{ (1+z)^n }{z^{k+1}} \ud z
      \label{eq:label2}
  \end{IEEEeqnarray}

  % ## table
  \renewcommand\arraystretch{1.25}
  % break lines within cells with fixed-width columns using `array`

  \begin{table}[h]
    \centering
    \caption{Caption}
    \label{tab:label}
    \begin{tabular}{cc}
      \toprule
      A & \R \footnotemark \\
      \midrule
      a & b \\
      c & d \\
      \midrule
      e & f \\
      g & h \\
      \bottomrule
    \end{tabular}
    % OR caption & label here ...
  \end{table}

  \footnotetext{footnotemark--footnotetext}

  % ## figure
  \insertfig{death-star}{Caption for ToC}{Caption}{label}{0.2}  % `fig:label`

  % ## cite
  \noindent TensorFlow\footnote{footnote} \citep{tf}, \citet{tf}.\\
  Section \ref{sec:label} on a page \pageref{sec:label},
  table \ref{tab:label}, figure \ref{fig:label},
  equations \eqref{eq:label1} and \eqref{eq:label2}.

  \section{Other CO\texorpdfstring{\textsubscript{2}}{2}}
  \subsection*{Subfigures}
  % use `minipage` for individual figures numbering

  \begin{figure}[htb]
    \centering
    %
    \begin{subfigure}[b]{.475\textwidth}
      \frame{\includegraphics[width=\textwidth]{death-star}}
      \caption{Caption 1}  % or \subcaption
      \label{fig:1}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{.475\textwidth}
      \includegraphics[width=\textwidth]{death-star}
      \caption{Caption 2}
      \label{fig:2}
    \end{subfigure}
    %
    \begin{subfigure}[b]{.3\textwidth}
      \frame{\includegraphics[width=\textwidth]{death-star}}
      \caption{Caption 3}
      \label{fig:3}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{.3\textwidth}
      \includegraphics[width=\textwidth]{death-star}
      \caption{Caption 4}
      \label{fig:4}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{.3\textwidth}
      \includegraphics[width=\textwidth]{death-star}
      \caption{Caption 5}
      \label{fig:5}
    \end{subfigure}
    %
    \caption{The caption. \emph{Top}: top. \emph{Bottom}: bottom.}
    \label{fig:subfigures}
  \end{figure}

  \subsection*{Proof}
    \begin{proof}[\unskip\nopunct]
      The proof is easy and is left to a reader.
    \end{proof}
    % \noindent\answertext{42}
    % \\
    % \noindent\answer{ \dd[\log (1+\exp x) ]{x} }

  \subsection*{Test math}
    \begin{gather*}
      {\textstyle \underset{\mu}{\sum} \; \sum_{\mu}} \; \R^{n\times m}\; \bra{\frac{\Psi}{1}} \; \ket{\frac{\Psi}{1}} \;
      \braket{\frac{\Psi}{1}} \;
      \matrixelement{n}{\prod_k U_k}{\frac{x}{1}} \;
      \mel{n}{\prod_k U_k}{\frac{x}{1}} \\
      \text{Normal}(\mathrm{x} \mid \mu, \sigma^2 ) \\
      \textnormal{Normal}(\mathrm{x} \mid \mu, \sigma^2 ) \\
      \mathrm{Normal}(\mathrm{x} \mid \mu, \sigma^2 ) \\
      Y \sim \text{U}[0,1] \propto \varpropto \text{Beta}(a,b\;;\;c,d) \;
      \Gamma \p{x \mathrel{\bigg |} \alpha + \sum_{k=0}^n \theta_k}
      \mathcal{N}(\mathrm{x} \mid \mu, \sigma^2 ) \\
      \sum_{\mathclap{n=-\infty}}^{+\infty} f(x) \geqslant \geq \ge \med X \\
      \eps + \e^{-\frac{(x-2)^2}{2\sigma ^2}} + \const{} \\
      \dot{a} \epsilon \phi \varphi \\
      \not\propto \not\subset \not\in \notin \\
      \equiv \doteq \approx \subset \supset \ni \mid \parallel \neq \ne \\
      \ln p = \Tr A = \tr A = \var X = \KL\p{P \parallel Q} = \DKL\p{P \parallel Q} \\
      \star \ast \circ \bullet \oplus \otimes \odot \dagger \ddagger \text{\dag \ddag} \\
      \bigoplus \bigotimes \bigodot \bigcup \bigcap \\
      \leftarrow \gets \rightarrow \to \mapsto
      \Leftarrow \Rightarrow \Longleftrightarrow \iff \overrightarrow{AB} \rightrightarrows \\
      []\lbrack \rbrack \lbrace \rbrace \langle \rangle
      | \vert \| \Vert \lfloor \rfloor \arrowvert \Arrowvert \\
      \ell \emptyset \Re \Im \bot \top \angle \Box \\
      \thicksim \thickapprox \backsim \varpropto \risingdotseq \fallingdotseq \\
      \hbar \square \blacksquare \bigstar \varnothing
    \end{gather*}

    \begin{equation}
      \begin{Vmatrix}
        1 & 2 \\
        3 & 4
      \end{Vmatrix}
      =
      \abs{\oint_{A}^{B} f(z) \ud z}
      =
      \frac{\d u}{\d x}
      = \mathcal{F}\mathfrak{F}
      = \frac{\displaystyle \sum a_{ij} }{\displaystyle \sum b_{ij} }
      = \sum_{\mathclap{\text{big long thing}}} a_k
      = \frac{\displaystyle \P\B{\frac{X}{\E X} \leq \epsilon}}
             {\displaystyle \Pr{\mathrm{Poisson}(\lambda=3) > 5}}
    \end{equation}

    \begin{equation}
      \pd \cdot \pdd{x} \cdot \pdd[f]{x} \cdot \pdd[f][3]{x}
      \cdot \evalat{\pdd{x}\frac{x^2+1}{x^3+1}}{x=0}
      =
      \d \cdot \dd{x} \cdot \dd[f]{x} \cdot \dd[f][3]{x}
      \cdot \evalat{\dd{x}\frac{x^2+1}{x^3+1}}{x=0}
    \end{equation}

    \begin{equation}
      \o{a} \enskip
      A \stackrel{*}{\approx} B \enskip
      \sum_{\substack{0<i<n \\ j \neq i}}f(i) \enskip
      \sqrt[3]{P(x)+Q(x)} \enskip
      \frac{3}{8} \tfrac{3}{8} \dfrac{3}{8} 3/8 \enskip
      x=x\phantom{=}x\mathbin{=}x
    \end{equation}

  \subsection*{Math fonts}
    \begin{gather}
      \mathrm{ABCDEFabcdef} \tag{mathrm} \\
      \mathbf{ABCDEFabcdef} \tag{mathbf} \\
      \mathsf{ABCDEFabcdef} \tag{mathsf} \\
      \mathtt{ABCDEFabcdef} \tag{mathtt} \\
      \mathit{ABCDEFabcdef} \tag{mathit} \\
      \mathcal{ABCDEFabcdef} \tag{mathcal} \\
      \mathnormal{ABCDEFabcdef} \tag{mathnormal}
      \\[0.5em]
      \boldsymbol{ABCabc \Gamma\Omega\Xi \gamma\omega\xi } \tag{boldsymbol} \\
      \mathscr{ABCDEFabcdef} \tag{mathscr} \\
      \mathfrak{ABCDEFabcdef} \tag{mathfrak} \\
      \mathbb{ABCDEFabcdef12345} \tag{mathbb} \\
      \mathbbm{ABCDEFabcdef12345} \tag{mathbbm}
    \end{gather}

  \subsection*{Text fonts}
    \noindent
    \textrm{ABCDEFabcdef}  % RoMan
    \textsf{ABCDEFabcdef}  % Sans seriF
    \texttt{ABCDEFabcdef}  % Type wriTer
    \\[0.5em]
    \textmd{ABCDEFabcdef}  % MeDium
    \textbf{ABCDEFabcdef}  % Bold Face
    \\[0.5em]
    \textup{ABCDEFabcdef}  % UPright
    \textit{ABCDEFabcdef}  % ITalic
    \textsl{ABCDEFabcdef}  % SLanted
    \textsc{ABCDEFabcdef}  % Small Caps
    \\[0.5em]
    \textnormal{ABCDEFabcdef}  % document font
    \emph{ABCDEFabcdef}    % EMPHasize

  \subsection*{General formatting}
    \begin{itemize}[noitemsep]
      \item x \hspace{\stretch{1}} y \hspace{\stretch{1}} z \hspace*{\stretch{1}}
      \item ``quote''
      \item Ph.~D.
      \item Ph.\ D.
      \item Ph. D.
      \item A. B
      \item A\@. B
      \item \verb*+yo wazup+
    \end{itemize}

  \subsection*{Semantic}
  \begin{gather}
    \alg{RMSProp} \; \alg{Adam} \; \cmd{pmatrix} \\
    \s{a} \; \w{x} \; \bs{\alpha} \; \m{A} \; \q{A} \\
    \g{not}\text{ gate} \; \q{CNOT}\text{ gate} \\
    \c{X} \; \c{Y} \; \c{D}
  \end{gather}

  % # Bibliography
  \newpage
  % ## bibliography using `biblatex`
  % \setlength\bibitemsep{1.5\itemsep}
  % \printbibliography

  % ## bibliography using `natbib`
  \setlength\bibsep{1.5\itemsep}
  \renewcommand{\bibname}{Bibliography}
  \bibliography{bibliography/bib}

\end{document}


% Part: first-order-logic
% Chapter: natural-deduction
% Section: propositional-rules

\documentclass[../../../include/open-logic-section]{subfiles}

\begin{document}

\olfileid{fol}{ntd}{prl}

\olsection{Propositional Rules}

\subsection{Rules for $\land$}

\begin{defish}
\AxiomC{$!A$}
\AxiomC{$!B$}
\RightLabel{\Intro{\land}}
\BinaryInfC{$!A \land !B$}
\DisplayProof
\hfill
\begin{tabular}{r}
\AxiomC{$!A \land !B$}
\RightLabel{\Elim{\land}}
\UnaryInfC{$!A$}
\DisplayProof
\\[3ex]
\AxiomC{$!A \land !B$}
\RightLabel{\Elim{\land}}
\UnaryInfC{$!B$}
\DisplayProof
\end{tabular}
\end{defish}

\subsection{Rules for $\lor$}

\begin{defish}
\begin{tabular}{r}
\AxiomC{$!A$}
\RightLabel{\Intro{\lor}}
\UnaryInfC{$!A \lor !B$}
\DisplayProof
\\[3ex]
\AxiomC{$!B$}
\RightLabel{\Intro{\lor}}
\UnaryInfC{$!A \lor !B$}
\DisplayProof
\end{tabular}
\hfill
\AxiomC{$!A \lor !B$}
\AxiomC{$\Discharge{!A}{n}$}
\DeduceC{$!C$}
\AxiomC{$\Discharge{!B}{n}$}
\DeduceC{$!C$}
\DischargeRule{\Elim{\lor}}{n}
\TrinaryInfC{$!C$}
\DisplayProof
\end{defish}

\subsection{Rules for $\lif$}

\begin{defish}
\AxiomC{$\Discharge{!A}{n}$}
\DeduceC{$!B$}
\DischargeRule{\Intro{\lif}}{n}
\UnaryInfC{$!A \lif !B$}
\DisplayProof
\hfill
\AxiomC{$!A \lif !B$}
\AxiomC{$!A$}
\RightLabel{\Elim{\lif}}
\BinaryInfC{$!B$}
\DisplayProof
\end{defish}

\subsection{Rules for $\lnot$}

\begin{defish}
\AxiomC{$\Discharge{!A}{n}$}
\noLine
\DeduceC{$\lfalse$}
\DischargeRule{\Intro{\lnot}}{n}
\UnaryInfC{$\lnot !A$}
\DisplayProof
\hfill
\AxiomC{$\lnot !A$}
\AxiomC{$!A$}
\RightLabel{\Elim{\lnot}}
\BinaryInfC{$\lfalse$}
\DisplayProof
\end{defish}

\subsection{Rules for $\lfalse$}

\begin{defish}
\AxiomC{$\lfalse$}
\RightLabel{\FalseInt}
\UnaryInfC{$!A$}
\DisplayProof
\hfill
\AxiomC{$\Discharge{\lnot !A}{n}$}
\DeduceC{$\lfalse$}
\DischargeRule{\FalseCl}{n}
\UnaryInfC{$!A$}
\DisplayProof
\end{defish}

Note that $\Intro{\lnot}$ and $\FalseCl$ are very similar: The
difference is that $\Intro{\lnot}$ derives a negated
!!{sentence}~$\lnot !A$ but $\FalseCl$ a positive !!{sentence}~$!A$.

\end{document}


% -*- root: 00-main.tex -*-
\section*{Conclusion}
\label{sec:regseg-conclusion}

\Regseg{} is a variational framework for the simultaneous segmentation and
  registration of 3D \gls*{dmri} data obtained from the human brain, where within-subject
  anatomical information is used as a reference.
The registration method segments the target multivariate image into several competing regions, which are
  defined explicitly by their limiting surfaces.
The surfaces are active and they evolve on a free-form deformation field supported by the B-spline basis.
A descent optimization strategy is guided by shape gradients computed on the current partition
  of the target image.
\Regseg{} uses active contours without edges and it searches for
  homogeneous regions within the image.
We tested \regseg{} using digital phantoms by simulating \gls*{t1} and \gls*{t2} \acrshort*{mri}
  warped with smooth and random deformations.
The resulting misregistration of the contours was significantly lower than the image resolution
  of the phantoms.

We proposed \regseg{} for simultaneously segmenting and registering \gls*{dmri} data to
  their corresponding \gls*{t1} image from the same subject.
We demonstrated the accuracy of the proposed method based on visual assessments of the results
  obtained by \regseg{} and cross-comparisons with a widely used technique.
Moreover, \regseg{} does not require any images in addition to the minimal acquisition protocol,
  which only utilizes \gls*{t1} and \gls*{dmri}.
As well as the proposed application to \gls*{dmri} data, other potential uses of \regseg{} are
  atlas-based segmentation and tracking objects in time-series.

\section*{Availability and reproducibility statement}
\label{sec:regseg-availability}
We considered the reproducibility of our results as a design requirement.
Therefore, we used real data obtained from the Human Connectome Project \citep{essen_human_2012}
  and all of the software utilized in this study is also publicly available.
\Regseg{} was implemented on top of ITK-4.6 (Insight Registration and
  Segmentation Toolkit, \url{http://www.itk.org}).
The evaluation instruments (\autoref{fig:regseg-evworkflows}) were implemented using
  \emph{nipype} \citep{gorgolewski_nipype_2011} to assess their reproducibility.
All of the research elements (data, source code, figures, manuscript sources, etc.) involved in this study
  are publicly available under a unique package \citep{esteban_regseg_2015}.


%
% CMPT 354: Database Systems I - A Course Overview
% Section: Functional Dependencies
%
% Author: Jeffrey Leung
%

\section{Functional Dependencies}
	\label{sec:functional-dependencies}
\begin{easylist}

	& \emph{Functional dependency:} For any given value of an attribute(s), there can only be a single value for a set of attributes
		&& Similar to the notion of a \hyperref[subsec:design:identification-of-tables]{superkey} for a set of attributes
		&& $x$ functionally determines $y$ and $y$ is functionally determined by $x$ if there can only be one $y$ value for a given $x$ value
		&& Mathematical definition: If $R$ is a relation and $\alpha \subseteq R$ and $\beta \subseteq R$, then $\alpha \rightarrow \beta$
		&& E.g. All of the same car models have the same capacity; therefore, capacity is functionally determined by the model. \\
		model $\rightarrow$ capacity
		&& Used to:
			&&& Specify additional constraints
			&&& Decompose tables to reduce repeated data
		&& Statement about all possible legal instances of a relation
			&&& Relation $R$ satisfies the set of functional dependencies $F$ if $R$ is legal under $F$

	& %TODO Functional dependencies notes in NORMALIZATION
	& Primary key is a special case of a functional dependency
		&& If X $\rightarrow$ Y on R...

	& Set of functional dependencies can be used to identify additional functional dependencies
		&& F implies the additional functional dependencies

\end{easylist}
\subsection{Armstrong's Axioms}
	\label{subsec:functional-dependencies:armstrongs-axioms}
\begin{easylist}

	& Named after William Armstrong

	& Axioms:
		&& Where X and Y are sets of attributes
		&& \emph{Reflexivity (A-Axioms):} If X $\supseteq$ Y, then X $\rightarrow$ Y
			&&& E.g. sin, firstName $\rightarrow$ sin
			&&& E.g. sin $\rightarrow$ sin
			&&& \emph{Trivial functional dependency:} X $\rightarrow$ Y where X $\supseteq$ Y
		&& \emph{Augmentation (A-Axioms):} If X $\rightarrow$ Y, then XZ $\rightarrow$ Y, Z for any Z
			&&& E.g. If sin $\rightarrow$ firstName, then sin, lastName $\rightarrow$ firstName, lastName
		&& \emph{Transitivity (A-Axioms):} If X $\rightarrow$ Y and Y $\rightarrow$ Z, then X $\rightarrow$ Z %TODO
			&&& E.g. If sin $\rightarrow$ age and age $\rightarrow$ seniority, then sin $\rightarrow$ seniority

	& Derivable axioms:
		&& \emph{Union:} %TODO
		&& \emph{Decomposition (A-Axioms):} If X $\rightarrow$ YZ, then X $\rightarrow$ Y and X $\rightarrow$ Z
			&&& Proof: %TODO
		&& \emph{Pseudo-transitivity (A-Axioms):} If X $\rightarrow$ Y and WY $\rightarrow$ Z, then XW $\rightarrow$ Z
			&&& Proof: See... %TODO ref? don't use tabular?

			\begin{tabular}{ l }
			Assume X $\rightarrow$ Y. \\
			By the axiom of augmentation, XW $\rightarrow$ WY. \\
			Assume WY $\rightarrow$ Z. \\
			By the axiom of transitivity, XW $\rightarrow$ Z.
			\end{tabular}

	& \emph{F:} Set of identified functional dependencies
	& \emph{F\textsuperscript{+}:} Closure of \emph{F}; all functional dependencies implied by \emph{F}
	& \emph{X\textsuperscript{+}:} Attribute closure of the set of attributes in X; list of attributes containing all attributes that are functionally dependent on X

	& \emph{Sound:} Axioms which do not generate incorrect functional dependencies
	& \emph{Complete:} Axioms which allow F\textsuperscript{+} to be generated from a given F

\end{easylist}
\subsection{Removing Redundant Functional Dependencies}
	\label{subsec:functional-dependencies:removing-redundant-functional-dependencies}
\begin{easylist}

	& \emph{Extraneous attribute:} Attribute \
		&& Attribute $a$ is extraneous in $X$ if $a \in X$ %TODO notes

	& Testing if attribute $a \in X$ is extraneous in $X$, compute $(\{X\} - a)^{+}$ with... %TODO

	& \emph{Canonical cover (of F):} Minimal set of functional dependencies equivalent to \emph{F} without redundant or partial dependencies
		&& F logically implies all dependencies in F\textsubscript{c}; \\
		F\textsubscript{c} logically implies all dependencies in F; \\
		no functional dependency in F\textsubscript{c} contains an extraneous attribute; \\
		each left side of a dependency in F\textsubscript{c} is unique

		&& Denoted by \emph{F\textsubscript{c}}
	%TODO more notes

\end{easylist}
\clearpage


\documentclass[letterpaper]{article}

\usepackage{fullpage}
\usepackage{nopageno}
\usepackage{amsmath}
\usepackage{amssymb}
\allowdisplaybreaks

\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}

\begin{document}
\title{Notes}
\date{September 15, 2014}
\maketitle
\section*{assignment}
Section 1.4: \# 17, 19, 20, 23, 24, 27.
\renewcommand{\labelenumi}{1.\arabic{enumi}}
\renewcommand{\labelenumii}{\arabic{enumii}.}
\renewcommand{\labelenumiii}{(\alph{enumiii})}
\begin{enumerate}
\setcounter{enumi}{3}
\item
  \begin{enumerate}
  \setcounter{enumii}{16}
  \item
  Using the formula for $\varphi(n)$, compute $\varphi(27),\varphi(81),$ and $\varphi(p^\alpha),$ where $p$ is a prime number. Give a proof that the formula for $\varphi(n)$ is valid when $n=p^{\alpha},$ where $p$ is a prime number.

  \begin{align*}
    \varphi(27)&=27(1-\frac{1}{3})=18\\
    \varphi(81)&=81(1-\frac{1}{3})=54\\
    \varphi(p^{\alpha})&=p^{\alpha}(1-\frac{1}{p})=p^{\alpha}\frac{p-1}{p}=p^{\alpha}-p^{\alpha-1}
  \end{align*}
  The result can be obtained by observing that the integers less than $p^{\alpha}$ that are not relatively prime to $p^{\alpha}$ are
  \setcounter{enumii}{18}
  \item
  Find all integers $n>1$ such that $\varphi(n)=2$
  \begin{align*}
    \varphi(n)&=n(1-\frac{1}{p1})(1-\frac{1}{p_2})\dots(1-\frac{1}{p_k})\\
    p1p2p3...pk\le n&\therefore2=(p1-1)(p2-l)\dots(pk-1)
  \end{align*}
  \setcounter{enumii}{19}
  \item
  \begin{align*}
    \varphi(1)&=1\\
    \varphi(p^n)&=p^n-p^{n-1}\text{ from 17}\\
    \varphi(p^{n+1})&=p^{n+1}-p^{n}\\
    p^{n+1}&=\varphi(p^{n+1})+p^n
  \end{align*}
  so by induction
  \setcounter{enumii}{22}
  \item
  Show that if $n>1$ then the sum of all positive integers less than $n$ and relatively prime to $n$ is $n\varphi(n)/2$. that is $\sum\limits_{0<a<n,(a,n)=1}{a}=n\varphi(n)/2$

  observation:$(a,n)=1\Leftrightarrow(n-a,n)=1$

  $\sum\limits_{0<a<n,(a,n)=1}{a}=\sum\limits_{0<a<n,(a,n)=1}{n-a}$ just summed in opposite oreder, because $0<a<n$
  $\sum\limits_{0<a<n,(a,n)=1}{a}=\frac{1}{2}\sum\limits_{0<a<n,(a,n)=1}{a+n-a}$. Now $\varphi(n)$ is defined as the number of a's such that $(a,n)=1, 0<a<n$. So $\sum\limits_{0<a<n,(a,n)=1}{a}=\frac{1}{2}\varphi(n)n$
  \item

  \setcounter{enumii}{26}
  \item
  \end{enumerate}
\end{enumerate}
\section*{2.3 permutations}
a function that maps a set $S$ to $S$ is a permutation if the function is one to one and onto (bijective)

$\text{Sym}(S)=\{\sigma:S\to S|\sigma\text{ bijective}\}$.

special case: $S$ is finite with $n$ elements. $S=\{1,2,\dots,n\}$ for simplified notation.

observations:
\begin{enumerate}
\item
$\sigma,\gamma\in\text{Sym}(S)$ then $\sigma\circ\gamma\in\text{Sym(S)}$.
\item
$1_S:S\to S$, $1_S\in\text{Sym}(S)$
\item
$\sigma\in\text{Sym}(S)\Rightarrow\sigma^{-1}\in\text{Sym}(S)$ and $\sigma\sigma^{-1}=\sigma^{-1}\sigma=1_S$
\end{enumerate}
notation for the case $S=\{1,2,\dots,n\}$
\begin{align*}
\sigma=\left(
  \begin{aligned}
    1&&2&&\dots&&n-1&&n\\
    \sigma(1)&&\sigma(2)&&\dots&&\sigma(n-1)&&\sigma(n)
  \end{aligned}
  \right)
\end{align*}
  Notation: $S_n=\text{Sym}(S)$ while $S$ has $n$ elements

  observe $\left\lvert S_n\right\rvert=n!$
  \begin{align*}
    \sigma&=\left(\begin{aligned}
      1,2,3,4\\
      3,4,2,1
    \end{aligned}\right)&
    \tau&=\left(\begin{aligned}
      1,2,3,4\\
      2,1,3,1
    \end{aligned}\right)
    \sigma\tau&=\left(\begin{aligned}
      1,2,3,4\\
      4,3,2,1
    \end{aligned}\right)
    \sigma^{-1}&=\left(\begin{aligned}
      1,2,3,4\\
      4,3,1,2
    \end{aligned}\right)
  \end{align*}

\subsection*{cycles}
let $\sigma\in S_n$. $\sigma$ is called a cycle of length $k$ if there exist $a_1,a_2,\dots,a_k\in\{1,2,\dots,n\}$ such that $\sigma(a_1)=a_2,\sigma(a_2)=a_3,\dots\sigma(a_k)=a_1$ and $\sigma(x)=x$ for $x\ne\{a_1,a_2,\dots,a_k\}$
\begin{align*}
    \sigma&=\left(\begin{aligned}
      1,2,3,4\\
      2,4,3,1
    \end{aligned}\right)=(1,2,4)\in S_4
\end{align*}
this is a cycle of length 3

definition of disjoint cycles

observe that for $\sigma\in S_n$ be a cycle of length $k$ then $\sigma^k=1_S$ (identity). In fact k is the smallest positive integer that has this property.

\subsection*{proposition}
take two disjoint cycles $\sigma,\tau$. then $\sigma\tau=\tau\sigma$ (note that permutations are not in general commutative).

\subsubsection*{proof}
take $x\in\{1,\dots,n\}$. if $x=a_i$ for some $i$. then $\sigma\tau(a_i)=a_{i+1}$ because

\section*{main theorem}
every permutation $m$ in $S_n$ can be written as a product of disjoint cycles. moreover, the cycles of length at least two that appear are unique. note that cycles of length one are identity, multiply them all day, nothing changes.

$\sigma=(2143)=(12)(34)$
\end{document}


\chapter{Congruence}

Look at this picture of two geometric figures.

\begin{tikzpicture}

  	\draw [thick] (-1,-4) --  (0, -2);
	\draw [thick] (0,-2) --  (-1, 0) ;
	\draw [thick] (-1,0) --  (-3, -1);
	\draw [thick] (-3,-1) --  (-1, -4);

	\draw [thick] (-1,1) --  (1, 0);
	\draw [thick] (1,0) --  (3, 1) ;
	\draw [thick] (3,1) --  (2, 3);
	\draw [thick] (2,3) --  (-1, 1);

  \draw[help lines, step = 1cm] (-4, -4) grid (4, 4);

\end{tikzpicture}

They are the same shape, right? If you cut one out with scissors, it
would lay perfectly on top of the other. In geometry, we say they are
\emph{congruent}.

What is the official definition of ``congruent''? Two geometric
figures are congruent if you can transform one into the other using
only rigid transformations.  So, what are rigid transformations?

A transformation is \emph{Rigid} if it doesn't change the distances
between the points or the measure of the angles between the lines they
form. These are all rigid transformations:
\begin{itemize}
\item Translations
\item Rotations
\item Reflection
\end{itemize}

If you once again imagine cutting one figure out with scissors and trying to match it with the second:
\begin{itemize}
\item Translations - sliding the cutout left and right and up and down
\item Rotations	- rotating the cutout clockwise and counterclockwise
\item Reflection - flipping the piece of paper over
\end{itemize}

A transformation is rigid if and only if it is some combination of translations, rotations, and reflections.

\section{Triangle Congruency}

If the sides of two triangles have the same length, the triangles must be congruent:

\begin{tikzpicture}
  	\draw [thick] (-2,0) -- node[outer sep = 1pt, right]{4 cm}  (-2, 4) ;
	\draw [thick] (-2,4) --  node[outer sep = 3pt, above]{2 cm} (-4, 3);
	\draw [thick] (-4,3) --  node[outer sep = 2pt, left]{3 cm} (-2, 0);

	\draw [thick] (-1,1) --  node[outer sep = 2pt, below]{4 cm}  (3, 1) ;
	\draw [thick] (3,1) --  node[outer sep = 2pt, right]{2 cm} (2, 3);
	\draw [thick] (2,3) --  node[outer sep = 4pt, above]{3 cm} (-1, 1);
\end{tikzpicture}

To be precise, the Side-Side-Side Congruency Test says that two triangles are
congruent if three sides in one triangle are the same length as the
corresponding sides in the other. We usually refer to this as the SSS test.

Note that two triangles with all three angles equal are not necessarily congruent.
For example, here are two triangles with the same interior angles, but they are
different sizes:

\begin{tikzpicture}
  \coordinate [circle, fill, inner sep=1pt] (a1) at (0,0) ;
  \coordinate [circle, fill, inner sep=1pt] (b1) at (4,0) ;
  \coordinate [circle, fill, inner sep=1pt] (c1) at (3,2) ;

   \coordinate [circle, fill, inner sep=1pt] (a2) at (5,0) ;
  \coordinate [circle, fill, inner sep=1pt] (b2) at (11,0) ;
  \coordinate [circle, fill, inner sep=1pt] (c2) at (9.5,3) ;
        \draw (a1) --   (b1);
        \draw (b1) -- (c1);
        \draw (c1)--  (a1);
        \pic [draw, "$63^\circ$", angle eccentricity=1.5] {angle = c1--b1--a1};
        \pic [draw, "$34^\circ$", angle eccentricity=2.0] {angle = b1--a1--c1};
        \pic [draw, "$83^\circ$", angle eccentricity=1.5] {angle = a1--c1--b1};

       \draw (a2) --  (b2);
        \draw (b2) -- (c2);
        \draw (c2)--  (a2);
        \pic [draw, "$63^\circ$", angle eccentricity=1.5] {angle = c2--b2--a2};
        \pic [draw, "$34^\circ$", angle eccentricity=2.0] {angle = b2--a2--c2};
        \pic [draw, "$83^\circ$", angle eccentricity=1.5] {angle = a2--c2--b2};

\end{tikzpicture}

These triangles are not congruent, but they are \emph{similar}. That
is, they have the same shape, but are not necessarily the same
scale.

If you know two angles, you can calculate the third. So it makes sense
to say ``If two triangles have two angles that are equal, they are
similar triangles.''  And if two similar triangles have one side that
is equal in length, they must be the same scale -- so they are
congruent. Thus, the Side-Angle-Angle Congruency Test says that
two triangles are congruent if two angles and side match.

What if you know that two triangles have two sides that are the same
length and that the angle between them is also equal?

\begin{tikzpicture}
  \coordinate [circle, fill, inner sep=1pt] (a1) at (0,0) ;
  \coordinate [circle, fill, inner sep=1pt] (b1) at (4,0) ;
  \coordinate [circle, fill, inner sep=1pt] (c1) at (3,2) ;

  \coordinate [circle, fill, inner sep=1pt] (a2) at (5,0) ;
  \coordinate [circle, fill, inner sep=1pt] (b2) at (9,0) ;
  \coordinate [circle, fill, inner sep=1pt] (c2) at (8,2) ;

  \draw (a1) --  node[outer sep = 0.5pt, below]{4}  (b1);
  \draw [dashed] (b1) -- (c1);
  \draw (c1)--  node[outer sep = 5pt, above]{3.5} (a1);
  \pic [draw, "$34^\circ$", angle eccentricity=2.0] {angle = b1--a1--c1};

  \draw (a2) --  node[outer sep = 0.5pt, below]{4}  (b2);
  \draw [dashed] (b2) -- (c2);
  \draw (c2)--  node[outer sep = 5pt, above]{3.5} (a2);
  \pic [draw, "$34^\circ$", angle eccentricity=2.0] {angle = b2--a2--c2};
\end{tikzpicture}

Yes, they must be congruent. This is the Side-Angle-Size Congruency Test.

What if the angle isn't the one between the two known sides? If it is
a right angle, you can be certain the two triangles are congruent.
(How do I know? Because the Pythagorean Theorem tells us that we can
calculate the length of the third side. There is only one possibility,
thus all three sides must be the same length.)

\begin{tikzpicture}
  \coordinate [circle, fill, inner sep=1pt] (a1) at (0,0) ;
  \coordinate [circle, fill, inner sep=1pt] (b1) at (4,0) ;
  \coordinate [circle, fill, inner sep=1pt] (c1) at (4,2) ;

   \coordinate [circle, fill, inner sep=1pt] (a2) at (5,0) ;
  \coordinate [circle, fill, inner sep=1pt] (b2) at (9,0) ;
  \coordinate [circle, fill, inner sep=1pt] (c2) at (9,2) ;
        \draw (a1) --  node[outer sep = 0.5pt, below]{3.5}  (b1);
        \draw [dashed] (b1) -- (c1);
        \draw (c1)--  node[outer sep = 5pt, above]{4} (a1);
        \pic [draw] {right angle = a1--b1--c1};

        \draw (a2) --  node[outer sep = 0.5pt, below]{3.5}  (b2);
        \draw [dashed] (b2) -- (c2);
        \draw (c2)--  node[outer sep = 5pt, above]{4} (a2);
        \pic [draw] {right angle = a2--b2--c2};
\end{tikzpicture}

In this case, the third side of each triangle must be $\sqrt{4^2 - 3.5^2} \approx 1.9$.

What if the know angle is less than $90^\circ$? \emph{The triangles
  are not necessarily congruent.} For example, lets say that there are
two triangles with sides of length 5 and 7 and that the corresponding
angle (at the end of the side of length 7) on each is $45^\circ$. Two
different triangles satisfy this:

\begin{tikzpicture}
  \coordinate [circle, fill, inner sep=1pt] (a1) at (0,0) ;
  \coordinate [circle, fill, inner sep=1pt] (b1) at (7,0) ;
  \coordinate [circle, fill, inner sep=1pt] (c1) at (4,3) ;
  \coordinate [circle, fill, inner sep=1pt] (a2) at (8,0) ;
  \coordinate [circle, fill, inner sep=1pt] (b2) at (15,0) ;
  \coordinate [circle, fill, inner sep=1pt] (c2) at (11,4) ;
  \draw (a1) --  node[outer sep = 0.5pt, below]{7}  (b1);
  \draw [dashed] (b1) -- (c1);
  \draw (c1)--  node[outer sep = 5pt, above]{5}(a1);
  \pic [draw, "$45^\circ$", angle eccentricity=1.5] {angle = c1--b1--a1};
  \draw (a2) --  node[outer sep = 0.5pt, below]{7}  (b2);
  \draw [dashed] (b2) -- (c2);
  \draw (c2)--  node[outer sep = 5pt, above]{5} (a2);
  \pic [draw, "$45^\circ$", angle eccentricity=1.5] {angle = c2--b2--a2};
\end{tikzpicture}

I think it will help to see how this happens if I lay one triangle on top of the other:

\begin{tikzpicture}
  \coordinate [circle, fill, inner sep=1pt] (a1) at (0,0) ;
  \coordinate [circle, fill, inner sep=1pt] (b1) at (7,0) ;
  \coordinate [circle, fill, inner sep=1pt] (c1) at (4,3) ;
  \coordinate [circle, fill, inner sep=1pt] (c2) at (3,4) ;
  \coordinate (d) at (2,5);
  \coordinate (e) at (3.5,3.5);

  \draw (a1) --  node[outer sep = 0.5pt, below]{7}  (b1);
  \draw [dashed,->] (b1) -- (d);
  \draw [dashed] (a1) -- (e);
  \draw (c1)--  node[outer sep = 5pt, below]{5}(a1);
  \draw (c2)--  node[outer sep = 5pt, above]{5}(a1);
  \pic [draw, "$45^\circ$", angle eccentricity=1.5] {angle = c1--b1--a1};
   \pic [draw, angle radius=8] {right angle = b1--e--a1};
\end{tikzpicture}

So there is \emph{not} a general Side-Side-Angle Congruency Test.

Here, then, is the list of common congruency tests:
\begin{itemize}
\item Side-Side-Side: All three sides have the same measure
\item Side-Angle-Angle: Two angles and one side have the same measure
\item Side-Angle-Side: Two sides and the angle between them have the same measure
\item Side-Side-Right: They are right triangles and two sides have the same measure
\end{itemize}

\begin{Exercise}[title={Congruent Triangles}, label=con_triangles]
  Ted is terrible at drawing triangles: he always draws them exactly
  the same. Fortunately, he has marked these diagrams with the sides
  and angles that he measured. For each pair of triangles, write if
  you know them to be congruent and which congruency test proves
  it. For example:

\begin{tikzpicture}[scale=0.7]
  \coordinate [circle, fill, inner sep=1pt] (a1) at (0,1) ;
  \coordinate [circle, fill, inner sep=1pt] (b1) at (5,0) ;
  \coordinate [circle, fill, inner sep=1pt] (c1) at (4,2) ;

  \coordinate [circle, fill, inner sep=1pt] (a2) at (6,1) ;
  \coordinate [circle, fill, inner sep=1pt] (b2) at (11,0) ;
  \coordinate [circle, fill, inner sep=1pt] (c2) at (10,2) ;
  \draw (a1) --  node[outer sep = 0.5pt, below]{3.5}  (b1);
  \draw (b1) -- node[outer sep = 2pt, right]{4} (c1);
  \draw (c1)--  node[outer sep = 2pt, above]{} (a1);
  \pic [draw,  "$120^\circ$", angle eccentricity=2.0] {angle = c1--b1--a1};

  \draw (a2) --  node[outer sep = 0.5pt, below]{3.5}  (b2);
  \draw  (b2) -- node[outer sep = 2pt, right]{4} (c2);
  \draw (c2) -- node[outer sep = 2pt, above]{} (a2);
  \pic [draw,  "$120^\circ$",, angle eccentricity=2.0] {angle = c2--b2--a2};
\end{tikzpicture}

  (These drawings are clearly not accurate, but you are told the measurements are.) The answer is ``Congruent by the Side-Angle-Side test.''

  \begin{multicols}{2}

\begin{tikzpicture}[scale=0.6]
  \coordinate [circle, fill, inner sep=1pt] (a1) at (0,1) ;
  \coordinate [circle, fill, inner sep=1pt] (b1) at (5,0) ;
  \coordinate [circle, fill, inner sep=1pt] (c1) at (4,2) ;

  \coordinate [circle, fill, inner sep=1pt] (a2) at (6,1) ;
  \coordinate [circle, fill, inner sep=1pt] (b2) at (11,0) ;
  \coordinate [circle, fill, inner sep=1pt] (c2) at (10,2) ;
  \draw (a1) --  node[outer sep = 0.5pt, below]{3.5}  (b1);
  \draw (b1) -- node[outer sep = 2pt, right]{} (c1);
  \draw (c1)--  node[outer sep = 2pt, above]{6} (a1);
  \pic [draw,  "$90^\circ$", angle eccentricity=2.0] {angle = c1--b1--a1};

  \draw (a2) --  node[outer sep = 0.5pt, below]{3.5}  (b2);
  \draw  (b2) -- node[outer sep = 2pt, right]{} (c2);
  \draw (c2) -- node[outer sep = 2pt, above]{6} (a2);
  \pic [draw,  "$90^\circ$",, angle eccentricity=2.0] {angle = c2--b2--a2};
\end{tikzpicture}

\hspace{3cm}

\begin{tikzpicture}[scale=0.6]
  \coordinate [circle, fill, inner sep=1pt] (a1) at (0,1) ;
  \coordinate [circle, fill, inner sep=1pt] (b1) at (5,0) ;
  \coordinate [circle, fill, inner sep=1pt] (c1) at (4,2) ;

  \coordinate [circle, fill, inner sep=1pt] (a2) at (6,1) ;
  \coordinate [circle, fill, inner sep=1pt] (b2) at (11,0) ;
  \coordinate [circle, fill, inner sep=1pt] (c2) at (10,2) ;
  \draw (a1) --  node[outer sep = 0.5pt, below]{7}  (b1);
  \draw (b1) -- node[outer sep = 2pt, right]{4} (c1);
  \draw (c1)--  node[outer sep = 2pt, above]{9} (a1);

  \draw (a2) --  node[outer sep = 0.5pt, below]{7}  (b2);
  \draw  (b2) -- node[outer sep = 2pt, right]{4} (c2);
  \draw (c2) -- node[outer sep = 2pt, above]{9} (a2);
\end{tikzpicture}

\begin{tikzpicture}[scale=0.6]
  \coordinate [circle, fill, inner sep=1pt] (a1) at (0,1) ;
  \coordinate [circle, fill, inner sep=1pt] (b1) at (5,0) ;
  \coordinate [circle, fill, inner sep=1pt] (c1) at (4,2) ;

  \coordinate [circle, fill, inner sep=1pt] (a2) at (6,1) ;
  \coordinate [circle, fill, inner sep=1pt] (b2) at (11,0) ;
  \coordinate [circle, fill, inner sep=1pt] (c2) at (10,2) ;
  \draw (a1) --  node[outer sep = 0.5pt, below]{7}  (b1);
  \draw (b1) -- node[outer sep = 2pt, right]{} (c1);
  \draw (c1)--  node[outer sep = 2pt, above]{} (a1);
  \pic [draw,  "$35^\circ$",, angle eccentricity=2.0] {angle = c1--b1--a1};
  \pic [draw,  "$62^\circ$",, angle eccentricity=2.0] {angle = b1--a1--c1};

  \draw (a2) --  node[outer sep = 0.5pt, below]{7}  (b2);
  \draw  (b2) -- node[outer sep = 2pt, right]{} (c2);
  \draw (c2) -- node[outer sep = 2pt, above]{} (a2);
  \pic [draw,  "$35^\circ$",, angle eccentricity=2.0] {angle = c2--b2--a2};
  \pic [draw,  "$62^\circ$",, angle eccentricity=2.0] {angle = b2--a2--c2};

\end{tikzpicture}

\hspace{3cm}

\begin{tikzpicture}[scale=0.6]
  \coordinate [circle, fill, inner sep=1pt] (a1) at (0,1) ;
  \coordinate [circle, fill, inner sep=1pt] (b1) at (5,0) ;
  \coordinate [circle, fill, inner sep=1pt] (c1) at (4,2) ;

  \coordinate [circle, fill, inner sep=1pt] (a2) at (6,1) ;
  \coordinate [circle, fill, inner sep=1pt] (b2) at (11,0) ;
  \coordinate [circle, fill, inner sep=1pt] (c2) at (10,2) ;
  \draw (a1) --  node[outer sep = 0.5pt, below]{8}  (b1);
  \draw (b1) -- node[outer sep = 2pt, right]{6} (c1);
  \draw (c1)--  node[outer sep = 2pt, above]{} (a1);
  \pic [draw,  "$28^\circ$",, angle eccentricity=2.0] {angle = b1--a1--c1};

  \draw (a2) --  node[outer sep = 0.5pt, below]{8}  (b2);
  \draw  (b2) -- node[outer sep = 2pt, right]{6} (c2);
  \draw (c2) -- node[outer sep = 2pt, above]{} (a2);
  \pic [draw,  "$28^\circ$",, angle eccentricity=2.0] {angle = b2--a2--c2};

\end{tikzpicture}

\end{multicols}
\end{Exercise}

\begin{Answer}[ref=con_triangles]
\begin{multicols}{2}
Congruent by the Side-Side-Right Congruency Test.

Congruent by the Side-Side-Side Congruency Test.

Congruent by the Side-Angle-Angle Congruency Test.

We don't know if they are congruent. The measured angle is not between the measured sides.

\end{multicols}

\end{Answer}


\chapter{\label{cha:materials:alternative:formulations}Alternative Material Model Formulations}

\section{\label{sec:materials:formulations:viscoelastic}Viscoelastic Formulations}

The viscoelastic formulations presently used in PyLith are described
in Section \vref{sec:materials:viscoelastic}. In some cases there
are alternative formulations that may be used in future versions of
PyLith, and those are described here.

\subsection{\label{sub:Effective-Stress-Formulation-Maxwell}Effective Stress
Formulation for a Linear Maxwell Viscoelastic Material}

An alternative technique for solving the equations for a Maxwell viscoelastic
material is based on the effective stress formulation described in
Section \vref{sec:materials:formulation:viscoelastic:effective}. A linear
Maxwell viscoelastic material may be characterized by the same elastic
parameters as an isotropic elastic material ($E$ and $\nu$), as
well as the viscosity, $\eta$. The creep strain increment is
\begin{gather}
\underline{\Delta e}^{C}=\frac{\Delta t\phantom{}^{\tau}\underline{S}}{2\eta}\,\,.\label{eq:D1}
\end{gather}
Therefore,
\begin{gather}
\Delta\overline{e}^{C}=\frac{\Delta t\sqrt{^{\tau}J_{2}^{\prime}}}{\sqrt{3\eta}}=\frac{\Delta t\phantom{}^{\tau}\overline{\sigma}}{3\eta}\,,\,\mathrm{and}\,^{\tau}\gamma=\frac{1}{2\eta}\,\,.\label{eq:D2}
\end{gather}
Substituting Equations \vref{eq:46}, \vref{eq:D1}, and \vref{eq:D2}
into \vref{eq:43}, we obtain
\begin{gather}
^{t+\Delta t}\underline{S}=\frac{1}{a_{E}}\left\{ ^{t+\Delta t}\underline{e}^{\prime}-\frac{\Delta t}{2\eta}\left[(1-\alpha)^{t}\underline{S}+\alpha\phantom{}^{t+\Delta t}\underline{S}\right]\right\} +\underline{S}^{I}\,\,.\label{eq:D3}
\end{gather}
Solving for $^{t+\Delta t}\underline{S}$,
\begin{gather}
^{t+\Delta t}\underline{S}=\frac{1}{a_{E}+\frac{\alpha\Delta t}{2\eta}}\left[^{t+\Delta t}\underline{e}^{\prime}-\frac{\Delta t}{2\eta}(1-\alpha)^{t}\underline{S}+\frac{1+\mathrm{\nu}}{E}\underline{S}^{I}\right]\,\,.\label{eq:D4}
\end{gather}
In this case it is possible to solve directly for the deviatoric stresses,
and the effective stress function approach is not needed. To obtain
the total stress, we simply use
\begin{gather}
^{t+\Delta t}\sigma_{ij}=\phantom{}^{t+\Delta t}S_{ij}+\frac{\mathit{1}}{a_{m}}\left(\,^{t+\Delta t}\theta-\theta^{I}\right)\delta_{ij}+P^{I}\delta_{ij}\,\,.\label{eq:D5}
\end{gather}

To compute the viscoelastic tangent material matrix relating stress
and strain, we need to compute the first term in Equation \vref{eq:58}.
From Equation \vref{eq:D4}, we have
\begin{gather}
\frac{\partial\phantom{}^{t+\Delta t}S_{i}}{\partial\phantom{}^{t+\Delta t}e_{k}^{\prime}}=\frac{\delta_{ik}}{a_{E}+\frac{\alpha\Delta t}{2\eta}}\,\,.\label{eq:D12}
\end{gather}
Using this, along with Equations \vref{eq:58}, \vref{eq:59}, and \vref{eq:60},
the final material matrix relating stress and tensor strain is
\begin{gather}
C_{ij}^{VE}=\frac{1}{3a_{m}}\left[\begin{array}{cccccc}
1 & 1 & 1 & 0 & 0 & 0\\
1 & 1 & 1 & 0 & 0 & 0\\
1 & 1 & 1 & 0 & 0 & 0\\
0 & 0 & 0 & 0 & 0 & 0\\
0 & 0 & 0 & 0 & 0 & 0\\
0 & 0 & 0 & 0 & 0 & 0
\end{array}\right]+\frac{1}{3\left(a_{E}+\frac{\alpha\Delta t}{2\eta}\right)}\left[\begin{array}{cccccc}
2 & -1 & -1 & 0 & 0 & 0\\
-1 & 2 & -1 & 0 & 0 & 0\\
-1 & -1 & 2 & 0 & 0 & 0\\
0 & 0 & 0 & 3 & 0 & 0\\
0 & 0 & 0 & 0 & 3 & 0\\
0 & 0 & 0 & 0 & 0 & 3
\end{array}\right]\,.\label{eq:D13}
\end{gather}
Note that the coefficient of the second matrix approaches $E/3(1+\nu)=1/3a_{E}$
as $\eta$ goes to infinity. To check the results we make sure that
the regular elastic constitutive matrix is obtained for selected terms
in the case where $\eta$ goes to infinity.
\begin{gather}
C_{11}^{E}=\frac{E(1-\nu)}{(1+\nu)(1-2\nu)}\,\,\nonumber \\
C_{12}^{E}=\frac{E\nu}{(1+\nu)(1-2\nu)}\,.\label{eq:D14}\\
C_{44}^{E}=\frac{E}{1+\nu}\,\,\nonumber
\end{gather}
This is consistent with the regular elasticity matrix, and Equation
\vref{eq:D13} should thus be used when forming the stiffness matrix.
We do not presently use this formulation, but it may be included in
future versions.


%%% ===================================================================
%%% RCS Header:
%%% $RCSfile: local.el,v $
%%% $Revision: 1.27 $
%%% $Date: 2014/11/07 15:36:00 $
%%% $Author: backofen $
%%% $Locker:  $
%%% ===================================================================

\subsection{Proof}
\def\E{\mathbb{E}}
\begin{frame}{Universal Hashing}{Proof}
  \textbf{Given:}
  \begin{itemize}
    \item<1->
      We pick two random keys
      {\color{MainA}$x, y \in \mathbb{S} \mid x \neq y$} and
      a random hash function {\color{MainA}$h \in \mathbb{H}$}
    \item<2->
      We know the probability of a collision:
      {\color{MainA}\[P(h(x) = h(y)) \leq c \cdot \frac{1}{m}\]}
  \end{itemize}
  \onslide<3->
  \textbf{To proof:}
  {\color{MainA}\[\E\left[\vert \mathbb{S}_i \vert\right]
    \leq 1 + c \cdot \frac{\vert \mathbb{S} \vert}{m}
    \hspace*{1.5em} \forall \, i\]}
\end{frame}

%-------------------------------------------------------------------------------

\begin{frame}{Universal Hashing}{Proof}
  \textbf{We know:}
  {\color{MainA}\[\mathbb{S}_i = \{x \in \mathbb{S}: h(x) = i\}\]}\\
  \onslide<2->{
    If {\color{MainA}$\mathbb{S}_i = \emptyset \Rightarrow
      \vert \mathbb{S}_i \vert = 0$}
  }%
  \onslide<3->{
    otherwise, let {\color{MainA}$x \in \mathbb{S}_i$ be any key}
  }\\[0.5em]
  \onslide<4->
  We define an indicator variable:
  {\color{MainA}
  \begin{displaymath}
    I_y = \begin{cases}
      1, & \text{if } h(y) = i\\
      0, & \text{else}
    \end{cases} \hspace*{1.5em} y \in \mathbb{S} \setminus \{x\}
  \end{displaymath}
  \onslide<5->
  \begin{align*}
    \Rightarrow \hspace{1.5em} \vert \mathbb{S}_i \vert
    & = 1 + \sum_{y \in \mathbb{S} \setminus x} I_y
  \end{align*}
  \onslide<6->
  \begin{align*}
    \Rightarrow \hspace{1.5em} \E\left(\vert \mathbb{S}_i \vert\right)
      & = \E\left(1 + \sum_{y \in \mathbb{S} \setminus x} I_y\right)
        = 1 + \sum_{y \in \mathbb{S} \setminus x} \E(I_y)
  \end{align*}}
\end{frame}

%-------------------------------------------------------------------------------

\begin{frame}{Universal Hashing}{Proof}
  \textbf{Auxiliary calculation:}
   {\color{MainA}
     \onslide<1->
     \vspace*{-1.7em}
     \begin{eqnarray*}
       \hspace{6em}
         \E[I_y] & = & P(I_y = 1)\\
         {} & = & P(h(y) = i)\\
         {} & = & P(h(y) = h(x))\\
         {} & \leq &  c \cdot \frac{1}{m}
   \end{eqnarray*}}
   {\color{MainA}
     \onslide<3->
     \vspace{-0.6em}
     \begin{alignat*}{3}
\hspace*{-2cm} \text{\textbf{Hence:\quad{}}}        \E\left[\vert \mathbb{S}_i \vert\right]
=1 + \sum_{y \in \mathbb{S} \setminus x}  \E\left[l_y \right]
      &\ \leq\ 1 + \sum_{y \in \mathbb{S} \setminus x} c \cdot \frac{1}{m}\\
     &\onslide<4->\ =\ 1 + \left(\vert \mathbb{S}  \vert - 1\right) \cdot c \cdot \frac{1}{m}\\
     &\onslide<5->\ \leq\ \displaystyle 1 + \vert \mathbb{S} \vert \cdot c \cdot \frac{1}{m}\\
&\onslide<5->\ =\  1 + c \cdot \frac{\vert \mathbb{S} \vert}{m}
&\quad\square
     \end{alignat*}
}
\end{frame}

%%% ===================================================================
%%% This should be at the END of the file !!!!!!
%%%
%%% Local Variables:
%%% mode: latex
%%% TeX-master: "~/TeX/TeXinput/Scripts/Algo-Data-EMS/AlgoDat/Lecture-5/Lecture.tex"
%%% End:
%%% ===================================================================