\chapter{Machine Learning Basics}\label{sec:ml_basic}
This chapter provides a brief introduction to the fundamental theory of machine learning that is used throughout the rest of this thesis.
Machine learning is the study of computer algorithms that learn by analyzing data. Most machine learning algorithms can be divided into the two categories of supervised learning and unsupervised learning. In the following, we will describe the informal distinction between these and the most common tasks used for supervised learning. As neural networks are considered supervised learning the following sections will focus only on theory for supervised learning algorithms.
\\
\\
We proceed to describe the challenge of finding patterns and generalizing these to new data while describing the various machine learning components such as capacity, loss functions and regularization. Most machine learning algorithms are based on the idea of minimizing a loss function by using an optimization algorithm to select the optimal parameters. One of the most common optimization algorithms used in the context of machine learning is called stochastic gradient descent. We cover this algorithm and its variants in in section \ref{sec:gradient_optimization}. We will later use the theory of stochastic gradient descent in combination with backpropagation in section \ref{sec:backprop} for describing learning in neural networks.

\section{Supervised Learning}
Supervised learning use labels on datapoints, which we call examples, as targets to create prediction rules for learning the label of new examples. In unsupervised learning useful properties of the data structure are learned without the model being provided with target labels to predict. However as mentioned by \cite{Goodfellow-et-al-2016} the distinction is not formally defined, since there is no objective way to determine whether a value is a feature or a target label.\\
\\
One example of a learning algorithm that is typically viewed as supervised is classic linear regression that uses examples $x$ and their labels $y$ to create a linear function to determine $y$-values of new examples $x$. An example of a learning algorithm that is typically viewed as unsupervised is $k$-means clustering, that divides a dataset into $k$ clusters based on distance. \\
\\
As neural networks and Bayesian neural networks use target labels to predict labels of new examples it is considered a supervised learning algorithm. Therefore the following chapter will only cover the machine learning basics of supervised algorithms. \\
\\
Typical tasks of supervised machine learning algorithms are:
\begin{itemize}
    \item Classification when the output space $\YY$ are labels. This is called binary classification in cases with only two possible labels. In this case the algorithm must learn to separate examples of these two labels. Binary labels (be it male/female, yes/no etc.) are typically translated to a numerical representation by letting $\YY = \{ \pm 1 \}$ or $\YY = \{0,1 \}$. When working with more than two, but still a finite number of labels, we call the task multiclass classification. Depending of the setting and amount of labels it can be preferable to use regression instead of multiclass classification.
    \item Regression when the output space $\YY = \mathbb{R}$. For example predicting someones weight could be modeled as a regression problem.
    \item Structured prediction when the output is not merely a number or a label, but a structured object. An example of this is machine translation where the input is a sentence in one language and the output a sentence in another language.
\end{itemize}

\section{Loss Functions} \label{sec:loss_func}
In order to evaluate the performance of a supervised machine learning algorithm we use a quantitative measure called a loss function. The loss function is an encoding, of how much the algorithm should care about making certain kinds of mistakes, and it is based on this measure that the algorithm selects a hypothesis $h$ from a set of possible hypothesis $\mathbb{H}$ called the hypothesis space. A hypothesis (also called prediction rule) can be understood as a recipe, that the algorithm develops to perform its task. An example would be the weights $\boldsymbol{w}$ for performing the linear regression $\hat{y}^{(i)} = \boldsymbol{w}^\top \boldsymbol{x}^{(i)}$. Another example would be a specific range of values for a specific set of features for classification, like a set of RGB color-values to classify if an input-image is depicting an apple or a pear. We treat the hypothesis as a function $h\lr{\boldsymbol{x}^{(i)}}$ taking $\boldsymbol{x}^{(i)}$ as input and predicting $\hat{y}^{(i)}$. We define the loss function $\ell \lr{h \lr{\boldsymbol{x}^{(i)}}, y^{(i)}}$ as being the loss of predicting $h \lr{\boldsymbol{x}^{(i)}}$ when the target label is $y^{(i)}$.\\
\\
We want the algorithm to return the "best" hypothesis $h^*$, which in this context should be interpreted as the hypothesis that gives the least amount of expected loss on new samples $\lr{\mathbf{x}^{(i)}_{\text{new}}, \text{y}^{(i)}_{\text{new}}}$
\begin{equation*}
    L(h) \equiv \E \lrs{\ell \lr{h \lr{\mathbf{x}_{\text{new}}^{(i)}}, \text{y}_{\text{new}}^{(i)}}}
\end{equation*}
where the expectation is taken with respect to the data distribution $p_{\text{data}}$. If such a value is satisfyingly low, it means that $h^*$ is a useful tool for performing its task on new data. We define the optimal hypothesis formally by
\begin{equation*}
   h^* = \argmin_{h \in \mathbb{H}} L(h)
\end{equation*}
\\
\\
Common loss functions in regression are squared loss (also called L2 loss)
\begin{equation} \label{eq:mse}
    \ell\left(h\lr{\boldsymbol{x}^{(i)}}, y^{(i)} \right) = \lr{ h\lr{\boldsymbol{x}^{(i)}} - y^{(i)} }^2
\end{equation}
or absolute loss (also called L1 loss)
$$ \ell\left(h\lr{\boldsymbol{x}^{(i)}}, y^{(i)}\right) = \lra{h\lr{\boldsymbol{x}^{(i)}} - y^{(i)}}$$
These functions train the algorithm to create hypothesis that minimize the differences from the target labels.
\\
\\
A typical loss function used for binary classification is the zero-one loss
$$ \ell\left(h\lr{\boldsymbol{x}^{(i)}}, y^{(i)}\right)=\1\left(h\lr{\boldsymbol{x}^{(i)}} \neq y^{(i)} \right)=\left\{\begin{array}{ll}
0, & \text { if } h\lr{\boldsymbol{x}^{(i)}}=y^{(i)} \\
1, & \text { otherwise }
\end{array}\right. $$
giving $0$ loss for when the predicted label $h\lr{\boldsymbol{x}^{(i)}}$ is equal to the true label $y^{(i)}$ and $1$ otherwise. This loss function trains the algorithm to generate hypothesis, that makes the least number of mistakes.
\\
\\
Another classification loss function that do not just look at the predicted label, but at the probability of a certain prediction, is the cross-entropy loss or log-loss function. This assumes that instead of predicting the label, the hypothesis $h\lr{\boldsymbol{x}^{(i)}}$ generated by the model is a set of probabilities that each provide the probability that the example belong to each of the classes. For binary classification it is defined as
\begin{equation}\label{eq:cross_entr}
    \ell\left(h\lr{\boldsymbol{x}^{(i)}}, y^{(i)} \right) = - \lr{y^{(i)} \log h\lr{\boldsymbol{x}^{(i)}} + \lr{1 - y^{(i)}}} \log \lr{1 - h\lr{\boldsymbol{x}^{(i)}}}
\end{equation}
which is actually the negative log likelihood (so minimizing this maximizes likelihood, see section \ref{sec:mle} for maximum likelihood estimation) when assuming Bernoulli distribution for the targets. For multiclass classification this can be generalized by assuming multinomial distribution with one trail for the targets and taking the negative log likelihood. These probabilities are typically provided by a sigmoid or softmax function that maps $\mathbb{R} \rightarrow [0,1]$, these are explained further in section \ref{sec:feedforward_nn}. With a model, that outputs these probabilities as hypothesis, predictions are typically done by choosing the class for which the example has the highest probability of belonging to.
\\
\\
These are some of the convenient general choices, and there are many more. But such general loss functions is not necessarily the right choice for a particular application. For example if a company knows the exact monetary loss of packing a wrong item as a result of a misclassification happening in a robotic packaging system. Then it might be beneficial to use a table (perhaps formally written as a sum of indicatorfunctions) of the costs of packaging each of their items as a loss function instead of using the general ones found in the literature.

\section{Training \& Testing} \label{sec:train_val}
The central challenge in machine learning is making sure that the algorithm creates a hypothesis that perform well on new data that was not used in selecting the hypothesis. As mentioned in section \ref{sec:loss_func} this is done by minimizing $L(h)$, but as we do not know $p_\text{data}$ (if we did we wouldn't need the algorithm) we can not evaluate $L(h)$. We must instead approximate $L(h)$ by its empirical estimate
\begin{equation*}
    \hat{L}(h,S)= \frac{1}{N} \sum_{i=1}^N \ell \lr{h\lr{\boldsymbol{x}^{(i)}}, y^{(i)}}
\end{equation*}
However when we select a hypothesis $\hat{h}^*_S$ in $\mathbb{H}$ based on empirical loss $\hat{L} \lr{h, S}$ then the loss of this hypothesis $\hat{h}^*_S$ becomes a biased estimate of $L(\hat{h}^*_S)$. This is because $\hat{h}^*_S$ is selected based on the minimum empirical error on $S$, so from the perspective of $\hat{h}^*_S$ new samples might not be interchangeable with samples in $S$, since including these new samples could result in a different hypothesis that minimizes the loss on $S$. \\
\\
To get an unbiased estimate of $L(\hat{h}^*_S)$ for evaluating a model it is common practice to split the sample set $S$ into a dataset $S_\text{train}$ and a test set $S_\text{test}$. One can then find the best hypothesis using the training set $h^*_\text{train}$ and afterwards use the test set for computation of $\hat{L}\lr{h^*_\text{train}, S_\text{test}}$ to evaluate the performance. Based on the assumption that new samples $\lr{\mathbf{x}^{(i)}_{\text{new}}, \text{y}^{(i)}_{\text{new}}}$ are distributed identically with the samples in $S_\text{test}$ then these new samples are exchangeable with the ones in $S_\text{test}$ from the perspective of $h^*_\text{train}$. This means that $\E \lrs{\ell \lr{h^*_\text{train} \lr{\boldsymbol{x}^{(i)}}, y^{(i)}}} = \E \lrs{\ell \lr{h^*_\text{train} \lr{\mathbf{x}^{(i)}_{\text{new}}}, \text{y}_{\text{new}}^{(i)}}}$ and therefore $\hat{L}\lr{h^*_\text{train}, S_\text{test}}$ is an unbiased estimate of $L(h^*_\text{train})$.

\subsection{Cross-Validation} \label{sec:cv}
Splitting a dataset into a training set and a test set can be problematic if the resulting test set is too small. Such a small test set will give rise to statistical uncertainties around the estimated average loss and will make it problematic to evaluate the model. When we have large dataset this is not a problem, but when we work with small datasets it can become a serious issue. \\
\\
To circumvent this issue is to use cross-validation. Cross-validation repeats the training and testing procedure on different randomly chosen splits of the original dataset. This enables us to use more examples for estimating the average test loss for the cost of computational power. This approach is also useful for choosing hyperparameters like regularization constants $\alpha$ covered in section \ref{sec:regularization}.\\
\\
A common method for cross-validation is the $K$-fold cross-validation that splits the dataset into $K$ non-overlapping and roughly equally sized subsets. On trial $i$ the $i$th subset is used as the test set while the rest $K - 1$ subsets are used in the training set. The test loss is then estimated by the cross-validation loss computed as the average test loss across the $K$ trials.\\
\\
One problem with this approach is however measuring the uncertainty since there is no unbiased estimators of the variance of the average error estimators as mentioned by \cite{Bengio04}, one must instead use approximations. Another discussed issue is how to choose the value for $K$. This problem does not have a clear answer but 5- or 10-fold cross-validation are generally recommended as a good compromise, see \cite{brieman_spector_1992} and \cite{kohavi_1992}.

\section{Overfitting \& Underfitting}
The average loss attained on the training set when selecting the best hypothesis $\hat{L}\lr{h^*_\text{train}, S_\text{train}}$ is what we will call the training error or training loss. The average loss attained from the test set $\hat{L}\lr{h^*_\text{train}, S_\text{val}}$, used for testing the model, is what we will call test error or test loss.\\
\\
Having a low training error means that we have made a prediction rule that fit our training set well. Having a small gap between training and test error means that the prediction rule generalizes well on new data, which is why the test error is also sometimes called generalization error. These factors are what corresponds to underfitting and overfitting, which is used to measure performance of the model.\\
\\
Underfitting occurs when the model is not able to obtain a sufficiently low training error. This is seen in figure \ref{fig:regr_example} in the case of linear regression of degree 1, where the model fitted on the training data lies far from the training samples, meaning it must have a large mean squared error (MSE) on the training data. Overfitting occurs when the gap between the training error and test error is too large. This large gap indicates that the model faithfully reflects idiosyncrasies of the training data rather than the true underlying process generating the data. This can be seen in figure \ref{fig:regr_example} in the case of linear regression of degree 15, where the model lies very close to all of the training samples meaning it must have a very small MSE on the training data. But if we sample points from the true function we see that many of these will very likely lie far from the model, resulting in a large test error relative to the training error. Overfitting and underfitting are both something we want to avoid. \\
\\
We can control whether a model under- or overfits by changing its capacity. Capacity is a model's ability to learn a larger variety of hypothesis. A model with a low capacity might underfit the training set if it can not capture the structure of the data. A model with high capacity might overfit if it learns properties of the training set that is not representative of the test set. As an example we can increase the capacity of linear regression by allowing it to fit higher order polynomials, thus increasing the hypothesis space from which we draw prediction rules. This example is seen in figure \ref{fig:regr_example}, where we see that a low capacity causes underfitting while a very large capacity causes the model to overfit.

\begin{figure}
    \centering
    \includegraphics[width=\textwidth,keepaspectratio]{over_underfitting.pdf}
    \caption{Regression of different degree polynomials on a $\cos$ curve. We see that a linear regression of degree 1 is insufficient to fit the training set and is underfitting. A polynomial of degree 4 fits the curve almost perfectly while a polynomial of degree 15 learns the noise of the data and overfits. How well the different regressors generalize to a test set is evaluated quantitatively by calculating the mean squared error using the loss function in equation \ref{eq:mse} using 10-fold cross-validation. We see that this error is lowest when using 4-degree polynomial regression. Cross-validation is covered in section \ref{sec:cv}. The standard error of these losses are shown in the parenthesis. The Python code for producing this figure can be seen in appendix \ref{app:overfitting}.}
    \label{fig:regr_example}
\end{figure}
\noindent

\section{Regularization}\label{sec:regularization}
So far we have covered how to control over- and underfitting by changing a model's capacity. Another method is regularization which encodes the objective function with preferences for certain functions instead of excluding functions from the hypothesis space. The objective function is what we ask a machine learning algorithm to minimize, and has in the previous been synonymous with average loss. The hypothesis $h$ is what we will now embed in the choice of model parameters $\boldsymbol{\theta}$.\\
\\
A popular type of regularization is norm penalties, where a norm penalty function $\Omega(\boldsymbol{\theta})$ is added to the objective function  $J$ (which is usually average loss). Then the regularized objective function is
\begin{equation} \label{eq:reg_obj_func}
    \Tilde{J}(\boldsymbol{\theta}; \mathbf{X}, \mathbf{y}) = J(\boldsymbol{\theta}; \mathbf{X}, \mathbf{y}) + \alpha \Omega(\boldsymbol{\theta})
\end{equation}
where $\alpha \in [0, \infty]$ is a parameter chosen before minimizing $\Tilde{J}$, that controls how much the penalty term $ \Omega(\boldsymbol{\theta})$ contributes relatively to the objective function $J$. When $\alpha=0$ we have no regularization while larger values for $\alpha$ results in more regularization. In this way minimizing $ \Tilde{J}$ becomes a trade-off between fitting the training data and having small weights.\\
\\
A common norm penalty function is the L2 parameter norm penalty also known as weight decay $\Omega \lr{\boldsymbol{\theta}} = \frac{1}{2} \lra{\lra{\boldsymbol{\theta}}}_2^2$. L2 regularization is also known as ridge regression or Tikhonov regularization. In a regression setting $\boldsymbol{y} = \boldsymbol{x} \boldsymbol{w} + \boldsymbol{b}$, if we assume no bias parameter $\boldsymbol{b}$ and that model parameters are only weights $\boldsymbol{\theta} = \boldsymbol{w}$, the regularized objective function is defined by
\begin{equation} \label{eq:L2_reg}
    \Tilde{J}(\boldsymbol{\theta}; \mathbf{X}, \mathbf{y}) = J(\boldsymbol{\theta}; \mathbf{X}, \mathbf{y}) + \frac{\alpha}{2} \boldsymbol{\theta}^\top \boldsymbol{\theta}
\end{equation}
We can clearly see from this, that larger values of $\alpha$ punishes larger weights. \\
\\
A regularization method often used for neural networks is early stopping. We cover this method in section \ref{sec:early_stopping} providing a popular intuitive approach to regularization other than weight decay.
\\
\\
There are many other techniques to reduce overfitting, which we will not cover. One such related type of regularization is norm penalties as constrained optimization problems where $\Omega \lr{\boldsymbol{\theta}}$ is constrained by some constant $k$ while seeking to minimize equation \ref{eq:reg_obj_func}. Another is data augmentation that increases the size of the training set by augmenting existing data and adding the augmented copy to the dataset. A last worthy mention, used specifically for neural networks, is dropout (\cite{srivastava2014dropout}), which regularize by ignoring a random set of network-nodes during training.

\section{Gradient Based Optimization}\label{sec:gradient_optimization}
Supervised machine learning algorithms are often based on learning parameters by minimizing a regularized loss function. This can be done for a differentiable convex loss function by minimizing its gradient, which is what we do in gradient based optimization.

\subsection{Gradient Descent}
Gradient descent is an algorithm suggested by \cite{Cauchy1847}. The algorithm is an iterative process of selecting parameters that decrease the gradient of the loss function $J(\boldsymbol{\theta},\boldsymbol{X},y)$. We will consider sampled examples from the sample space $\mathbb{X}$ and aim to estimate the model parameters $\boldsymbol{\theta}$. The goal is to minimize the loss function which we assume to be convex, so that the function is minimized where the gradient is zero
\begin{equation*}
    \nabla J(\boldsymbol{\theta},\boldsymbol{X},y)=\left[\frac{\partial J}{\partial\theta_1},\frac{\partial J}{\partial\theta_2},\ldots,\frac{\partial J}{\partial\theta_d}\right]=\boldsymbol{0}
\end{equation*}
The gradient tells us which direction that has the steepest slope on $J$, and we can thus change the model parameters such that we move in the opposite direction of where the gradient is pointing. This will bring us in the direction that locally decreases the objective function the fastest. In order to run the algorithm we need to initialize the model parameters $\boldsymbol{\theta}\equiv\boldsymbol{\theta}^0$, evaluate the objective function $J(\boldsymbol{\theta},\boldsymbol{X},y)$ and calculate the gradient respectively $\nabla J(\boldsymbol{\theta},\boldsymbol{X},y)$. Further we also need to specify a learning rate $\eta$, which affects the size of the step i.e. change in parameters in the direction of the negative gradient for each iteration. The model parameters are then updated iteratively by
\begin{equation*}
    \boldsymbol{\theta}^{(t+1)}=\boldsymbol{\theta}^{(t)}-\eta \nabla J(\boldsymbol{\theta}^{(t)},\boldsymbol{X},y)
\end{equation*}
This means, that when we are far away from optimum (large gradient) we are taking larger steps, but when we get closer to optimum (smaller gradient) we take smaller and smaller steps. The algorithm is not assured to reach a global minimum unless the loss function is strictly convex. \cite{bishop2007} mentions that in order to find a "good" minimum, we might be forced to run the algorithm multiple times, where we each time initialize with a randomly chosen starting point $\boldsymbol{\theta}^0$.

\subsection{Stochastic Gradient Descent} \label{sec:sgd}
A widely popular and more computational feasible approach is to learn parameters using the stochastic gradient descent (SGD) algorithm. SGD is a stochastic approximation of the gradient descent algorithm described above, which replaces the gradient calculated on the entire dataset by an estimate calculated from a randomly selected subset of the data. This is useful when one faces high-dimensional optimization problems as it reduces the computational burden significantly (\cite{bottou2008tradeoffs}). Some authors append the term "mini-batch" to the algorithm name expect for the case where only one example has been used as a subset of the data.\\
\\
The pseudocode for SGD can be seen in algorithm \ref{alg:sgd}. Convergence and stopping criteria of descent algorithms have been studied extensively in the literature, and will not be discussed rigorously here. We will only provide some of the  most common stopping criteria found in software-packages. One is to stop when the norm of the gradient is beneath some threshold, $\norm{\nabla J \lr{\boldsymbol{\theta}^{(t-1)}}} < \epsilon$ another is when the decrease in norm of the gradient drops under some threshold, $\norm{\nabla J \lr{\boldsymbol{\theta}^{(t -1 )}}} - \norm{ \nabla J \lr{\boldsymbol{\theta}^{(t)}}} < \epsilon$. Most software also include a number of max iterations as a stopping criteria to prevent non-converging cases to run forever.\\
\begin{algorithm} \label{alg:sgd}
    \SetAlgoLined
    \KwInput{A dataset $S$}
    \KwInput{A learning rate $\eta$}
    \KwInput{Gradient function of objective function $\nabla J \lr{\boldsymbol{\theta}, \boldsymbol{x}, \boldsymbol{y}}$}
    \KwInput{Initial parameter vector $\boldsymbol{\theta}^{0}$}
    \KwOutput{Resulting parameters $\boldsymbol{\theta}$}
    initialize $\boldsymbol{\theta}\leftarrow \boldsymbol{\theta}^0$ \\
    \Repeat{stopping criteria is met}{
        $\text{pick randomly } S^\prime \in S$ \\
        $\boldsymbol{g} \leftarrow \frac{1}{\left|S^{\prime}\right|} \sum_{(\boldsymbol{x}, \boldsymbol{y}) \in S^{\prime}} \nabla_{\boldsymbol{\theta}} J\left(\boldsymbol{\theta},\boldsymbol{x},\boldsymbol{y}\right)$\\
        $\boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \eta \boldsymbol{g} $
    }
     \caption{Mini-Batch Stochastic Gradient Descent}
\end{algorithm}

\subsection{Stochastic Gradient Descent with Momentum}
One issue with SGD, shown by \cite{sutton1986}, is that it performs poorly with surfaces containing ravines, i.e places where the curvature is more steep in one dimension than in another, which is common around optima. In these cases SGD will oscillate across the slopes of the ravines while slowly making progress towards the optima. A method for accelerating the process towards the optima while dampening oscillation is introducing momentum. This is done by adding a fraction $\gamma$ of the update vector from time $t -1$, to the update vector for time $t$. Thus updating $\boldsymbol{\theta}$ by
\begin{equation*}
    \begin{split}
        v_{t} &= \gamma v_{t -1} + \eta \nabla J \lr{\boldsymbol{\theta}^{(t)}} \\
        \boldsymbol{\theta}^{(t+1)} &= \boldsymbol{\theta}^{(t)} - v_t
    \end{split}
\end{equation*}
The momentum term increases for dimensions with gradients pointing in the same directions and reduces updates for dimensions where gradients change directions. This results in faster convergence and reduced oscillation.

\subsection{Adaptive Gradiant Algorithm (AdaGrad)}
Another possible improvement to SGD is by adapting the learning rate to the parameters, so that stepsize decreases as we perform each iteration. This makes it possible to move faster initially and afterwards slow down to avoid overstepping the optima, which will result in faster convergence. This method was proposed by \cite{duchi2011adaptive} with an SGD-extension called Adaptive Gradiant Algorithm (AdaGrad). AdaGrad performs the parameter-update by
\begin{equation*}
    \theta^{(t+1)}_i = \theta^{(t)}_i - \frac{\eta}{\sqrt{\varepsilon + \sum_{\tau = 1}^t \lr{\nabla J \lr{\theta_i}^{(t)}}^2}} \nabla J \lr{\theta_i}^{(t)}
\end{equation*}
making each parameters have its own learning rate $\frac{\eta}{\sqrt{\varepsilon + \sum_{\tau = 1}^t \lr{\nabla J \lr{\theta}^{(t)}}^2}}$, where $\varepsilon$ is a small number that ensures that we do not divide by $0$. As the sum of gradients increases with each step the learning rate decreases, which results in a smaller stepsize as the algorithm progresses.
\\
\\
Another upside to this approach, besides faster convergence, is that we do not need to manually tune the learning rate as it is now adapted. AdaGrad is also especially efficient when working with sparse data, as parameters for features that are often $0$ (infrequent features), can receive more impact-full updates (higher learning rate) without affecting the impact on features that are often non-zero (frequent features). Without adaptive learning rates for each parameter we might inefficiently have a learning rate that decreases too slowly for frequent features or too quickly for infrequent features.

\subsection{Root Mean Square Propagation (RMSprop)}
AdaGrad's weakness is that the learning rate eventually can become infinitesimally small, making it impossible for the algorithm to learn i.e. update the parameters further. The Root Mean Square Propagation (RMSprop) method suggested by \cite{Tieleman_Hinton2012} is an extension of AdaGrad, that seeks to overcome the weakness of AdaGrad's aggressive monotonically decreasing learning rate. RMSprob does this by replacing the sum of past gradients by an expectation of the gradient, thus updating parameters by
\begin{equation*}
    \theta^{(t+1)}_i = \theta^{(t)}_i - \frac{\eta}{\sqrt{\varepsilon + \E \lrs{\nabla J \lr{\theta_i^{(t)}}^2}}} \nabla J \lr{\theta_i^{(t)}}
\end{equation*}
where
\begin{equation*}
    \E \lrs{\nabla J \lr{\theta_i^{(t)}}^2} = \lr{1 - \gamma}\nabla J \lr{\theta_i^{(t)}}^2 + \gamma \E \lrs{\nabla J \lr{\theta_i^{(t-1)}}^2}
\end{equation*}
which makes it possible for the learning rate to either increase or decrease for each epoch. RMSprob also efficiently define the sum of gradients recursively as a decaying average of past gradients instead of inefficiently storing all of the past gradients as in AdaGrad. The running average $\E \lrs{\nabla J \lr{\theta_i^{(t)}}^2}$ only depends on a hyperparameter-defined fraction $\gamma$ (similar to the momentum term), the previous average and the current gradient.

\subsection{Adaptive Moment Estimation (ADAM)} \label{sec:ADAM}
\cite{Kingma_Ba_2015} proposes an algorithm called Adaptive Moment Estimation (ADAM) which takes a similar approach to computing adaptive learning rate by storing an exponentially decaying average of past gradients $\boldsymbol{v}^{(t)}$ like RMSprop. ADAM however extends this idea by also keeping an exponentially decaying average of past gradients $\boldsymbol{m}^{(t)}$, that works similar to momentum. These components are calculated by
\begin{equation*}
    \begin{split}
        \boldsymbol{m}^{(t)} &= \beta_1 \boldsymbol{m}^{(t-1)} + \lr{1- \beta_1}\nabla J \lr{\boldsymbol{\theta}^{(t)}}\\
        \boldsymbol{v}^{(t)} &= \beta_2 \boldsymbol{v}^{(t-1)} + \lr{1 - \beta_2}\nabla J \lr{\boldsymbol{\theta}^{(t)}}^2
    \end{split}
\end{equation*}
where $\nabla J \lr{\boldsymbol{\theta}^{(t)}}^2$ indicates the elementwise square $\nabla J \lr{\boldsymbol{\theta}^{(t)}} \odot \nabla J \lr{\boldsymbol{\theta}^{(t)}}$ and where $\boldsymbol{m}^{(t)}$ and $\boldsymbol{v}^{(t)}$ are estimates of the first moment and second moment of the time-$t$ gradient respectively, hence the name Adaptive Moment Estimation.
The authors observed that as $\boldsymbol{m}^{(t)}$ and $\boldsymbol{v}^{(t)}$ are initialized as $\boldsymbol{0}$ they will be biased estimates. They counteract this bias by deriving the bias-corrected estimates
\begin{equation*}
    \begin{split}
        \hat{\boldsymbol{m}}^{(t)} &= \frac{\boldsymbol{m}^{(t)}}{1 - \beta_1^t}\\
        \hat{\boldsymbol{v}}^{(t)} &= \frac{\boldsymbol{v}^{(t)}}{1 - \beta_2^t}
    \end{split}
\end{equation*}
The parameters are then updated by
\begin{equation*}
    \boldsymbol{\theta}^{(t+1)} = \boldsymbol{\theta}^{(t)} - \frac{\eta}{\sqrt{\hat{\boldsymbol{v}}^{(t)}} + \epsilon}\hat{\boldsymbol{m}}^{(t)}
\end{equation*}
The pseudocode for ADAM can be seen in algorithm \ref{alg:adam}.

\begin{algorithm}\label{alg:adam}
    \SetAlgoLined
    \KwInput{A dataset $S$}
    \KwInput{A learning rate $\eta$}
    \KwInput{Exponential decay rates for moment estimates $\beta_1, \beta_2 \in [0,1)$}
    \KwInput{Gradient function of objective function $\nabla J \lr{\boldsymbol{\theta}}$}
    \KwInput{Initial parameter vector $\boldsymbol{\theta}^{0}$}
    \KwOutput{Resulting parameters $\boldsymbol{\theta}$}
    initialize $\boldsymbol{\theta}\leftarrow \boldsymbol{\theta}^0$ \\
    initialize $\boldsymbol{m}^{(0)}, \boldsymbol{v}^{(0)}\leftarrow \boldsymbol{0}$\\
    initialize $t \leftarrow 0$\\
    \Repeat{stopping criteria is met}{
        $t \leftarrow t + 1$
        $\text{pick randomly } S^\prime \in S$ \\
        $\boldsymbol{g}^{(t)} \leftarrow \frac{1}{\left|S^{\prime}\right|} \sum_{(\boldsymbol{x}, \boldsymbol{y}) \in S^{\prime}} \nabla_{\boldsymbol{\theta}} J\left(\boldsymbol{\theta}^{(t-1)},\boldsymbol{x},\boldsymbol{y}\right)$\\
        $\boldsymbol{m}^{(t)} \leftarrow \beta_1 \boldsymbol{m}^{(t-1)} + (1 - \beta_1)\boldsymbol{g}^{(t)}$\\
        $\boldsymbol{v}^{(t)} \leftarrow \beta_2 \boldsymbol{v}^{(t-1)} + (1 - \beta_2)\boldsymbol{g}^{(t)^2}$\\
        $\hat{\boldsymbol{m}}^{(t)} \leftarrow \frac{\boldsymbol{m}^{(t)}}{1 - \beta_1^t}$\\
        $\hat{\boldsymbol{v}}^(t) \leftarrow \frac{\boldsymbol{v}^{(t-1)}}{1 - \beta_2^t}$\\
        $\boldsymbol{\theta}^{(t)} \leftarrow \boldsymbol{\theta}^{(t-1)} - \frac{\eta}{\sqrt{\hat{\boldsymbol{v}}^{(t)}} + \varepsilon} \hat{\boldsymbol{m}}^{(t)}$
    }
     \caption{ADAM (with mini-batch)}
\end{algorithm}


% $Id$
\documentclass[12pt,a4paper]{article}
\usepackage{graphicx}
\setlength{\textwidth}{190mm}
\setlength{\textheight}{253mm}
\setlength{\oddsidemargin}{-14mm}
\setlength{\evensidemargin}{-14mm}
\setlength{\topmargin}{-12mm}

\setlength{\parindent}{0pt}
\setlength{\parskip}{1ex plus 0.5ex minus 0.2ex}

\setcounter{secnumdepth}{-1}   % disable section numbering

\newcommand{\E}{\mathrm{E}}
\newcommand{\C}{\mathrm{C}}
\newcommand{\dd}[2]{\frac{\partial{#1}}{\partial{#2}}}

\begin{document}
\title{Corrected error calculation for iterative Bayesian unfolding}
\author{Tim Adye, Rutherford Appleton Laboratory}
\date{17th October 2011}
\maketitle

The unfolding method based on iterative application of Bayes' theorem
described by D'Agostini~\cite{D'Agostini:1994zf}
(though similar to the iterative procedure of
M\"ulthei and Schorr~\cite{Multhei:1986ps})
is a convenient method, popular in Particle Physics.

\subsection{Measurement uncertainties}

As with all unfolding methods, it is important to understand
the uncertainties in the unfolded distribution, and especially the bin-to-bin correlations
that ensue as a result of the regularisation process (in the Bayes method without additional smoothing, regularisation
comes about as a result of limiting the number of iterations).
In many cases, the largest source of uncertainty is from propagation of the measurement
uncertainties through the unfolding matrix.

D'Agostini (\cite{D'Agostini:1994zf}~section~4) gives the unfolded distribution (``estimated causes''), $\hat{n}(\C_i)$,
as the result of applying the unfolding matrix, $M_{ij}$, to the measurements (``effects''), $n(\E_j)$:

\begin{equation}
\hat{n}(\C_i) = \sum_{j=1}^{n_{\E}} M_{ij} n(\E_j)
\label{eq:nCi}
\end{equation}

\noindent where

\begin{equation}
M_{ij} = \frac{P(\E_j|\C_i) n_0(\C_i)}{\epsilon_i f_j}
\end{equation}

\noindent and
$P(\E_j|\C_i)$ is the response matrix,
$\epsilon_i \equiv \sum_{j=1}^{n_{\E}} P(\E_j|\C_i)$ are the efficiencies, and
$f_j \equiv \sum_{l=1}^{n_{\C}} P(\E_j|\C_l) n_0(C_l)$ is the folded
prior distribution, $n_0(C_l)$ --- initially arbitrary (eg. flat or MC model), but updated on
subsequent iterations.

D'Agostini then calculates the covariance matrix, which here we call $V(\hat{n}(\C_k),\hat{n}(\C_l))$,
by error propagation from $n(\E_j)$,
but assumes that $M_{ij}$ is itself independent of $n(\E_j)$. That is only true for the first iteration.
For subsequent iterations, $n_0(\C_i)$ is replaced by $\hat{n}(\C_i)$ from the
previous iteration, and $\hat{n}(\C_i)$ depends on $n(\E_j)$ (Eq.~(\ref{eq:nCi})).

To take this into account, we compute the error propagation matrix

\begin{equation}
\dd{\hat{n}(\C_i)}{n(\E_j)} = M_{ij} +
\frac{\hat{n}(\C_i)}{n_0(\C_i)} \dd{n_0(\C_i)}{n(\E_j)} -
\sum_{k=1}^{n_{\E}} \sum_{l=1}^{n_{\C}} \frac{n(\E_k) \epsilon_l}{n_0(\C_l)} M_{ik} M_{lk} \dd{n_0(\C_l)}{n(\E_j)}
\label{eq:dnCidnEj}
\end{equation}

where $\hat{n}(\C_i)$ is the unfolded result from Eq.~(\ref{eq:nCi}).
This depends upon the matrix $\dd{n_0(\C_i)}{n(\E_j)}$, which is $\dd{\hat{n}(\C_i)}{n(\E_j)}$ from the previous iteration.
In the first iteration, $\dd{n_0(\C_i)}{n(\E_j)}=0$ and we get $\dd{\hat{n}(\C_i)}{n(\E_j)} = M_{ij}$.

We can use the error propagation matrix to obtain the covariance matrix on the unfolded distribution

\begin{equation}
V(\hat{n}(\C_k),\hat{n}(\C_l)) = \sum_{i,j=1}^{n_{\E}} \dd{\hat{n}(\C_k)}{n(\E_i)} V(n(\E_i),n(\E_j)) \dd{\hat{n}(\C_l)}{n(\E_j)}
\label{eq:Vij}
\end{equation}

\noindent from the covariance matrix of the measurements, $V(n(\E_i),n(\E_j))$.

Without the additional terms in Eq.~(\ref{eq:dnCidnEj}),
the error is underestimated if more than one iteration
is used, but agrees well with toy Monte Carlo tests if the full error propagation is used,
as shown in Fig.~\ref{fig:bayes_errors}.%
\begin{figure}[ht]
\makebox[\textwidth]{\includegraphics[width=.47\textwidth,clip]{bayes_errors_old.eps}\hfill
                     \includegraphics[width=.47\textwidth,clip]{bayes_errors_new.eps}}%
\caption
[Bayesian unfolding measurement errors compared to toy MC]%
{Bayesian unfolding measurement errors compared to toy MC RMS for 1, 2, 3, and 9 iterations.
The left-hand plot shows the errors using D'Agostini's original method,
ignoring any dependence on previous iterations (only the first term of Eq.~(\ref{eq:dnCidnEj})).
The right-hand plot shows the full error propagation.}%
\label{fig:bayes_errors}%
\end{figure}

D'Agostini takes a multinomial distribution for the bin contents, and hence

\begin{equation}
V(n(\E_i),n(\E_j)) = n(\E_i) \delta_{ij} - \frac{n(\E_i) n(\E_j)}{\hat{N}_{\mathrm{true}}}
\end{equation}

where $\hat{N}_{\mathrm{true}} \equiv \sum_{i=1}^{n_{\C}} \hat{n}(\C_i)$.
That describes a histogram with fixed normalisation, ie. fixed total number of measured events.
On the other hand, in counting experiments common in particle physics, each bin is independently Poisson distributed, with

\begin{equation}
V(n(\E_i),n(\E_j)) = n(\E_i) \delta_{ij}
\end{equation}

Other, arbitrary, bin errors (perhaps even correlated) may also be used in Eq.~(\ref{eq:Vij}).

\subsection{Response matrix uncertainties}

The response matrix, $P(\E_j|\C_i)$, is usually estimated by Monte Carlo. If only limited MC statistics
are available, then there will be uncertainties on the matrix elements.
These may be propagated as an uncertainty on the unfolded result.

A correction is also required to D'Agostini's treatment of this response matrix error propagation
to account for the fact that, after the first iteration, the prior, $n_0(\C_i)$, depends on the response matrix.
Taking this into account, the error propagation matrix for the response is

\begin{eqnarray}
\dd{\hat{n}(\C_i)}{P(\E_j|\C_k)} & = & \frac{1}{\epsilon_i} \left( \frac{n_0(\C_i) n(\E_j)}{f_j} - \hat{n}(\C_i) \right) \delta_{ik}  - \frac{n_0(\C_k) n(\E_j)}{f_j} M_{ij} + \nonumber \\
& & \frac{\hat{n}(\C_i)}{n_0(\C_i)} \dd{n_0(\C_i)}{P(\E_j|\C_k)} - \frac{\epsilon_i}{n_0(\C_i)} \sum_{l=1}^{n_{\E}} \sum_{r=1}^{n_{\C}} n(\E_l) M_{il} M_{rl} \dd{n_0(\C_r)}{P(\E_j|\C_k)}
\label{eq:dnCidR}
\end{eqnarray}

where $\dd{n_0(\C_i)}{P(\E_j|\C_k)}$ is the error propagation matrix from the previous iteration,
$\dd{\hat{n}(\C_i)}{P(\E_j|\C_k)}$. For the first iteration, this is zero and the final two terms in
Eq.~\ref{eq:dnCidR} disappear, leaving us with an expression compatible with D'Agostini's for $\dd{M_{ki}}{P(\E_r|\C_u)}$.

The covariance matrix due to these errors is given by

\begin{equation}
V(\hat{n}(\C_k),\hat{n}(\C_l)) = \sum_{j,s=1}^{n_{\E}} \sum_{i,r=1}^{n_{\C}} \dd{\hat{n}(\C_k)}{P(\E_j|\C_i)} V(P(\E_j|\C_i),P(\E_s|\C_r)) \dd{\hat{n}(\C_l)}{P(\E_s|\C_r)}
\end{equation}

where $V(P(\E_j|\C_i),P(\E_s|\C_r))$ can be taken as multinomial (as D'Agostini does), Poisson, or other distribution.

The effect of the new response matrix error calculation can be seen in Fig.~\ref{fig:bayes_errors_sys}.%
\begin{figure}[ht]
\makebox[\textwidth]{\includegraphics[width=.47\textwidth,clip]{bayes_errors_sys_old.eps}\hfill
                     \includegraphics[width=.47\textwidth,clip]{bayes_errors_sys_new.eps}}%
\caption
[Bayesian unfolding response matrix errors compared to toy MC]%
{Bayesian unfolding response matrix errors compared to toy MC RMS for 1, 2, 3, and 9 iterations.
The left-hand plot shows the errors using D'Agostini's original method,
ignoring any dependence on previous iterations (only the first line of Eq.~(\ref{eq:dnCidR})).
The right-hand plot shows the full error propagation.}%
\label{fig:bayes_errors_sys}%
\end{figure}

\begin{thebibliography}{99}

\bibitem{D'Agostini:1994zf}
  G.~D'Agostini,
  ``A Multidimensional unfolding method based on Bayes' theorem,''
  Nucl.\ Instrum.\ Meth.\  A {\bf 362} (1995) 487.
  %%CITATION = NUIMA,A362,487;%%

\bibitem{Multhei:1986ps}
  H.~N.~M\"ulthei and B.~Schorr,
  ``On an Iterative Method for the Unfolding of Spectra,''
  Nucl.\ Instrum.\ Meth.\  A {\bf 257} (1987) 371.
  %%CITATION = NUIMA,A257,371;%%

\end{thebibliography}
\end{document}


\section{R1CS Definition}\label{sec:r1cs-def}

\parhead{Background: statement representations}
Many known zk-SNARKs are generic and can express any \emph{NP statement} (i.e., any statement expressed as ``for the instance $x$ there exists a witness $r$ such that relation $D$ holds on $x$ and $w$'' where the relation $D$ is efficiently computable) \cite{ZKProofSecurity}. In principle this already includes convenient high-level representations, and indeed there are research prototypes, for example, for SNARKs supporting general C programs~\cite{BCGTV13csnark-crypto}. However, each zk-SNARK cryptographic construction has a ``native language'' which is a specific type of NP languages. Anything else needs to be converted (reduced) to the native representation, with some overhead. Here we discuss ``R1CS'', one such low-level native representation, and how it can be used to efficiently express useful NP statements to be proven by zk-SNARK constructions.

Moreover, expressing NP statements for SNARKs differs from usual programming (whether in JavaScript or assembly) in the following sense. Usual programming aims to \emph{compute} the output of some function of the given inputs, whereas the NP statement's relation is already given the instance (e.g., the claimed output of a function on some input), and aim to \emph{check} its correctness given some additional information from the witness. This affects the programming style and creates opportunities for important optimizations, such as providing additional advice within the witness that aid the checking.

\parhead{Constraint systems}

Today's best zk-SNARK constructions have a native language consisting of \emph{constraint systems}, i.e., a set of constraints on the instance and witness variables.%
%
\footnote{Colloquially, zk-SNARK developers often refer to constraint systems as ``circuits'', but as we shall soon see, the typical constraint system formalism is more general than Boolean or arithmetic circuits.} %
%
The most common such language is \emph{Rank 1 Constraint System (R1CS)}, also known as \emph{Bilinear Constraint Systems (BCS)}, where each constraint is a (bilinear) quadratic equation over a large finite field.  This is the native language of many efficient zk-SNARKs, such as \cite{GGPR13qsp,PHGR13pinocchio,BCGTV13csnark-crypto,BCTV13von,Groth16size} and many others (see~\cite{ZKProofImplementation}), and is used in practical deployments.
This style of constraint system was introduced in~\cite{GGPR13qsp}.%
%
\footnote{The initial works~\cite{GGPR13qsp,PHGR13pinocchio} used an equivalent formulation called \emph{Quadratic Arithmetic Programs (QAP)}, where the constraints are expressed in terms of polynomials. Later works tend to abstract this away, and do the conversion into polynomials internally.} %
%
The variant defined below is very similar to the definition in \cite[Appendix~E]{BCGTV13csnark-eprint}, used in \href{https://github.com/scipr-lab/libsnark}{libsnark}.%
%
\footnote{To be precise, ``R1CS'', as defined here, is the same as the ``system of rank-1 quadratic equations'' defined in \cite[Appendix~E]{BCGTV13csnark-eprint}, with a minor change in notation. In defining BCS we let $\vwit$ denote just the witness, whereas \cite{BCGTV13csnark-eprint} let $\mathbf{w}$ denote the concatenation of the instance and the witness (and thus added corresponding consistency checks between $\mathbf{w}$ and the instance $\mathbf{x}$). These are also essentially identical to the ``Rank 1 Constraint System (R1CS)'' of \href{https://github.com/scipr-lab/libsnark}{libsnark}.}

An R1CS reasons about two vectors: the \emph{instance} consisting of $\NumInst$ elements and denoted $\vinst=(\inst_1,\ldots,\inst_\NumInst)$, and the \emph{witness} consisting of $\NumWit$ elements and denoted $\vwit=(\wit_1,\ldots,\wit_\NumWit)$. The constraint system says that the two are related by some number of constraints, $\NumConstr$, each of which is a quadratic equation of a specific form. All of the elements and operations are over a large prime field $\Field$, which we will represent here as the integers modulo a large prime $p$.%
\footnote{The specific prime, and the representation of field elements, are related to the elliptic curve used in the QAP-based zkSNARK constructions.}

\begin{definition}
\label{def:bcs}
Let $\NumInst,\NumWit,\NumConstr$ be positive integers, and $\NumConc=\NumInst+\NumWit+1$.
A \defem{Rank 1 Constraint System (R1CS)},
over $\Field$ is a tuple
$\System = \big((\vec{\lcoef}_{j},\vec{\rcoef}_{j},\vec{\ocoef}_{j})_{j=1}^{\NumConstr},\NumInst\big)$
where $\vec{\lcoef}_{j},\vec{\rcoef}_{j},\vec{\ocoef}_{j} \in \Field^{\NumConc}$.
Such a system $\System$ is \defem{satisfiable} with an input $\vinst \in \Field^{\NumInst}$ if the following is true:

\NPstatement
  {For these
    $\System = \big((\vec{\lcoef}_{j},\vec{\rcoef}_{j},\vec{\ocoef}_{j})_{j=1}^{\NumConstr},\NumInst\big)$
   and $\vinst \in \Field^{\NumInst}$}
  {there exists a witness $\vwit \in \Field^{\NumWit}$}
  {such that
  $\bigip{\vec{\lcoef}_{j}}{\vconc} \cdot \bigip{\vec{\rcoef}_{j}}{\vconc}  = \bigip{\vec{\ocoef}_{j}}{\vconc}$ \hskip0.4em for all for $j \in [\NumConstr]$, where $\vconc=(1,\vinst,\vwit)\in \Field^{\NumConc}$
  }
\noindent
Above, $\ip{\cdots}{\cdots}$ denotes inner product of vectors over $\Field$, and $\cdot$ denotes multiplication in $\Field$.
\end{definition}

For example, the R1CS
  \begin{equation}\label{eq:bcs-example-formal}
  \System=\left(\left(
  \begin{aligned}
   \vec{\lcoef}_1 = (0,3,0,0),\, \vec{\rcoef}_1 = (1,1,0,0),\, \vec{\ocoef}_1 = (4,0,5,0) \\
   \vec{\lcoef}_2 = (0,1,2,0),\, \vec{\rcoef}_2 = (6,0,1,0),\, \vec{\ocoef}_2 = (7,0,0,1)
  \end{aligned}
  \right),1 \right)
  \end{equation}
represents the two bilinear constraints
  \begin{equation}
  \begin{aligned}
  \bigip{(0,3,0,0)}{\vconc}\cdot\bigip{(1,1,0,0)}{\vconc} &= \bigip{(4,0,5,0)}{\vconc} \\
  \bigip{(0,1,2,0)}{\vconc}\cdot\bigip{(6,0,1,0)}{\vconc} &= \bigip{(7,0,0,1)}{\vconc}
  \end{aligned}
  \quad \mbox{ where } \vconc=(1,\inst_1,\wit_1,\wit_2)
  \end{equation}
or simply:
  \begin{equation}\label{eq:bcs-example-informal}
  \begin{aligned}
  3\inst_1 \cdot (1+\inst_1) &= 5\wit_1 + 4 \\
  (\inst_1 + 2\wit_1) \cdot (\wit_1+6) &= \wit_2 + 7
  \end{aligned}
  \end{equation}
%with $\NumInst=1,\NumWit=2,\NumConstr=2$

By definition, this system $\System$ is satisfiable with input $\inst_1$ if the following is true:
\NPstatementNOperiod
  {For the instance $\inst_1$}
  {there exist a witness $\wit_1,\wit_2$}
  {such that
  \begin{equation*}
  \begin{aligned}
  3\inst_1 \cdot (1+\inst_1) &= 5\wit_1 + 4 \\
  (\inst_1 + 2\wit_1) \cdot (\wit_1+6) &= \wit_2 + 7
  \end{aligned}
  \end{equation*}
  }

When we prove this NP statement using zk-SNARK, the instance will be the public input that is visible to the verifier, and the witness will be the additional data that is known only to the prover, and used in the construction of the proof, but whose privacy is protected by the zero-knowledge property.

\parhead{Complexity measure}
For backends that consume R1CS natively, the cost of producing the zk-SNARK proofs (in time and memory) typically depends mostly on the \textbf{number of constraints}, $\NumConstr$.

In particular, multiplying variables (or linear combinations thereof) is ``expensive'' (each such multiplication needs a new constraint and thus increases $\NumConstr$), but linear combinations (i.e., additions and multiplications by constants) come ``for free'' in each constraint.


\documentclass[letterpaper,11pt]{article}
\usepackage[utf8]{inputenc}
\usepackage{graphicx}
\usepackage{rotating}
\usepackage{amsbsy}
\usepackage{array}
\usepackage{booktabs}
\setlength{\heavyrulewidth}{1.5pt}
\setlength{\abovetopsep}{4pt}
\newenvironment{packed_item}{
\begin{itemize}
  \setlength{\itemsep}{1pt}
  \setlength{\parskip}{0pt}
  \setlength{\parsep}{0pt}
}{\end{itemize}}
\newenvironment{packed_enum}{
\begin{enumerate}
  \setlength{\itemsep}{1pt}
  \setlength{\parskip}{0pt}
  \setlength{\parsep}{0pt}
}{\end{enumerate}}
\usepackage{epstopdf}
\usepackage{amssymb}
\usepackage{hyperref}
\usepackage{amsthm}
\usepackage{amsmath}
\setlength{\parindent}{0in}
\usepackage[top=3cm, bottom=3cm, left=3cm, right=3cm]{geometry}
\usepackage{fancyhdr}
\pagestyle{fancy}
\newcommand{\V}{\mathbf{V}} % expectation
\newcommand{\E}{\mathbf{E}} % expectation
\newcommand{\Cov}{\mathbf{Cov}} % covariance
\newcommand{\cond}{\, | \,} % conditional
\renewcommand{\P}{\mathbf{P} } % prob
\renewcommand{\d}{\, \mathrm{d}} % differential
\renewcommand{\t}{^\intercal} % transpose
\newcommand{\1}{\mathbf{1}} % indicator
\newtheorem{theorem}{Theorem}[section]
\newtheorem{property}{Property}[section]
\newtheorem{definition}{Definition}[section]
\begin{document}
\lhead{STA521: Linear Algebra and Linear Models}
\rhead{Spring 2017}

\begin{center}
\Large \textbf{STA521: Linear Algebra and Some Linear Model Theory} \\
\vspace{.5pc}
\large Víctor Peña \\
\end{center}
\tableofcontents

\newpage

\section{Matrix Algebra}
Basically taken from Sam Roweis' notes\footnote{Link:
  \url{http://www.cs.nyu.edu/~roweis/notes/matrixid.pdf}. Also of
  interest: \url{http://www.cs.nyu.edu/~roweis/notes/gaussid.pdf} (Gaussian identities).}:
\begin{align*}
A(B+C) &= AB + AC \\
(A+B)\t &= A\t + B\t \\
(AB)^{-1} &= B^{-1} A^{-1} \\
(A^{-1}) \t &= (A \t)^{-1} \\
|AB| &= |A| |B| \\
|A^{-1}| &= 1/|A| \\
|A| &= \prod \mathrm{evals}(A) \\
|c A_{n \times n}| &= c^n |A_{n \times n}| \\
\mathrm{tr}(A) &= \sum \mathrm{evals}(A) \\
\mathrm{tr}(ABC) &= \mathrm{tr}(BCA) = \mathrm{tr}(CAB) \\
\mathrm{rank}(A) &= \mathrm{rank}(A \t A) = \mathrm{rank}(A A \t) \\
X \t X &\text{ and } X X \t \text{ are positive-semidefinite and symmetric}.
\end{align*}
\section{Linear Algebra}
Most of the results here can be found in Appendix B in Christensen.
\subsection{Column Space and Rowspace}
Let $X$ be $n \times p$.
\begin{align*}
C(X) &= \{X a \in \mathbb{R}^n : a \in \mathbb{R}^p\} \\
N(X) &= \{v \in \mathbb{R}^p : X v = 0\} \\
p & = \mathrm{dim}(C(X)) + \mathrm{dim}(N(X)) \\
C(X \t) &= N(X)^\perp \text{ (they are orthogonal complements)}\\
N(X \t) &= C(X)^\perp \\
C(X) &= C(X \t X)
\end{align*}
\subsection{Orthogonal Matrices}
Let $Q$ be a $n \times n$ matrix.
\begin{packed_item}
\item $Q$ is orthogonal if its columns
form an orthonormal basis of $\mathbb{R}^n$ (that is, if they are
orthogonal unit vectors).
\item Equivalently, $Q$ is orthogonal if $Q^T =
Q^{-1}$.
\item If $Q$ is orthogonal, then $|Q|^2 = 1$.
\end{packed_item}
\subsection{Spectral Theorem and Decompositions}
\textbf{Spectral Theorem:} If $X$ is a symmetric matrix, it has
real eigenvalues and it can be
decomposed as $X = U \Lambda U^\intercal$, where $\Lambda$ is a
diagonal matrix with the eigenvalues of $X$ and $U$ is an orthogonal
matrix with the eigenvectors of $X$. The rank of $X$ equals the number of nonzero eigenvalues of
$X$. Based on this decomposition, one can define matrix powers as $X^p
= U \Lambda^p U^\intercal$. \\

\textbf{SVD:} Let $X$ be a $n \times p$ matrix, then one can find $U$ ($n
\times n$), $\Sigma$ ($n \times p$) and $V$ ($p \times p$) such that
$X = U \Sigma V \t$, where $U$ and $V$ are orthogonal matrices. The
columns of $U$ are eigenvectors of $X X \t$, and the columns of $V$
are eigenvectors of $X \t X$. $\Sigma$ is a rectangular diagonal matrix
with the square roots of the eigenvalues of $X \t X$ or $X X \t$ (they
are the same). \\

\textbf{Cholesky:} Any positive definite matrix $X$ can be factorized
as $X = L L \t$, where $L$ is a lower triangular matrix. \\

\textbf{QR:} Any square matrix $X$ can be decomposed as $X = QR$,
where $Q$ is an orthogonal matrix and $R$ is upper triangular.

\subsection{Generalized Inverses}

\textbf{Definition:} A generalized inverse of a matrix $X$ is any matrix $X^{-}$ such that $XX^{-}X = X$. Generalized inverses exist for arbitrary matrices. \\

\textbf{Results:}
\begin{packed_item}
\item Let $X$ be a symmetric matrix. The Moore-Penrose generalized inverse is defined as follows. First, decompose $X = U \Lambda U^\intercal$. Then $X^{-} = U \Lambda^{-} U^\intercal$ where $\lambda^{-}_i = 1/\lambda_i$ if $\lambda_i \neq 0$ and $\lambda^{-}_i = 0$ if $\lambda_i = 0$. The Moore-Penrose generalized inverse is nice because it is symmetric and reflexive (i.e. $A^{-} A A^{-} = A^{-}$).
\item If $G$ and $H$ are generalized inverses of $X^\intercal X$, then $XGX^\intercal X = XHX^\intercal X = X$ and $XGX^\intercal = XHX^\intercal$.
\end{packed_item}

\subsection{Orthogonal Projections}
\textbf{Definition:} $P$ is a perpendicular projection operator (ppo) onto $C(X)$ if and only if
\begin{packed_item}
\item $v \in C(X)$ implies $Pv = v$.
\item $v \perp C(X)$ implies $Pv = 0$.
\end{packed_item}

\textbf{Properties:}
\begin{packed_item}
\item $P = X (X\t X)^- X\t$ is the ppo onto $C(X)$. If $X \t X$ is
  invertible, $P = X (X\t X)^{-1} X\t$.
\item $P$ is a ppo onto $C(P)$ if and only if $PP = P$ (idempotent) and $P^\intercal = P$ (symmetry).
\item Ppos are unique.
\item If $P$ is ppo onto $C(X)$, then $C(X) = C(P)$.
\item If $P$ is ppo onto $C(X)$, $(I - P)$ is ppo onto $C(X)^\perp$.
\item Let $P_1$ and $P_2$ be ppos onto $C(P_1)$, $C(P_2)$, then $P_1 +
  P_2$ is the ppo onto $C(P_1,P_2)$ if and only if $C(P_1) \perp
  C(P_2)$.
\item If $\{o_1, o_2, \, ... \, , o_r\}$ is an orthonormal basis of $C(X)$
  and we construct $O$ such that its columns are $\{ o_1, o_2, \, ... \,
  , o_r \}$, then $O O \t$ is the ppo onto $C(X)$.
\item If $P$ is ppo its eigenvalues are either 0 or 1, and $\mathrm{tr}(P) = \sum \mathrm{evals}(P) = r(P)$.
\end{packed_item}

\section{Random Vectors}
\textbf{Definition:} Let $Y$ be a random vector such that $E(Y) = \mu$. The covariance matrix of $Y$ is
$$
\mathrm{Cov}(Y) = E[(Y-\mu)(Y-\mu)^\intercal].
$$
The elements of the diagonal are variances, the off-diagonal elements are covariances.  \\

\textbf{Properties} \\
Let $Y$ be a random vector with $E(Y)
= \mu$ and $\mathrm{Cov}(Y) = \Sigma$, and let $A$ and $b$ be a matrix
and a vector with constants, respectively:
\begin{packed_item}
\item $E(AY + b) = A \mu + b$.
\item $\mathrm{Cov}(AY + b) = A \Sigma A^\intercal$.
\item \textbf{Expectation of a quadratic form:} Assume $A$ is symmetric, then $E(Y^\intercal A Y) = \mathrm{tr}(A\Sigma) + \mu^\intercal A \mu$.
\end{packed_item}

\section{Multivariate Normal}
These are notes I took after watching the videos on the
Multivariate Normal that Jeff Miller uploaded to his YouTube\footnote{YouTube
  channel:
  \url{https://www.youtube.com/user/mathematicalmonk}.}. \\

\textbf{Definition:} $Y = (Y_1, \, ... \, Y_n)^\intercal$ follows a Multivariate Normal (MVN) with mean $\mu$ and covariance $\Sigma$ if any linear combination of the components is $v^\intercal Y \sim$ Normal$(v^\intercal \mu, v^\intercal \Sigma v)$, for $v \in \mathbb{R}^n$. If $|\Sigma| = \mathrm{det}(\Sigma) \neq 0$, $Y$ has the following pdf:
$$
f(Y) = \frac{1}{\sqrt{|2\pi\Sigma|}} \exp\left\{-\frac{1}{2}\left[ (Y-\mu)^\intercal \Sigma^{-1} (Y-\mu)\right] \right\}.
$$

\textbf{Remark:} Recall that if $c \in \mathbb{R}$ and $A$ is an $n \times n$ matrix, then $|c A| = c^n |A|$.

\subsection{Zero Correlation and Independence}

Let $Y \sim$ MVN($\mu, \Sigma$). For an arbitrary partition
$$
\left[ \begin{array}{c} Y_1 \\ Y_2 \end{array} \right] \sim \text{MVN} \left( \left[ \begin{array}{c} \mu_1 \\ \mu_2 \end{array} \right] , \left[ \begin{array}{cc} \Sigma_{11} & \Sigma_{12} \\ \Sigma_{21} & \Sigma_{22} \end{array} \right] \right),
$$
then $\mathrm{Cov}(Y_1, Y_2) = \Sigma_{12} = 0$ if and only if $Y_1$ and $Y_2$ are independent. That is, uncorrelation implies independence. \\

\textbf{Corollary:} $Y \sim$ MVN($\mu, \sigma^2 I$) and $AB^\intercal = 0$, then $AY$ and $BY$ are independent. \\

\textbf{Caution!} $X_1$ and $X_2$ normally distributed does not imply $(X_1, X_2) \sim$ MVN. For example, let $X_1 \sim$N($0,1$) and
$$
X_2 = \begin{cases} X_1 & \text{if } |X_1| \le 1 \\ - X_1 & \text{if } |X_1| > 1. \end{cases}
$$
$X_1$ and $X_2$ are Normal, but $(X_1, X_2)$ is not Multivariate Normal. Also, if two Normal random variables are uncorrelated, it doesn't mean they're independent (unless they are jointly MVN!).

\subsection{Affine Property}

Any affine transformation of a MVN is MVN. If $X \sim$ MVN($\mu, \Sigma$), then $AX + b \sim$ MVN($A\mu +b , A \Sigma A^\intercal)$, for any matrix of constants $A$ and vector $b$ of conformable sizes. \\

Some operations:
\begin{packed_item}
\item \textbf{Constructing:} If $X_1, X_2, \, ... \, , X_n \sim$
  N$(0,1)$ are independent, then $(X_1, X_2, \, ... \, , X_n) = X
  \sim$ MVN($0, I$). We have $AX + \mu \sim $ MVN($\mu, \Sigma)$,
  where $\Sigma = AA^\intercal$. Using the spectral theorem, one can
  find $A$ such that $AA^\intercal = \Sigma$ (i.e. you can construct
  any MVN from standard Normal rvs). Recall that, by the spectral theorem,  $\Sigma = U \Lambda U^\intercal = U \Lambda^{1/2} \Lambda^{1/2}U^{\intercal} = AA^\intercal$, where $U$ is an orthogonal matrix.
\item \textbf{Sphering:} If $Y \sim$ MVN($\mu, \Sigma$) and $\Sigma$ is invertible, then $A^{-1}(Y-\mu) \sim $ MVN$(0,I)$, where $\Sigma = AA^\intercal$. Alternatively, one can write this as follows: $\Sigma^{-1/2} (Y - \mu) \sim$ MVN$(0,I)$.
\end{packed_item}

\subsection{Marginals and conditionals}

Let $Y \sim$ MVN($\mu, \Sigma$). Marginal distributions are Normal with the same parameters as in the MVN. \\

Let
$$
\left[ \begin{array}{c} Y_1 \\ Y_2 \end{array} \right] \sim \text{MVN} \left( \left[ \begin{array}{c} \mu_1 \\ \mu_2 \end{array} \right] , \left[ \begin{array}{cc} \Sigma_{11} & \Sigma_{12} \\ \Sigma_{21} & \Sigma_{22} \end{array} \right] \right),
$$
then
$$
Y_1 | Y_2 = a \sim \text{MVN}(\mu_1 + \Sigma_{12}\Sigma^{-1}_{22}(a - \mu_2), \Sigma_{11} - \Sigma_{12}\Sigma^{-1}_{22}\Sigma_{21}).
$$

\section{Chi-square Distribution}

\textbf{Definition:} Let $Z_j$ for $j \in \{1, 2, \, ... \, , p\}$ be iid
$N(0,1)$, and let $Z = (Z_1, Z_2, \, ... \, , Z_p)^\intercal$. Then, $X = Z^\intercal Z \sim \chi^2_p$.  \\

\textbf{Noncentral chi-square:} Let
$Z_j$ for $j \in \{1, 2, \, ... \, , p\}$ be iid $N(\mu,1)$ and let $Z
= (Z_1, Z_2, \, ... \, , Z_p)^\intercal$. Then, $Z^\intercal Z \sim \chi^2_p(\sum_{j=1}^p \mu^2_j)$. \\

\textbf{Quadratic forms and $\chi^2$ distributions:} Let $Y \sim$
MVN$(0, I)$ and let $P$ be a symmetric $n \times n$ matrix of rank
$k$. Then $Y^\intercal P Y \sim \chi^2_{k}$ if and only if $P$ is a
rank $k$ perpendicular projection operator (ppo). In the noncentral case, if $Y
\sim$ MVN($\mu, I$), then $Y^\intercal P Y \sim
\chi^2_{k}(\mu^\intercal P \mu/2)$ if and only of $P$ is a
ppo. Application: let $Y \sim$ MVN$(\mu, I)$ with
$\mu \in C(X)$ and $(I-P)$ be a rank $k$ ppo onto $C(X)^\perp$. Then
$Y^\intercal (I-P) Y \sim \chi^2_k$.

\section{A Little Bit of Linear Model Theory}

A lot of important results are not even mentioned here. I strongly
recommend looking at Faraday's ``Practical Regression and ANOVA using
R''\footnote{Link:
  \url{http://cran.r-project.org/doc/contrib/Faraway-PRA.pdf}} if
you think Christensen is too dry and need some extra intuition. \\

In matrix notation:
$$
Y = X\beta + \varepsilon
$$
where
\begin{packed_item}
\item $Y$ is the response variable, which is observable.
\item $X$ is the design matrix, also observable.
\item $\beta$ are the unknown coefficients, which we want to estimate.
\item $\varepsilon$ are the errors, also unobservable.
\end{packed_item}

Assume $X$ is full rank, then the Ordinary Least Squares (OLS) estimator is
$$
\hat{\beta}_{OLS} = \arg \min_\beta \sum_{i=1}^n (Y_i - x^\intercal_i
\beta)^2 =  \arg \min_\beta (Y - X\beta)^\intercal (Y - X\beta) =
\arg \min_\beta || Y - X \beta||^2_2 = (X \t X)^{-1} X \t Y.
$$
If $X$ is not full rank $\hat{\beta}_{\mathrm{OLS}} = (X \t X)^{-} X \t Y$. \\

Let $Y | \beta \sim $ MVN($X\beta, \sigma^2 I)$. Then
$\hat{\beta}_{\mathrm{MLE}} = \hat{\beta}_{\mathrm{OLS}} = \hat{\beta}$. Now, let $\mu = X\beta$. Then
$\hat{\mu} = X \hat{\beta} = P_X Y$ is the MLE of $\mu$, where $P_X$
is the ppo onto $C(X)$. On the other hand, the MLE of $\sigma^2$ is
$\hat{\sigma}^2 = ||(I-P_X)Y||^2/n = e^\intercal e/n$, where $e =
(I-P_X)Y$ are the residuals and $(I-P_X)$ is the ppo onto the
orthogonal complement of $C(X)$.

\subsection{Inference}

Again, $Y \mid \beta \sim$ MVN($X\beta, \sigma^2I$). \\

The MLE of $\mu = X\beta$ is $\hat{\mu} = P_X Y$, and it is an unbiased estimate:
$$
E(\hat{\mu}) = E(P_X Y) = P_X E(Y) = P_X \mu = \mu,
$$
because $\mu \in C(X)$. \\

The residuals $e = (I-P_X)Y$ have mean zero:
$$
E(e) = E[(I-P_X)Y] = (I - P_X) E(Y) = (I - P_X) \mu = 0,
$$
because $\mu \perp C(X)^\perp$. \\

The residuals are correlated:
$$
\mathrm{Cov}(e) = \mathrm{Cov}[(I-P_X)Y] = \sigma^2 (I-P_X)(I-P_X) = \sigma^2 (I - P_X).
$$
Therefore $e \sim $ MVN($0, \sigma^2(I-P_X))$ by the affine property
of the MVN. \\

Now we find the expectation of the MLE of $\sigma^2$, which is $\hat{\sigma}^2 = e^\intercal e /n = ||(I - P_X) Y||^2/n$:
$$
E(||(I - P_X) Y||^2) = E[ Y^\intercal (I-P_X) Y] = \sigma^2 \mathrm{tr}(I-P_X) + \mu^\intercal (I - P_X) \mu = \sigma^2(n-r(X)),
$$
Therefore, the MLE is biased, but $e^\intercal e/(n-r(X))$ is
unbiased. \\

Let $Y | \beta \sim$ MVN($X\beta, \sigma^2 I)$. The MLE for $\beta$ is $\hat{\beta} = (X^\intercal X)^{-} X^\intercal Y$. We take the Moore-Penrose generalized inverse. Then, the distribution of $\hat{\beta}$ is MVN because it's just a linear combination of $Y$, which is MVN. \\

Taking expectations,
$$
E(\hat{\beta}) = (X^\intercal X)^{-} X^\intercal X \beta,
$$
and the variance is
$$
\mathrm{Cov}(\hat{\beta}) = \sigma^2 (X^\intercal X)^{-} X^\intercal X (X^\intercal X)^{-} = \sigma^2 (X^\intercal X)^{-},
$$
because if $A^{-}$ is a Moore-Penrose generalized inverse, then it is symmetric and $A^{-} A A^{-} = A^{-}$ (reflexive).  \\

Therefore
$$
\hat{\beta} \sim \text{MVN}((X^\intercal X)^{-} X^\intercal X \beta, \sigma^2 (X^\intercal X)^{-})
$$
If $X$ full rank,
$$
\hat{\beta} \sim \text{MVN}(\beta, \sigma^2 (X^\intercal X)^{-1}).
$$
The standard error for a particular component is
$\mathrm{se}(\hat{\beta}_i) = \sqrt{(X^\intercal X)^{-1}_{ii} \sigma
}$. \\

\subsection{Gauss-Markov Theorem}

\begin{packed_item}
\item Gauss-Markov says that OLS has minimum variance in the class of all linear unbiased estimators.
\item Requires just first and second moments, no normality required.
\item Under normality assumption, OLS = MLE has minimum variance out of all unbiased estimators, not just the linear ones.
\item However, we can find estimators with smaller MSE if we allow some bias.
\end{packed_item}

\section{Bayesian Linear Regression}
As usual, we have
$$
Y = X\beta + \varepsilon,
$$
and assume that $\varepsilon$ is normal, yielding
$$
Y \mid \beta, \phi \sim \text{MVN}(X \beta, \phi^{-1} I),
$$
where $\phi$ is the precision (a scalar). \\

Since we're Bayesians now, we need to specify priors on $\beta$ and $\phi$. \\

A convenient (conjugate) choice of priors is
\begin{align*}
\beta \mid \phi &\sim \text{MVN}(b_0, \Phi^{-1}_0/\phi) \\
\phi &\sim \text{Gamma}(v_0/2, SS_0/2).
\end{align*}
Posteriors are
\begin{align*}
\beta \mid \phi, Y &\sim \text{MVN}(b_n, \Phi_n^{-1}/\phi) \\
\phi \mid Y &\sim \text{Ga}(v_n/2,  SS_n/2),
\end{align*}
where
\begin{align*}
b_n &= (X \t X + \Phi_0)^{-1} (X \t X
\hat{\beta}_{\mathrm{OLS}} + \Phi_0 b_0) = (X \t X + \Phi_0)^{-1} (X \t y +
\Phi_0 b_0) \\
\Phi_n &= (X \t X + \Phi_0) \\
v_n &= v_0+n \\
SS_n &= SS_0 +  Y \t Y + b_o\t \Phi_0 b_0 - b_n \t \Phi_n b_n
\end{align*}
\subsection{Marginal of $\beta$  and predictive distribution, all $t$}

The following theorem is really useful for deriving marginal
distributions. \\

\textbf{Theorem:} Let $\theta \mid \phi \sim$ MVN$(m, \Sigma/\phi)$ and $\phi
\sim $ Gamma($\nu/2, \nu \hat{\sigma}^2/2$). Then $\theta \sim
t_{\nu}(m, \hat{\sigma}^2\Sigma)$ with density
$$
p(\theta) \propto \left[ 1 + \frac{1}{\nu} \frac{(\theta-m)^\intercal
    \Sigma^{-1} (\theta - m) }{\hat{\sigma}^2} \right]^{-(p+\nu)/2}.
$$

\textbf{Trick for predictives:} Recall that $\beta \mid \phi, Y \sim$
MVN($b_n, \phi^{-1} \Phi^{-1}_n)$ and $\phi \mid Y \sim \text{Ga}(v_n/2,  SS_n/2)$. Then the new data $Y^\ast = X^\ast \beta + \varepsilon^\ast |
\phi, Y \sim$ MVN$(X^\ast b_n,  \phi^{-1} (X^\ast \Phi^{-1}_n X^{\ast
  \intercal} + I))$. Using the previous theorem, $Y^\ast \mid Y \sim
t_{v_n}(X^\ast b_n, \hat{\sigma}^2(X^\ast \Phi^{-1}_n X^{\ast
  \intercal}+I))$, where $\hat{\sigma}^2_n = SS_n/v_n$.

\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% UMB-CS114-2015F: Introduction to Programming in Java
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/UMB-CS114-2015F
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\def \topDirectory {.}
\def \texDirectory {\topDirectory/src/main/tex}

\documentclass[12pt,letterpaper,twoside]{article}
\usepackage{\texDirectory/template/style/directives}
\usepackage{\texDirectory/template/style/assignment}
\input{\texDirectory/template/config}

\usepackage{amsmath}

\begin{document}

\doc{title}{Solution to Assignment 1}
\doc{date-pub}{Sep 21, 2015 at 5:30 PM}
\doc{date-due}{Oct 05, 2015 at 5:30 PM}
\doc{points}{8}

\prepare{header}

\section*{Question 1}

The following code snippet does not compile. Rewrite the code snippet and create a program \texttt{HelloUMass.java} by resolving problems of the given code such that it compiles successfully and prints \textit{``Hello World!"} on the console once executed.

\lstset{language=Java}
\begin{lstlisting}
public class HelloUMass
	public static void main(String args) {
		System.out.pintln(Hello UMass Boston!")
	}
}
\end{lstlisting}

\subsection*{Solution}

\lstset{language=Java,tabsize=2}
\begin{lstlisting}
public class HelloUMass {
	public static void main(String[] args) {
		System.out.println("Hello UMass Boston!");
	}
}
\end{lstlisting}

\section*{Question 2}

Write a program \texttt{Weather.java} that takes two command-line arguments as \textit{city} and \textit{condition} and functions as shown in following examples.

\begin{terminal}
$ java Weather Boston sunny
Boston is sunny today!
$ java Weather Cambridge cloudy
Cambridge is cloudy today!
\end{terminal}

\subsection*{Solution}

\lstset{language=Java,tabsize=2}
\begin{lstlisting}
public class Weather {
	public static void main(String[] args) {
		System.out.println(args[0] + " is " + args[1] + " today!");
	}
}
\end{lstlisting}

\section*{Question 3}

Write a program \texttt{Temperature.java} that takes temperature in Celsius as command-line argument and prints it's equivalent in fahrenheit. Your program is expected to function as shown in the following example.

\begin{terminal}
$ javac TemperatureCalculator
$ java TemperatureCalculator 12
12.0 Celsius is 53.6 fahrenheit.
\end{terminal}

\subsection*{Solution}

\lstset{language=Java,tabsize=2}
\begin{lstlisting}
public class Temperature {
	public static void main(String[] args) {
		double celsius = Double.parseDouble(args[0]);
		double fahrenheit = celsius * 1.8 + 32;
		System.out.println(celsius + `` Celsius is " + fahrenheit + `` fahrenheit.");
	}
}
\end{lstlisting}

\section*{Question 4}

Given an airplane's acceleration \texttt{a} and take-off speed \texttt{v}, the minimum runway length needed for the airplane to take off can be computed using the formula given in Equation \ref{eq1}.

\begin{equation}
\textrm{length} = \frac{v^2}{2a}
\label{eq1}
\end{equation}

Write a program \texttt{Runway.java} that takes parameters \texttt{v} in meters/second and \texttt{a} in meters/second-squared respectively as command-line arguments and displays the minimum runway length required for it. An example to illustrate the output format is given below.

\begin{terminal}
$ javac Runway.java
$ java Runway 60 3.5
Speed of the airplane is assumed 60.000 m/s
Acceleration of the airplane is assumed 3.500 m/s2
Minimum runway length is 514.286 m
\end{terminal}

\subsection*{Solution}

\lstset{language=Java,tabsize=2}
\begin{lstlisting}
public class Runway {
	public static void main(String[] args) {
		double speed = Double.parseDouble(args[0]);
		double acceleration = Double.parseDouble(args[1]);
		double runwayLength = Math.pow(speed, 2) / (2 * acceleration);
		System.out.printf("Speed of the airplane is assumed %.3f m/s\n", speed);
		System.out.printf("Acceleration of the airplane is assumed %.3f m/s2\n", acceleration);
		System.out.printf("Minimum runway length is %.3f m\n", runwayLength);
	}
}
\end{lstlisting}

\section*{Question 5}

Due to the combined effects of gravitation and rotation, the Earth's shape is roughly that of a sphere slightly flattened in the direction of its axis. Consiquently, the Earth is often approximated by an oblate spheroid instead of a sphere. Mathematically, a spheroid is a quadric surface obtained by rotating an ellipse about one of its principal axes.

Equation \ref{eq2} describes a spheroid centred at the origin with its semi-axes aligned along the coordinate axes, in which semi-axis \texttt{a} is equatorial radius of the spheroid and \texttt{c} is the distance from centre to pole along the symmetry axis.

\begin{equation}
\frac{x^2+y^2}{a^2} + \frac{z^2}{c^2} = 1
\label{eq2}
\end{equation}

Write a program \texttt{Spheroid.java} that takes respectively parameters \texttt{a} and \texttt{c} as command-line arguments and calculates surface area of an oblate spheroid using Equation \ref{eq3}.

\begin{equation}
S_{\text{oblate}} = 2\pi a^2 + \pi \frac{c^2}{e} \ln \left(\frac{1+e}{1-e} \right)
\label{eq3}
\end{equation}

where variable \texttt{e} is as given in Equation \ref{eq4}

\begin{equation}
e = \sqrt{1 - \frac{c^2}{a^2}}
\label{eq4}
\end{equation}

The example given below illustrates the accepted output format.

\begin{terminal}
$ javac Spheroid.java
$ java Spheroid 6 5
Surface area is 403.050
\end{terminal}

\subsection*{Solution}

\lstset{language=Java,tabsize=2}
\begin{lstlisting}
public class Spheroid {
	public static void main(String[] args) {
		double a = Double.parseDouble(args[0]);
		double c = Double.parseDouble(args[1]);
		double e = Math.sqrt(1 - Math.pow(c, 2)/Math.pow(a, 2));
		double surface = 2 * Math.PI * Math.pow(a, 2) + Math.PI * Math.pow(c, 2) / e * Math.log( (1+e) / (1-e));
		System.out.printf("Surface area is %.3f\n", surface);
	}
}
\end{lstlisting}

\end{document}


\documentclass{article}
\usepackage{amsmath}
\renewcommand{\baselinestretch}{1.05}
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}
\usepackage{amsmath,amsthm,verbatim,amssymb,amsfonts,amscd, graphicx}
\usepackage[T1]{fontenc}
\usepackage{bigfoot} % to allow verbatim in footnote
\usepackage[numbered,framed]{matlab-prettifier}
\usepackage{filecontents}
\usepackage{graphics}
\topmargin0.0cm
\headheight0.0cm
\headsep0.0cm
\oddsidemargin0.0cm
\textheight23.0cm
\textwidth16.5cm
\footskip1.0cm
\theoremstyle{plain}
\newtheorem{theorem}{Theorem}
\newtheorem{corollary}{Corollary}
\newtheorem{lemma}{Lemma}
\newtheorem{proposition}{Proposition}
\newtheorem*{surfacecor}{Corollary 1}
\newtheorem{conjecture}{Conjecture}
\newtheorem{question}{Question}
\theoremstyle{definition}
\newtheorem{definition}{Definition}
\title{Chapter 1}
\author{Mike Desgrottes}
\date{October 2020}

\begin{document}
\maketitle

\section{I}
\begin{theorem}
	If $r$ is rational ($r \not = 0$) and $x$ is irrational. Prove that $r + x$ and $rx$ are irrational.
\end{theorem}

\begin{proof}
	Suppose that $r + x$ and $rx$ were rational with $r + x = \frac{p}{q}$ and $rx = \frac{a}{b}$. Then we see that $x = \frac{a}{rb} = \frac{p - qr}{q}$ which is a contradiction. Therefore, $r + x$, and $rx$ are both irrational.
\end{proof}

\begin{theorem}
	Prove that there is no rational number whose square is 12.
\end{theorem}
\begin{proof}
	Let $12 = \frac{p^{2}}{q^{2}}$ with $(p,q) = 1$. $$12q^{2} = p^{2} \implies 2\sqrt{3}q = p \implies 3 | p $$ and $$3 = \frac{p^{2}}{4q^{2}} = \frac{3k}{4q^{2}} \implies (p,q) \geq  3 $$ which is a contradiction.
\end{proof}

\begin{theorem}
	Prove the following

	(a) If $x \not = 0$, $xy = xz \implies y = z$

	(b) If $x \not = 0$, $xy = x \implies x = 1$

	(c) If $x \not = 0$, $xy = 1 \implies y = \frac{1}{x}$

	(d) If $x \not = 0$ $\frac{1}{\frac{1}{x}} = x$
\end{theorem}

\begin{proof}
	(a) $xy = xz \implies x^{-1}(xz) = x^{-1}(xy) \implies x = y$

	(b) $xy = x \implies x^{-1}xy = x^{-1}x \implies y = 1$

	(c) $xy = 1 \implies x^{-1}(xy) = x^{-1} \implies y = x^{-1} = \frac{1}{x}$

	(d) $(\frac{1}{x})(\frac{1}{\frac{1}{x}}) = \frac{x}{x} = 1$
\end{proof}

\begin{theorem}
	Let E be a nonempty ssubset of an ordered set. Suppose $\alpha$ is a lower bound of E, $\beta$ an upper bound of E. Prove that $\alpha \leq \beta$.
\end{theorem}

\begin{proof}
	$\alpha \leq x \leq \beta$ for all $x \in E$, then it follows that $\alpha \leq \beta$.
\end{proof}

\begin{theorem}
	Let A be a nonempty set of real numbers which is bounded below. Let $-A$ be the set of $-x$ for $x \in A$. Prove that $\inf A = -\sup(-A)$.
\end{theorem}

\begin{proof}
	Let B be the set of all lower bound of A. For all $\beta \in B$, $\beta \leq x$ for all x in A. Then $-x \leq -\beta$. So, $-B$ is the set of all upper bound of $-A$ and $\inf A \geq \beta$. This implies that $-\beta \leq - \inf A = - \sup(-A)$. As the infinum is the smallest upper bound for $-A$, it implies that $- \sup(-A) \leq \inf A$ and the result follows.
\end{proof}

\begin{theorem}
	For $b > 1$, prove the following

	(a) If $m,n,p,q$ are integers with $n >0$ and $q > 0$, $$r = \frac{m}{n} = \frac{p}{q} $$. Prove that $(b^{m})^{\frac{1}{n}} = (b^{p})^{\frac{1}{q}}$.

	(b) Prove that $b^{r + s} = b^{r}b^{s}$ if $r,s$ are rational.

	(c) If $x \in \mathbb{R}$, define $B(x) = \{ b^{t}: t \leq x \}$. Prove that $b^{r} = \sup B(r)$ when r is rational.

	(d) Prove that $b^{r + s} = b^{r}b^{s}$ for all real number r,s.
\end{theorem}

\begin{theorem}
	Prove that no order can be defined in the complex field that turn it into an ordered field.
\end{theorem}
\begin{proof}
	Let $<$ be an order of $\mathbb{C}$, then either $i > 0 \implies i^{2} > 0$ or $-i > 0 \implies (-i)^{2} < 0$. In either case, the ordered will violate one of the axiom of an ordered field.
\end{proof}

\begin{theorem}
	Prove that $||x| - |y|| \leq |x - y|$
\end{theorem}

\begin{proof}
	$$|x| = |x - y + y| \leq |x - y| + |y| \implies |x| - |y| \leq |x - y| $$
\end{proof}
\end{document}


\section{Signal modelling}
\label{sec:signal_model}

The shape of the signal and background \myy distributions is described with analytical functions. Like in the past analysis \cite{Hasib:2238687}, the shape of \myy invariant mass distribution for signal events in each category is modelled with a \emph{double-sided Crystal Ball} (DSCB) function. It is a composite function with 6 parameters formed by a Gaussian core, which models the peak, and two power-law tails, as detailed in \Eqn{\ref{eq:DSCB}}:
\begin{align}
	f_\text{DSCB}(\myy) = N \times
	\begin{cases}
		e^{-t^{2}/2} & \text{if } -\alpha_{low} \leq t \leq \alpha_{high} \\
		\frac{ e^{-{}^{1}_{2} \alpha_{low}^{2}} } { \left[ \frac{1}{R_{low}} \left(R_{low} - \alpha_{low} - t \right) \right]^{n_{low}} } & \text{if } t < -\alpha_{low} \\
		\frac{ e^{-{}^{1}_{2} \alpha_{high}^{2}} } { \left[ \frac{1}{R_{high}} \left(R_{high} - \alpha_{high} + t \right) \right]^{n_{high}} } & \text{if } t > \alpha_{high} \\
	\end{cases}
	\label{eq:DSCB}
\end{align}
where $N$ is a normalization factor and the six parameters are
\begin{itemize}
	\item $\mu_\text{CB}$ and $\sigma_\text{CB}$ describe the mean and the width of the Gaussian core, which are combined in $t = \left(\myy - \mu_\text{CB}\right) / \sigma_\text{CB}$;
	\item $\alpha_{low}$ and $\alpha_{high}$ are the positions of the transitions with respect to $\mu_\text{CB}$ from the Gaussian core to power-law tails, in unit of $\sigma_\text{CB}$, on the low and high mass sides respectively;
	\item $n_{low}$ and $n_{high}$ are the exponents of the low and high mass tails. With the $\alpha$'s, they define $R_{low} = \frac{n_{low}}{\alpha_{low}}$ and $R_{high}$ similarly.
\end{itemize}

In the previous analysis \cite{Hasib:2238687}, the DSCB has showed a good $\chi^2$ in the signal only fit and gives a slightly smaller bias on the fitted signal yield using injection tests on Asimov background and signal MC. \\ The advantage of the DSCB is to well separate the contribution coming from the core and from the tails, making easier to apply systematic variations on the scale (on $\mu_\text{CB}$) and on the resolution (to $\sigma_\text{CB}$). \\

Thanks to the very precise knowledge of the Higgs mass from Run1 from the combination of ATLAS and CMS measurements ($\mH=125.09\err{0.21}\stat\err{0.11}\syst \, \si{\GeV}$)~\cite{HIGG-2014-14} and to the relatively small error of the mass scale systematics ($<1\%$), it is possible to parametrize the shape using just one set of MC samples simulated with the Higgs mass \mH fixed at \SI{125}{\GeV}. The MC sample combines all the production modes assuming Standard Model cross sections and the three MC types (mc16a+d+e) are weighted taking into account the relative luminosity of 2015/2016, 2017 and 2018 datasets, respectively. The generalization of the fitted model to any value of \mH (in a small interval close \SI{125}{\GeV}) is done with a simple shift of the $\mu_{CB}$ parameter ($\mu_\text{CB}= m_H + \mu_\text{CB}^{\SI{125}{\GeV}} - \SI{125}{\GeV}$).\\

\textcolor{red}{Systematic from mH:  Higgs mass: 125.09 +/- 0.24 GeV as photon energy scale (PES).}

The signal shapes are fitted in the fixed mass range \SIrange{105}{160}{\GeV} while leaving floating all the six parameters of the DSCB function. The signal models for each category are shown in \Fig{\ref{fig:signal_shapes}}. The same procedure is performed both for SM and BSM Monte Carlo samples, in every VBF categories, Optimal Observable bins and CP-mixing hypothesis the \myy models are determined by their best fit results from MC.

\begin{figure}[htbp]
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_TTbin0.png}
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_TLbin0.png}
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_LTbin0.png}
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_LLbin0.png}\\
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_TTbin1.png}
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_TLbin1.png}
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_LTbin1.png}
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_LLbin1.png}\\
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_TTbin2.png}
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_TLbin2.png}
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_LTbin2.png}
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_LLbin2.png}\\
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_TTbin3.png}
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_TLbin3.png}
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_LTbin3.png}
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_LLbin3.png}\\
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_TTbin4.png}
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_TLbin4.png}
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_LTbin4.png}
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_LLbin4.png}\\
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_TTbin5.png}
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_TLbin5.png}
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_LTbin5.png}
  \includegraphics[width=.24\textwidth]{figure/sigmodel/SigFit_LLbin5.png}\\
  \caption{Fitted Doubles Sided Crystal ball shapes for all the reconstructed categories of the analysis. }
  \label{fig:signal_shapes}
\end{figure}

In the final statistical workspace:
\begin{itemize}
	\item the value of \mH is constrained to the one of \RunOne measurement, taking into account its error;
	\item all the six parameters of the DSCB are kept fix during the fit
	\item $\mu_\text{CB}$ and $\sigma_\text{CB}$ are each multiplied by a response function to take into account photon energy scale and photon energy resolution systematics, respectively (see \Sect{\ref{sssec:signal_shape_syst}})
\end{itemize}


\chapter{Extensions to Principal Component Methods}

\cite{Flury:1995} notes that in order to generalise principal components successfully, it is necessary to identify key properties of the method and define more general models where these properties are still valid.   He proposed an extension which retained an orthogonal transformation which maps correlated variables onto uncorrelated ones, this will be considered in section \ref{cpc}.   However, we also consider the principal curves approach which retains the self-consistency elements of principal components axes.   Firstly, we consider generalisations which attempt to deal with non-linearities in the data.

\section{Gnanadesikan's Generalised PCA}

One obvious extension to principal components was made by \cite{Gnanadesikan:1977}, and in a similar manner to polynomial regression adds quadratic and, where necessary, higher order terms to account for non-linearities in the day.   A conventional principal components analysis can then be carried out on this larger data set.   This technique has been succesfully demonstrated on artificial data lying on a circle, and is quite easy to implement:
\singlespacing
\begin{verbatim}
gpca <- function(X, ...){
 p <- dim(X)[2]
 p2 <- p + (p-1)*2
 counter <- p+1
    for (i in 1:p){
       for (j in i:p) {
           X[,counter] <- X[,i] * X[,j]
           counter <- counter + 1
                      }
                   }
  results <- prcomp(X, scale = TRUE, ...)
  return(results)
}

iris.gpca <- gpca(iris[,-5])
plot(iris.gpca$x[,1], gnad.iris$x[,2],
  col = as.numeric(iris[,5]))
plot(iris.gpca)
biplot(iris.gpca)
\end{verbatim}
\onehalfspacing

Whilst polynomial regression has been successful, \cite{Flury:1995} notes that few publications feature this methodology.   The presence of linear and quadratic terms on different scales, different origins with a new metric may all bear little relationship to the distance from a curve in the original $p$-dimensional space.
We therefore consider some other proposals to extend the basic principal component approach.

\section{Simple Components}

Simple components analysis is very much an adaptation of the existing method to make the results more to interpret.  The original, optimal, principal components are replaced or supplements by simpler ones.
This procedure is a data analytical heuristic, so whilst principal components are by definition 100\% optimal, the user of the technique can adjust the results so that they are more easily understood but at the expense of reduced optimality.   A user judgement then has to be made as to whether the loss in optimality is justified given the ease of interpretation.

We define the easier to interpret \emph{block-components} as those where all loadings are positive, and \emph{difference components} as those where some loadings are positive some are negative.   Most principal component analyses only contain one block-component, the simple component analysis allows the user to specify the number of block components to be fitted, provided the correlations among them are below some cut-off value (typically .3 or .4).   This is clearly a partitioning of the original untransformed variables, agglomerative hierarchical clustering is carried out at this stage.   Having done this, we only need to decide what we mean by simple difference comonents.   Those are obtained as simplified versions of some appropriate ``residual components''. The idea is to retain the large loadings (in absolute value) of these residual components and to shrink to zero the small ones. For each difference-component, the interactive procedure sca displays the loadings of the corresponding residual component (at the right side of the window), such that the user may know which variables are especially important for the definition of this component.

At the third stage of the interactive procedure sca, it is possible to remove some of the difference-components from the system.

For many examples, it is possible to find a simple system which is 90\% or 95\% optimal, and where correlations between components are below 0.3 or 0.4. When the structure in the correlation matrix is complicated, it might be advantageous to invert the sign of some of the variables in order to avoid as much as possible negative correlations. This can be done using the option `invertsigns=TRUE'.

In principle, simple components can be calculated from a correlation matrix or from a variance-covariance matrix. However, the definition of simplicity used is not well adapted to the latter case, such that it will result in systems which are far from being 100\% optimal. Thus, it is advised to define simple components from a correlation matrix, not from a variance-covariance matrix.

\singlespacing
\begin{verbatim}
X <- cor(iris[,-5])
sca(X)
 if(interactive()) {
+   r <- sca(cor(X), corblocks=0.4, qmin=5, interactive = TRUE)
+ }
\end{verbatim}
\onehalfspacing

\section{acpgen}

\section{Generalised principal components}

Further robust methods include proposals by \cite{Caussinus+Ruiz-Gazen:1993} which carries out a spectral analysis of a robust covariance matrix.

\singlespacing
\begin{verbatim}
library(amap)
X <- iris[,-5]
p <- acpgen(X,h1=1,h2=1/sqrt(2), scores = TRUE)

plot(p,main='ACP Iris data') ## the biplot

plot(p$scores[,1], p$scores[,2], col = as.numeric(iris[,5]), center = TRUE, reduce  TRUE)
\end{verbatim}
\onehalfspacing

A choice of kernels are available: gaussien: $\frac{1}{\sqrt{(2\pi)}} \exp(-u^{2}/2)$, quartic: $\frac{15}{16} (1-u^{2})^{2};  \mbox{if} |u| < 1$, triweight: $\frac{35}{32} (1-u^{2})^{3}; \mbox{if} |u| < 1$; epanechikov: $\frac{3}{4} (1-u^{2}) \mbox{if} |u| < 1$; cosinus: $\frac{\pi}{4} \cos (u \times \pi/2); \mbox{if} |u| < 1$ and require \verb+h+ to set the kernel bandwidth.   \verb+center+ and \verb+reduce+ set are binary indicators which centre and standardise the data.

%    sdev: the standard deviations of the principal components.
%
%   loadings: the matrix of variable loadings (i.e., a matrix whose columns
%      contain the eigenvectors).  This is of class '"loadings"':
%          see 'loadings' for its 'print' method.
%
%  scores: if 'scores = TRUE', the scores of the supplied data on the
%          principal components.
%
%     eig: Eigen values

\section{Kernel Based Methods}

Kernel methods are widely used in many multivariate applications, formal proposals for their use in connection with principal components seem to be much more recent.   \cite{Schoelkopf+etal:1998} made proposals for the use of kernels in eigen analysis which can be used as a form of non-linear principal component analysis.   This method has been made available within the \verb+kernlab+ library \cite{Karatzoglou+etal:2005} in \textbf{R}.   It should perhaps be noted that kernel solutions increase the computer power associated with analysis of a dataset of given dimensionaltiy.

Any kernel  function which computes a dot product between two vectors can be used, these are either specified by the name of a user defined function or by using one of the inbuilt kernel functions which include \verb+rbfdot+, a Gaussian radial basis kernel function and  \verb+laplacedot+, a Laplacian kernel function.   These  both require the inverse kernel width \verb+sigma+ to be specified by \verb+par = list(sigma)+.  Other kernels include \verb+polydot+, a polynomial kernel function requiring the parameters \verb+kpar = list(degree, scale, offset)+; \verb+vanilladot+,a linear kernel function; \verb+tanhdot+, a hyperbolic tangent kernel function which requires \verb+kpar = list(scale, offset)+ to be set; \verb+besseldot+, a Bessel kernel function requiring  \verb+kpar = list(sigma, order, degree)+ to be set.

A couple of points need to be noted.   \verb+features+ specifies the number of features to be retained, as  kernel method these aren't the same as the conventional form of principal components.   The default value of \verb+0+ therefore produces rather a large number of components, which can be limited by an argument to \verb+th+, set by default to 0.0001.   Again, this can't be considered in the same way as for say, the Kaiser criterion.

Consider an example with Fisher's Iris data.   We extract two principal components (\verb+features=2+), and select a Gaussian kernel with an inverse kernel width of 0.2

\begin{verbatim}
\library(kernlab)
data(iris)
test <- sample(1:150,20)

kpc <- kpca(~.,data=iris[-test,-5],
    kernel="rbfdot", kpar=list(sigma=0.2), features=2)
pc <- prcomp(~.,data=iris[-test,-5] )

par(mfrow = c(1,2))
plot(pc$x[,1], pc$x[,2], col = as.integer(iris[-test,5]),
    xlab="1st Principal Component",ylab="2nd Principal Component",
    main = "Conventional pca")

emb <- predict(pc, iris[test, -5])
points(emb,pch=as.numeric(iris[test,5]))

plot(rotated(kpc), col=as.integer(iris[-test,5]),
    xlab="1st Principal Component",ylab="2nd Principal Component",
    main = "Kernel PCA")

emb <- predict(kpc,iris[test,-5])
points(emb,pch=as.numeric(iris[test,5]))
\end{verbatim}

It should be noted that the \verb+kernlab+ package has been set out following S4 classes.   Therefore, whilst it is possible to extract eigenvalues from conventional principal components using \verb+pc$sdev^2+ (note that, as implied by its name, \verb+sdev+ returns the square roots of the eigenvalues), when using \verb+kernlab+ the extractor functons have to be used.   Therefore, \verb+eig(kpc)+ will give the eigenvalues from the kernel based decomposition.

\begin{figure}
\begin{center}
\includegraphics[width = 0.8\textwidth]{images/kernelpca}
\caption{Projection from conventional and kernal pca}
\label{kernpca}
\end{center}
\end{figure}

\section{Principal Curve Analysis}

\singlespacing
\begin{verbatim}
library(pcurve)
par(mfrow = c(3,2), cex = 0.5)
spec.fit <- pcurve(iris[,-5], start = "pca", plot.init = FALSE, plot.segs = FALSE, plot.resp = FALSE, maxit = 5)
\end{verbatim}
\onehalfspacing

\includegraphics[width = 0.7\textwidth]{images/irispcurve}

This does beg the question as to whether we should be performing one or three principal component analyses.   This generalisation of the technique will be considered in the next section.

\section{Principal components for more than one data set}
\label{cpc}

Finally, we consider principal components for more than one set of data, as discussed before this is a clear generalisation of the principal component technique.   Given the largely data analytic rationale for principal components analysis which dominates the discussion in chapter \ref{pca} it is perhaps understandable that models which could deal with covariance matrices arising from related but separate groups had not been developed.   Although \cite{Jolicoeur:1963} considered data from male and female turtles their analysis was essentially separate and informal comparisons were made between the groups.   Formal comparison of group specific principal components was first proposed by \cite{Krzanowski:1980} who examined the angles between eigenvectors.   He found that an optimum could be found by carrying out a spectral decomposition of:
\begin{equation}
\boldsymbol{H} = \sum_{i=1}^{k}\boldsymbol{L}_{i}^{T}\boldsymbol{L}_{i}
\end{equation}
where $\boldsymbol{L}_{i}$ are the $p \times q$ matrices of eigenvectors for each group $i$.   This method however does not weight the matrices $\boldsymbol{L}_{i}$ regardless of possibly different sample size and does not adequately account for instabilities in eigenvectors.

A formal model for simultaneous analysis has been proposed by \cite{Flury:1984}, which assumes common eigenvectors but group specific eigenvalues.

\begin{displaymath}
\boldsymbol{\Sigma}_{i} = \boldsymbol{E} \boldsymbol{\Lambda}_{i}  \boldsymbol{E}^{T}
\end{displaymath}
which assumes parallel ellipses of potentially different size.

This method continues to be updated with recent proposals by \cite{Boik:2002}.  However, the CPC model is readily demonstrated within \textbf{R}.

%
%library(grid)
% pltSplomT(USArrests,mainL="",hist="b",adjust=0.4,cex.diag = 0.7)
%library(cwhplot)

%%% Local Variables: ***
%%% mode:latex ***
%%% TeX-master: "../book.tex"  ***
%%% End: ***


\documentclass[12pt, a4paper]{article}
\usepackage{hyperref}
\usepackage{amsthm}
\usepackage{amsmath}
\usepackage{mathtools}
\usepackage[
    bindingoffset=-15pt,
	voffset=0pt,
	hoffset=10pt,
	textwidth=500pt,
    textheight=650pt,
	marginparwidth=20pt,
	footskip=60pt
]{geometry}
\usepackage{algorithm}
\usepackage{cleveref}

\headheight=28pt

\begin{document}

\newtheorem{lemma}{Lemma}
\newtheorem{theorem}{Theorem}

\section*{Ben-Or asynchronous Byzantine agreement algorithm}
\subsection*{Introduction}
The idea of the Byzantine agreement problem is to have a system of $N$ processes, $t$ of which are faulty. Each correct process starts with either 0 or 1, and after running the algorithm, all correct processes must end up with either 0 or 1 based on the following rules:
\begin{itemize}
    \item All correct processes end up with the same value.
    \item If all correct processes start with the same value $v$, then all correct processes must end with $v$.
    \item All correct processes must terminate (with probability 1).
\end{itemize}
Ben-Or algorithm allows the network to be asynchronous and does not restrict the malevolence of faulty processes. It works under assumption $t<\frac{N}{5}$.
\subsection*{Protocol}
Let $N$ - number of all processes, $t$ - number of faulty processes, $t<\frac{N}{5}$\\
Let's bring up the pseudo code\cite{Ben83}:
\begin{flushleft}
\textbf{Process} P: Initial value $x_p$.\\
\textbf{step 1}: Set $r\coloneqq 1$.\\
\textbf{step 0}: Send the message (1, $r$, $x_p$) to all the
processes.\\
\textbf{step 2}: Wait till messages of type (1, $r$, *) are received from $N-t$ processes. If more than $(N-t)/2$ messages have the same value $v$, then send the message (2, $r$, $v$, D) to all processes. Else send the message (2, $r$, ?) to all processes.\\
\textbf{step 3}: Wait till messages of type (2, $r$, *) arrive from $N-t$ processes.\\
(a) If there are at least $t+1$ D-messages (2, $r$, $v$, D), then set $x_p \coloneqq v$.\\
(b) If there are more than $(N + t)/2$ D-messages then \textbf{decide} $v$.\\
(c) Else set $x_p=$ 1 or 0 each with probability $\frac{1}{2}$.\\
\textbf{step 4}: Set $r \coloneqq r + 1$ and go to step 1.
\end{flushleft}
Processes locally keep track of current round number, and store current binary value.\\
Within each round they go through 2 stages, messages from each round from now on will be called type 1 or 2 messages.
\subsection*{Correctness}
Let's start with proving the following lemma:
\begin{lemma}\label{same-value}
If all correct processes start with the value $v$, then within one round they will all decide $v$.
\end{lemma}
\begin{proof}
Each process broadcasts message $(1,\ 1,\ v)$.\\
Process then waits for $N-t$ messages.\\
At most $t$ messages came from faulty processes, which means that at least $N-2\cdot t$ messages are correct. Because $N-2\cdot t>\frac{N+t}{2}$, all the correct processes send message $(2,\ 1,\ v,\ D)$.\\
Among $N-t$ accepted type 2 messages at most $t$ are incorrect, which means that step 3(b) will not execute.\\
Again, because at least $N-2\cdot t$ of type 2 messages are correct, then more than $\frac{N+t}{2}$ of them will have the same value (v). That means that every process will \textbf{decide $v$} this round.
\end{proof}
\begin{lemma}\label{one-choice}
If process sets $v$ in step 3(a) in round $r$, then it can't set $\neg v$ in the same round.
\end{lemma}
\begin{proof}
Let's say that process saw $\geq t+1$ messages  $(2,\ r,\ 0,\ D)$ and $\geq t+1$ messages $(2,\ r,\ 1,\ D)$. Let $A_x$ be equal to the number of messages $(2,\ r,\ x,\ D)$ originated from correct processes with $x=0,1$. Of course $A_0+A_1\geq t+2$.\\
Every process responsible for $A_x$ saw more than $\frac{N-t}{2}$ messages $(1,\ r,\ x)$ from correct process. But that means that there are more than $N-t$ correct processes, which is a contradiction.
\end{proof}
Then let's look at the next lemma, which states that the processes will decide with at most 2 round window.
\begin{lemma}
If for some round r, some correct process decides $v$ in step 3(b), then all other correct processes will decide $v$ within the next round.
\end{lemma}
\begin{proof}
Let $P$ be the process that has decided on $v$.
In order to \textbf{decide $v$}, $P$ must have received more than $\frac{N+t}{2}$ messages $(2,\ r,\ \_,\ D)$. That means that more than $\frac{N-t}{2}$ correct processes sent that message. Let $A_x$ be the number of $(2,\ r,\ x,\ D)$ messages from correct processes for $x=0,1$.\\
Because $P$ has decided $v$, it follows that $A_{\neg v}\leq t$. On top of that $A_v+A_{\neg v}>\frac{N-t}{2}$.\\
Simple calculations show that $A_v\geq t+1$. It means that every other process will set $v$ in step 3(a). From \cref{one-choice} we know that it will be the only value that they will set.\\
It follows that in the next round every correct process will start with $v$, so \cref{same-value} can be applied.
\end{proof}
\subsection*{Time complexity}
In each round processes have probability of at least $2^{-(N-t)+1}$ of all setting the same value, which using \textit{Lemma 1} means that the run algorithm would end next round. That means that expected number of rounds is bounded by $O(2^{N-t})$.\\
The following theorem shows tighter bound under stricter assumptions:
\begin{theorem}\cite{Ben83}
If $t=O(\sqrt{N})$ then the expected number of rounds to reach agreement in this protocol is constant, i.e., it does not depend on N.
\end{theorem}
\begin{thebibliography}{9}
\bibitem{Ben83} Michael Ben-Or. Another Advantage of Free Choice: Completely Asynchronous Agreement Protocols. In \textit{Proceedings of the second annual ACM symposium on Principles of distributed computing}, pages 27-30. ACM, 1983, doi: 10.1145/800221.806707.
\end{thebibliography}

\end{document}